GPT-Fem-Compact

GPT-Fem-Compact is a version of GPT-Compact finetuned on 18 GB of text on and about women.

Model Description

GPT-Fem-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality. It was trained on a diverse corpus of over 17 billion tokens.

Model Architecture

Layers	24
Heads	16
Embeddings	1280
Context Window	16384 tokens
Tokenizer	cl100k_base