christianrss/chris-gpt-2-124m

Chris-GPT-2 124M

Chris-GPT-2 124M is a GPT-2-style causal language model trained from scratch on approximately 10 billion tokens from FineWeb-Edu.

The model was developed as part of an independent experiment focused on reproducing the complete GPT-2 pretraining pipeline, including model implementation, distributed training, checkpointing, validation, evaluation, inference, and model export.

This repository provides the canonical model in Hugging Face Transformers / SafeTensors format.

A GGUF version is also available for inference runtime experiments:

https://huggingface.co/christianrss/chris-gpt-2-124m-GGUF

Model Details

Property	Value
Architecture	GPT-2
Parameters	124,475,904
Transformer layers	12
Attention heads	12
Embedding dimension	768
Context length	1,024 tokens
Model vocabulary size	50,304
Tokenizer	GPT-2 byte-level BPE
Tokenizer vocabulary	50,257
Training tokens	~10 billion
Training dataset	FineWeb-Edu
Training	From scratch
Framework	PyTorch
Model format	SafeTensors
Final validation loss	3.07248

The model uses a padded vocabulary size of 50,304 internally while retaining the original GPT-2 BPE tokenizer with 50,257 tokens.

Training

Chris-GPT-2 was pretrained from randomly initialized weights rather than fine-tuned from an existing GPT-2 checkpoint.

The training target was approximately 10 billion tokens:

524,288 tokens/step × 19,073 steps
≈ 10,000,000,000 tokens

Training was performed using:

• 4× NVIDIA A100 PCIe 40 GB GPUs
• Distributed Data Parallel (DDP)
• AdamW optimizer
• GPT-2 byte-level BPE tokenization
• 1,024-token context length
• FineWeb-Edu training data

The final checkpoint was produced at step 19,072.

Final validation loss:

3.0724804401397705

Dataset

The model was trained on approximately 10 billion tokens from FineWeb-Edu.

FineWeb-Edu is an educationally filtered subset of FineWeb designed for language-model pretraining research.

Usage

Chris-GPT-2 can be loaded directly with Hugging Face Transformers.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "christianrss/chris-gpt-2-124m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The future of artificial intelligence is"

inputs = tokenizer(
    prompt,
    return_tensors="pt"
)

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
)

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )
)

Example Generation

Prompt:

The future of artificial intelligence is

Generation parameters:

temperature = 0.8
top_k       = 50
top_p       = 0.95
max_new_tokens = 50

Example output:

The future of artificial intelligence is a critical factor in the global economy. It’s also an important factor in our society’s ability to cope with global warming.
What are the future of artificial intelligence?
Artificial intelligence has the potential to become an important component

This is an actual sampled generation from the trained model and was not manually curated.

Hugging Face Conversion

The original training checkpoint was stored as a PyTorch .pt checkpoint.

It was converted to the Hugging Face GPT2LMHeadModel representation and exported using SafeTensors.

The original Chris-GPT implementation uses torch.nn.Linear for the GPT-2 projection layers, while Hugging Face GPT-2 uses its Conv1D representation.

The following weights are therefore transposed during conversion:

attn.c_attn.weight
attn.c_proj.weight
mlp.c_fc.weight
mlp.c_proj.weight

The conversion process performs tensor-level verification after mapping the checkpoint parameters.

The resulting model is then reloaded using:

GPT2LMHeadModel.from_pretrained(...)

and a deterministic forward pass is executed to verify that the exported model produces finite logits with the expected dimensions.

The final exported model was also tested through the standard Hugging Face API:

AutoTokenizer.from_pretrained(...)
AutoModelForCausalLM.from_pretrained(...)

and successfully performed autoregressive text generation.

Repository Files

README.md
config.json
generation_config.json
model.safetensors
tokenizer.json
tokenizer_config.json
chris_conversion.json

model.safetensors

Contains the converted Chris-GPT-2 model parameters.

config.json

Defines the GPT-2 architecture used by Hugging Face Transformers.

generation_config.json

Contains the generation configuration associated with the exported model.

tokenizer.json

Contains the GPT-2 byte-level BPE tokenizer representation.

tokenizer_config.json

Contains tokenizer configuration used by Hugging Face Transformers.

chris_conversion.json

Contains metadata describing the conversion from the original Chris-GPT training checkpoint to the Hugging Face representation.

GGUF

A GGUF version of Chris-GPT-2 124M is available in a separate repository:

https://huggingface.co/christianrss/chris-gpt-2-124m-GGUF

The GGUF export is intended for lightweight inference runtimes, low-level inference experiments, and development of custom runtime infrastructure.

The conversion pipeline is:

Chris-GPT PyTorch checkpoint
            ↓
Hugging Face GPT2LMHeadModel
            ↓
SafeTensors
            ↓
GGUF conversion
            ↓
Chris-GPT-2-124M-F16.gguf

Currently available:

File	Precision	Description
Chris-GPT-2-124M-F16.gguf	F16	Reference half-precision GGUF export

For standard Hugging Face Transformers usage, the SafeTensors version in this repository is recommended.

For GGUF usage and runtime experiments, see:

https://huggingface.co/christianrss/chris-gpt-2-124m-GGUF

Model Formats

Chris-GPT-2 is currently distributed in two primary formats.

Transformers / SafeTensors

Repository:

https://huggingface.co/christianrss/chris-gpt-2-124m

Recommended for:

• Hugging Face Transformers
• PyTorch inference
• experimentation
• evaluation
• fine-tuning
• downstream research

GGUF

Repository:

https://huggingface.co/christianrss/chris-gpt-2-124m-GGUF

Recommended for:

• lightweight inference
• custom inference runtimes
• GGUF experiments
• quantization experiments
• low-level systems development

Evaluation

Validation was performed throughout pretraining.

Final validation loss:

3.07248

The training pipeline also included HellaSwag evaluation during development.

Chris-GPT-2 should not be interpreted as competitive with modern large language models.

The experiment is primarily intended to study and reproduce the complete pretraining and inference pipeline of a GPT-2-scale Transformer.

Limitations

Chris-GPT-2 is a relatively small base autoregressive language model.

It has not undergone:

• instruction tuning
• reinforcement learning from human feedback (RLHF)
• preference optimization
• safety fine-tuning
• conversational alignment

The model may generate:

• incorrect information
• repetitive text
• incoherent continuations
• biased content
• undesirable content

It should not be used as an authoritative source of factual information.

The model was developed primarily as a research and engineering experiment, rather than as a production conversational assistant.

Project Goals

The project was designed to explore the complete lifecycle of a language model rather than only fine-tuning an existing pretrained model.

dataset
   ↓
tokenization
   ↓
Transformer implementation
   ↓
distributed pretraining
   ↓
validation
   ↓
checkpointing
   ↓
evaluation
   ↓
inference
   ↓
Hugging Face export
   ↓
SafeTensors
   ↓
GGUF export
   ↓
custom inference runtimes

The resulting checkpoint is also being used as a reference workload for experiments with custom machine-learning and inference infrastructure.

Chris Llama

The GGUF export of Chris-GPT-2 is being used as a reference workload for development of Chris Llama, an experimental low-level LLM inference runtime.

Repository:

https://github.com/christianrss/chris-llama

Chris Llama explores topics including:

• GGUF parsing
• tensor loading
• GPT-2 inference
• KV caching
• quantization
• CPU inference
• heterogeneous compute
• AdaptiveCpp
• SYCL
• GPU acceleration

The project is intended to study how LLM inference systems work below high-level machine-learning frameworks.

Related Projects

Chris-GPT-2

Original GPT-2 training implementation and experiment code:

https://github.com/christianrss/chris-gpt-2

Chris Torch

Experimental machine-learning framework with custom autograd, neural-network components, optimizers, and native compute backends:

https://github.com/christianrss/chris-torch

Chris Llama

Experimental low-level LLM inference runtime with GGUF and heterogeneous compute experiments:

https://github.com/christianrss/chris-llama

Research

The pretraining experiment is documented in:

A Reproducible 10-Billion-Token Pretraining Run of GPT-2 124M

ResearchGate:

https://www.researchgate.net/publication/412096883_A_Reproducible_10-Billion-Token_Pretraining_Run_of_GPT-2_124M

The report documents:

• training methodology
• FineWeb-Edu
• training dynamics
• validation
• HellaSwag evaluation
• generation behavior
• compute infrastructure
• training cost
• reproducibility

Reproducibility

The project intentionally separates the major stages of the model lifecycle:

Training code
    │
    ▼
PyTorch checkpoint
    │
    ▼
Hugging Face conversion
    │
    ├───────────────┐
    ▼               ▼
SafeTensors        GGUF
    │               │
    ▼               ▼
Transformers    inference runtimes

This allows the trained checkpoint to be inspected and executed through standard machine-learning tooling while also serving as a reference model for lower-level inference experiments.

References

• Alec Radford et al. Language Models are Unsupervised Multitask Learners. OpenAI, 2019.
• Ashish Vaswani et al. Attention Is All You Need. 2017.
• Hugging Face FineWeb / FineWeb-Edu.
• Hugging Face Transformers.
• GGUF / GGML.
• llama.cpp.
• Andrej Karpathy, Let's reproduce GPT-2 (124M).

Author

Christian Rafael de Souza Silva

Independent Researcher

GitHub:

https://github.com/christianrss