GPT-2 Fine-tuning on SVAMP Dataset
This project fine-tunes OpenAI's GPT-2 model on the SVAMP (Simple Variants of Arithmetic Math Problems) dataset to improve mathematical problem-solving capabilities.
Overview
The model is trained to solve arithmetic word problems by fine-tuning GPT-2 on the SVAMP dataset. This enhances the model's ability to understand and solve mathematical reasoning tasks.
Model & Dataset Information
Base Model
- Model:
openai-community/gpt2 - Architecture: GPT-2 (Causal Language Modeling)
- Tokenizer: tiktoken (GPT-2 encoding)
- EOT Token ID: 50256 (used for end-of-text and padding)
Dataset
- Name:
ChilleD/SVAMP - Task: Mathematical problem solving (arithmetic word problems)
- Purpose: Improve mathematical reasoning and problem-solving capabilities
Training Configuration
Hyperparameters
- Epochs: 10
- Batch Size: 4
- Learning Rate: 1e-4
- Optimizer: AdamW (Adam with weight decay)
- Gradient Clipping: Max norm 1.0 (prevents gradient explosion)
- Max Sequence Length: 256 tokens
Hardware
- GPU: Tesla T4
- CUDA: Enabled
Training Results
The model was trained for 8 epochs with the following metrics:
| Epoch | Train Loss | Val Loss | Val Acc |
|---|---|---|---|
| 1 | 3.4209 | 2.0873 | 0.5634 |
| 2 | 2.1413 | 1.7239 | 0.6115 |
| 3 | 1.7929 | 1.5186 | 0.6445 |
| 4 | 1.6145 | 1.3977 | 0.6729 |
| 5 | 1.4897 | 1.3221 | 0.6829 |
| 6 | 1.4032 | 1.2842 | 0.6920 |
| 7 | 1.3729 | 1.2702 | 0.6955 |
| 8 | 1.3733 | 1.2693 | 0.6961 |
The model achieved a final validation accuracy of 69.61% with validation loss decreasing from 2.0873 to 1.2693 over 8 epochs. The training loss decreased from 3.4209 to 1.3733, indicating successful learning of the mathematical problem-solving task.
Training Details
Gradient Clipping
Gradient clipping with a maximum norm of 1.0 was applied during training to prevent gradient explosions and ensure training stability.
Tokenization
- Uses tiktoken tokenizer for GPT-2 encoding
- Sequences are truncated/padded to 256 tokens
- EOT token (50256) marks end of text and serves as padding token
Optimization
- AdamW optimizer with weight decay for regularization
- Learning rate of 1e-4 for stable convergence
- Batch size of 4 due to memory constraints on Tesla T4
Notes
- Training was performed on Google Colab with Tesla T4 GPU
- The model shows consistent improvement in loss reduction across all epochs
- Gradient clipping was essential for training stability
- Small batch size (4) was used due to usage limits