YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
XpertGPT Fine-Tuning Pipeline
This repository contains the fine-tuning pipeline and downstream evaluation results for the XpertGPT model (anonym5035/swi_glu_sw_64_16_8_4_xpert_gpt) trained for the BabyLM 2026 Strict-Small track.
The pipeline (finetune_eval.py) is designed to fine-tune the causal pre-trained model on 9 diverse sequence classification and multiple-choice tasks, bypassing autoregressive generation in favor of classification heads.
Core Hyperparameters (Global)
Across all tasks, the following core training configurations are enforced:
| Hyperparameter | Value | Notes |
|---|---|---|
| Optimizer | AdamW |
betas=(0.9, 0.999), eps=1e-8 |
| Weight Decay | 0.01 |
Lighter than pretraining (0.1) for small dataset finetuning |
| LR Schedule | Linear Decay | Preceded by a linear warmup phase |
| Warmup Ratio | 0.06 |
Warmup for 6% of total training steps |
| Gradient Clipping | 1.0 |
Matches pretraining |
| Max Sequence Len | 128 |
Truncated from 512 for efficiency on short tasks |
| Pooling Method | Exact Last Token | (mask * arange).argmax(dim=1) to ensure robust extraction |
| Hardware | 2x T4 GPUs | nn.DataParallel scales batch sizes automatically |
Task-Specific Hyperparameters
Because the datasets range from massive (105K examples) to tiny (250 examples), hyperparameters are bucketed by dataset size to prevent overfitting.
| Task | Type | Dataset Size | Epochs | Learning Rate | Base Batch Size |
|---|---|---|---|---|---|
| QNLI | 2-way Classification | Large (~105K) | 3 | 2e-5 |
16 |
| SWAG | 4-way Multiple Choice | Large (~73K) | 3 | 2e-5 |
16 |
| Social IQa | 3-way Multiple Choice | Large (~33K) | 3 | 2e-5 |
16 |
| WinoGrande | 2-way Multiple Choice | Large (~40K) | 3 | 2e-5 |
16 |
| SST-2 | 2-way Classification | Large (~67K) | 3 | 2e-5 |
16 |
| CoLA | 2-way Classification | Medium (~8.5K) | 5 | 3e-5 |
16 |
| OpenBookQA | 4-way Multiple Choice | Medium (~5K) | 5 | 3e-5 |
16 |
| COPA | 2-way Multiple Choice | Tiny (~400) | 5 | 5e-5 |
8 |
| CommitmentBank | 3-way Classification | Tiny (~250) | 5 | 5e-5 |
8 |
(Note: The "Base Batch Size" is per-GPU. Under a 2x T4 DataParallel setup, the effective batch sizes become 32 and 16 respectively).
Outputs
The script uploads two artifacts back to this repository upon completion:
results/finetune_results.json: The highest validation accuracy achieved for each dataset.results/training_history.json: Epoch-by-epoch tracking oftrain_loss,val_loss, andval_accfor advanced overfitting analysis.
Kaggle Execution
To execute this pipeline inside a Kaggle notebook (using the T4 x2 accelerator):
# 1. Install dependencies
!pip install transformers datasets huggingface_hub torch tqdm
# 2. Download the fine-tuning script
!wget -O finetune_eval.py https://huggingface.co/anonym5035/finetuning_original_model/raw/main/finetune_eval.py
# 3. Run the pipeline and push results to Hugging Face
!python finetune_eval.py --hf_token YOUR_HF_TOKEN_HERE