Instructions to use SRJ5035/XpertGPT-BabyLM2026-Strict-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SRJ5035/XpertGPT-BabyLM2026-Strict-Small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SRJ5035/XpertGPT-BabyLM2026-Strict-Small", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("SRJ5035/XpertGPT-BabyLM2026-Strict-Small", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SRJ5035/XpertGPT-BabyLM2026-Strict-Small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SRJ5035/XpertGPT-BabyLM2026-Strict-Small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SRJ5035/XpertGPT-BabyLM2026-Strict-Small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/SRJ5035/XpertGPT-BabyLM2026-Strict-Small

SGLang

How to use SRJ5035/XpertGPT-BabyLM2026-Strict-Small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SRJ5035/XpertGPT-BabyLM2026-Strict-Small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SRJ5035/XpertGPT-BabyLM2026-Strict-Small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SRJ5035/XpertGPT-BabyLM2026-Strict-Small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SRJ5035/XpertGPT-BabyLM2026-Strict-Small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use SRJ5035/XpertGPT-BabyLM2026-Strict-Small with Docker Model Runner:
```
docker model run hf.co/SRJ5035/XpertGPT-BabyLM2026-Strict-Small
```

XpertGPT Strict-Small

XpertGPT (Mixture of Experts with Parallelized Multi-Scale Information Transmission) is a sparse, data-efficient recurrent language model for the BabyLM 2026 challenge (Strict-Small (10M) track, 10M words). It combines sliding window attention global streams with sparse parallel expert blocks using Expert Choice routing. ~48.4M non-embedding parameters / ~52.6M total parameters. Custom code (trust_remote_code=True).

Architecture: 6 layers of MoEP-MSIT blocks. Each block combines a lower-dimensional dense global sliding window attention layer (dim = 256) with 4 parallel high-dimensional sparse expert blocks (dim = 384) routed via Expert Choice gating.
Track: BabyLM 2026 Strict-Small (10M) (10M words).
Tokenizer: Custom BPE tokenizer (vocab size: 16384).
Revision / Checkpoint: main

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("SRJ5035/XpertGPT-BabyLM2026-Strict-Small", revision="main", trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained("SRJ5035/XpertGPT-BabyLM2026-Strict-Small", revision="main")
ids = tok("The quick brown fox", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits

Intermediate checkpoints

Intermediate training checkpoints are provided as git revisions named chck_<N>M for the BabyLM challenge fast-eval.

License and citation

Released under CC BY-NC 4.0 (attribution required, non-commercial only). If you use this model or code, please cite (see CITATION.cff):

@misc{jain2026xpertgpt,
  title        = {XpertGPT: Mixture of Experts with Parallelized Multi-Scale Information Transmission for Data-Constrained Pretraining},
  author       = {Jain, Soham and Singh, Harsh and Dewan, Divija and Dev, Atul},
  year         = {2026},
  howpublished = {Hugging Face Repository},
  note         = {XpertGPT MoE language model, BabyLM 2026}
}

Provenance and integrity fingerprints are documented in PROVENANCE.md.

Downloads last month: 557

Safetensors

Model size

56.8M params

Tensor type

F32