metadata
license: apache-2.0
tags:
- nanoGPT
- code-generation
- fine-tuned
base_model: Ananda100/pocketcoder100M-distilled
100m-sftd-python
Full-parameter fine-tune of Ananda100/pocketcoder100M-distilled
on a custom Python coding-problem dataset, using the deepseek-ai/deepseek-coder-6.7b-base tokenizer.
- Fine-tune tokens processed: 356,253,696
- Best validation loss: 0.9270
- Held-out pass@1: 9.5%
- Block size: 512
- Vocab size: 32022
Loading
This is a custom architecture, not a native transformers model, so AutoModel.from_pretrained
won't work out of the box:
import json, torch
from safetensors.torch import load_file
with open("config.json") as f:
cfg = json.load(f)
config = GPTConfig(**cfg)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()