Sugutt/Kalenjin-qwen2.5-7B

LoRA adapter from continued pretraining for a Kalenjin-focused language model. The adapter was trained on Modal from Qwen/Qwen2.5-7B and is intended as the CPT checkpoint before later SFT, DPO, and GRPO stages.

Model Scope

• Base model: Qwen/Qwen2.5-7B
• Method: QLoRA Continued Pretraining
• Language/domain: Kalenjin-heavy text
• Adapter only: yes
• Training objective: causal language modeling
• Not instruction-tuned yet
• Not a translator yet

Contents

• adapter/: PEFT LoRA adapter weights and config
• tokenizer/: tokenizer saved with the CPT run
• training_config.json: Modal CPT training configuration
• metrics.json: training and evaluation metrics
• evaluation_report.json: CPT adapter evaluation report, if available
• base_evaluation_report.json: base model evaluation report, if available
• samples.txt: qualitative generation samples

Evaluation

Held-out file: /data/processed/kalenjin_cpt_eval.jsonl

Model	Eval loss	Perplexity	Metric
Base	5.4601	235.1302	batch mean
CPT adapter	2.7906	16.2915	batch mean

Token-weighted metrics may also be present in the JSON reports. The batch-mean metric matches the training/Trainer-style evaluation path.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "Qwen/Qwen2.5-7B"
adapter_repo = "Sugutt/Kalenjin-qwen2.5-7B"

tokenizer = AutoTokenizer.from_pretrained(adapter_repo)
model = AutoModelForCausalLM.from_pretrained(base_model, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_repo)

Training Notes

{
  "model_name": "Qwen/Qwen2.5-7B",
  "train_file": "/data/processed/kalenjin_cpt_train.jsonl",
  "eval_file": "/data/processed/kalenjin_cpt_eval.jsonl",
  "output_dir": "/outputs/kalenjin-cpt",
  "max_seq_length": 2048,
  "epochs": 2.0,
  "learning_rate": 2e-05,
  "batch_size": 2,
  "grad_accum": 8,
  "lora_r": 32,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "save_steps": 100,
  "logging_steps": 10,
  "max_steps": -1,
  "warmup_ratio": 0.03,
  "weight_decay": 0.01,
  "lr_scheduler_type": "cosine",
  "optimizer": "paged_adamw_8bit",
  "run_name": "kalenjin-cpt",
  "wandb_enabled": true,
  "wandb_project": "kalenjin-llm",
  "wandb_entity": "",
  "wandb_tags": "cpt,kalenjin,modal",
  "wandb_mode": "online"
}

This repository contains an adapter, not a merged full model.