Jamil82/bdcoderAI-1b

bdcoderAI

A fully custom, locally-trained Bangla/Banglish + Python language model built from scratch. No hosted AI APIs (no ChatGPT/Gemini/Claude/OpenRouter). Runs on CPU only.

⚠️ Phase 1 real training is NOT started. Training data and pipeline are verified. Run Phase 1 training only after explicit user hi-go-ahead (AGENTS.md).

Quick Start

# 1. Install deps
pip install -r requirements.txt

# 2. (Optional) Regenerate training data + splits
python scripts/phase_01_content.py
python -c "from bdcoder.data import run_full_pipeline; print(run_full_pipeline('data/training/phase_01/raw', 'data/training/phase_01'))"

# 3. Launch chat UI (no model until checkpoint exists)
python app.py                  # http://127.0.0.1:8000

# 4. Generate text once a checkpoint exists
python generate.py -p "print('hello')" -i          # interactive
python generate.py -p "অগ্রপট থেকে গ্রীষ্মকালে কী পর্যায়?"

# 5. Train (Phase 1 — after user go-ahead)
python train.py --checkpoint checkpoints/checkpoint_latest

Architecture

Layer	Implementation
Model	GPT-style decoder-only Transformer (model/transformer.py)
Norm	RMSNorm
Activation	SwiGLU MLP
Position	RoPE (rotary embeddings)
Attention	F.scaled_dot_product_attention (causal)
Tokenizer	Byte-level BPE, vocab 8192, tokenizers library (tokenizer/v2/)
Embeddings	Tied (tie_embeddings=True)
Training	CPU-only, grad accumulation, AdamW + warmup-cosine LR
LoRA	Q/K/V + output projections, rank 4, alpha 16 (~0.35% params)

Special tokens / chat template

<|user|>\n<prompt>\n<|assistant|>\n<response>\n<eos_token>

tokenizer/v2/tokenizer.json registers: <pad>, <unk>, <bos_token>, <eos_token>, <|assistant|>, <|user|>.

Project Layout

bdcoder/                # Core Python package
  data.py               # Dataset pipeline (load/validate/dedup/split/save/stats)
  train_loop.py         # Training system (TrainConfig, ChatDataset, checkpoints, auto-resume)
  state.py              # Atomic training state (state.json)
  inference.py          # InferenceEngine (KV-cache, top-k/p, repetition penalty, SSE)
  memory.py             # ConversationMemory (scrubbing, language/topic/skill detection)
  behavior.py           # BehaviorLogger (intent/classification/clustering)
  eval_core.py          # EvalExample, CategoryResult, rule-based checks
  chat_server.py        # FastAPI + SSE chat backend (endpoints list below)
model/                  # Model code (NOT a wrapper — fully custom)
  config.py             # ModelConfig (JSON-driven, scales to 300M+)
  transformer.py        # GPTModel
  param_counter.py      # Parameter counting + CLI
  lora.py               # LoRA adapter (apply/save/load/count)
tokenizer_train.py      # Tokenizer training script (byte-level BPE, 8192 vocab)
configs/
  model_small.json      # Phase 1 (~7.2M params)
  model_medium.json     # Reference (~42M)
  chat.json             # Chat server settings
evaluation/
  eval_set/eval_set.jsonl   # Fixed eval set (~80 items, never in training)
  reports/              # Evaluation reports
  eval_core.py          # (see bdcoder/eval_core.py)
scripts/
  phase_01_content.py   # Dataset generation (8 categories)
  validate_dataset.py   # Dataset validation CLI
  analyze_user_behavior.py  # Behavior analytics CLI
  check_hardware.py     # Hardware profiling
data/
  training/phase_01/
    raw/raw_dataset.jsonl
    train.jsonl / val.jsonl / test.jsonl
    manifest.json
  memories/{user_id}.jsonl  # Conversation memories
  chats/                 # Session storage
web/
  templates/index.html    # Vanilla HTML/CSS/JS chat UI (no build step, no CDN)
app.py                   # CLI entry point — auto-load latest checkpoint + FastAPI + Uvicorn
chat.py                  # `from app import main` — alias entry point
generate.py              # CLI text generation (prompt / interactive)
evaluate.py              # CLI evaluation against checkpoint
train.py                 # Training CLI
requirements.txt
AGENTS.md                # Session instructions + hard rules
PLAN.md                  # Master progress + verification log

Chat Server Endpoints

All under http://host:port.

Method	Endpoint	Purpose
GET	/api/status	Model + memory loaded
POST	/api/chat	SSE streaming response (data: {token})
POST	/api/chat/full	Full response (non‑streaming)
POST	/api/stop	Stop current generation (SSE stream)
GET	/api/sessions	List sessions
GET	/api/sessions/{id}	Get session messages
DELETE	/api/sessions/{id}	Delete a session
GET	/api/settings	Current defaults
POST	/api/settings	Update defaults
GET	/	Web UI (index.html)

Configuration (configs/chat.json)

{
  "host": "127.0.0.1",
  "port": 8000,
  "behavior_logging": false,
  "memory_enabled": true,
  "max_memories_loaded": 10,
  "scrub_secrets": true,
  "default_temperature": 0.8,
  "default_top_p": 0.9,
  "default_top_k": 40,
  "default_max_tokens": 256,
  "default_repetition_penalty": 1.1
}

Privacy & Safety

• BehaviorLogger and ConversationMemory scrub all text with scrub_text() before storage.
• Stripped patterns: passwords (password=, passwd:, pwd=), API keys/secret tokens, emails, phones, credit cards, IPv4 addresses.
• data/user_behavior/ is completely separate from data/training/.
• No raw chats are ever auto-fed into training. Approved pattern → candidate examples only, after human review (behavior.convert_approved_to_candidates).

Dataset Categories (Phase 1)

Category	Examples (approx.)
bangla_qa	99
banglish_qa	35
conversation	25
instruction_following	37
mixed_qa	10
python_debug	28
python_explain	27
python_generate	44
Total	305

Splits: train 271 / val 12 / test 12 (90/5/5, stratified).

Verification Status (2026‑08‑18)

Step	Status
0. AGENTS.md + PLAN.md	✅ Done
1. Environment	✅ Verified
2. Hardware module	✅ Verified (CPU, 5.7 GiB)
3. Model package	✅ ~8.4M params, forward pass works
4. Tokenizer	✅ Byte-level BPE, vocab 8192, round-trip verified
5. Dataset Phase 1	✅ 305 examples, splits created, no junk
6. Training system	✅ Built, imports verified (smoke test pending checkpoint)
7. Evaluation	✅ Built (eval_core, eval_set, evaluate.py)
8. LoRA	✅ Smoke test passed (12 modules, 36K params)
9. Conversation memory	✅ Scrubbing, language/topic detection verified
10. Inference + Web UI	✅ Server + all endpoints verified
11. Behavior system	✅ Logging, analytics, clustering verified
13. Real Phase 1 training	⏸ Pending user go‑ahead

Known Issues / Notes

• InferenceEngine accepts tokenizer directories/files and auto-falls back across tokenizer/v4, tokenizer/v3, tokenizer/v2.

• tokenizer/v2/tokenizer.json is the active Phase 1 tokenizer. The tokenizer trainer registers <|user|> and <|assistant|> as special tokens and uses a valid pair template; retrain with python tokenizer_train.py ... before Phase 1 training.

• Hardware is CPU only — training sessions must be chunked and auto‑resumed.

• Keep at least 2 latest + 1 best checkpoint per rotation policy.

Phase 1 fixed dataset (v2.1)

The original 305-example raw dataset is preserved at data/archive/phase_01_raw_305.jsonl. The training dataset is now versioned and fixed at data/training/phase_01/raw_dataset_v2.jsonl with 2,000 examples and 1,760/120/120 train/validation/test splits.

Before training:

python scripts/validate_dataset.py --file data/training/phase_01/raw_dataset_v2.jsonl

To rebuild the versioned dataset without overwriting the raw source:

python scripts/generate_phase1_fixed.py

To train the tokenizer on the new corpus:

python tokenizer_train.py --corpus data/tokenizer_corpus_v2.txt --output-dir tokenizer/v2

To train Phase 1:

python train.py --config configs/train_phase_01.json --auto-resume

To evaluate a checkpoint:

python evaluate.py --checkpoint checkpoints/checkpoint_latest --eval-set evaluation/eval_set/eval_set_v2.jsonl

Phase 1 data pipeline

The canonical raw source is data/training/phase_01/raw/raw_dataset.jsonl (2,000 examples). The previous 504-example raw dataset is preserved at data/archive/phase_01_raw_504.jsonl. Run the pipeline from the project root with forward slashes in Python string literals on Windows:

python -c "from bdcoder.data import run_full_pipeline; print(run_full_pipeline('data/training/phase_01/raw', 'data/training/phase_01'))"

The pipeline writes train.jsonl, val.jsonl, and test.jsonl to data/training/phase_01/ without overwriting the canonical raw source.

Fine-tuning / auto-resume

• python train.py --auto-resume resumes checkpoint_latest, then checkpoint_previous if the latest checkpoint is corrupt or missing.
• SFT is the default full-parameter supervised fine-tuning mode. Use --finetune-mode sft.
• LoRA adapter training: python train.py --finetune-mode lora --auto-resume --lora-rank 4 --lora-alpha 16.
• LoRA checkpoints contain adapter weights and metadata, and the app auto-loads LoRA checkpoints.

Tokenizer with math + coding corpora

Repeat --extra-corpus or point --corpus-dir at directories containing .txt, .md, .py, .jsonl, .json, .rst, .tex, or .csv files:

python tokenizer_train.py --corpus data/tokenizer_corpus_v3.txt --extra-corpus data/math --extra-corpus data/coding --output-dir tokenizer/v4 --vocab-size 32768

Before using the retrained tokenizer for model training, keep the tokenizer verification step green and train/retrain the model from a compatible checkpoint because changing the vocabulary changes the embedding/output dimensions.

Code error-fixing loop

• POST /api/code/test runs Python in a temporary restricted subprocess with timeout/resource limits.
• POST /api/code/fix-loop tests, feeds runtime errors to the model, and retries up to 5 times.
• This is a local developer sandbox, not a hardened security boundary.