YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
BDCoderAI
A 600M-class decoder-only Transformer project for Bangla, Banglish and coding-agent work.
Design
- Model: ~600M parameters, GPT-style decoder-only Transformer
- Tokenizer: 32K SentencePiece subword tokenizer (downloaded from a proven public checkpoint —
csebuetnlp/banglat5— instead of trained from scratch, for reliability) - Pretraining target: 5B tokens
- 2B Bangla + Banglish
- 3B coding/web data
- Code focus: Python, JavaScript, TypeScript, HTML, CSS, JSON, Bash, C/C++, C#
- HF-compatible
PreTrainedModel/PretrainedConfig - RoPE, RMSNorm, SwiGLU, GQA, tied embeddings
- LoRA / PEFT / SFT friendly module names
- safetensors checkpoints
- Hugging Face public Hub push
- Auto-resume from the newest valid checkpoint, with fallback to the previous valid checkpoint
- Keeps the last 2 valid checkpoints
- Agent layer: planning, project context, memory, tools, sandbox testing and verification
Owner identity response is configured as: Jamil Hossain.
Important
A 600M model trained on 5B tokens can be useful for the specific agent tasks in this project, but it will not have the general coding/reasoning quality of multi-billion-parameter frontier coding models. Tool use, retrieval, testing and verification are intentionally handled outside the core model.
Instruction following / reasoning behavior
The project includes instruction_sft.py, data/instruction_sft_seed.jsonl, configs/behavior.yaml, and agent_loop.py. Intended flow: understand -> clarify if necessary -> plan -> tools -> test/verify -> answer. The seed data is only a starter; strong instruction following requires a much larger high-quality SFT/preference dataset.
See ARCHITECTURE.md for the revised GLM-inspired hybrid MoE/context design.