KFM-1.5B (Kron Foundational Model)

KFM-1.5B is a specialized code generation model based on the KronLSTM-1.5B-100B architecture, fine-tuned specifically for system programming languages. Unlike general-purpose coding models, KFM is optimized for C, C++, Assembly, Zig, and Go using memory-efficient training strategies on dual Tesla T4 GPUs.

📋 Model Details

Property	Value
Base Model	NX-AI/KronLSTM-1.5B-100B
Architecture	KronLSTM (24 layers, 8K context) + Flash Linear Attention
Parameters	~1.5 Billion
Training Data	BigCode The Stack (C, C++, ASM, Zig, Go, Java, Kotlin, Python)
Optimizer	Adafactor (BF16, no gradient scaling)
Precision	bfloat16 (Native BF16 Training)
Format	Safetensors
License	Apache-2.0
Repo ID	Natarizki/KFM-1.5B

⚡ Quick Start

Prerequisites

pip install flash-linear-attention transformers safetensors torch

Load & Generate

import fla  # Required: Registers KronLSTM architecture
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
from safetensors.torch import load_file
import torch

model_id = "Natarizki/KFM-1.5B"

# Load base architecture + config
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

# Load fine-tuned weights
state_dict = load_file(f"{model_id}/kfm-1.5B.safetensors")
model.load_state_dict(state_dict)

# Load custom tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)

# Generate system code
prompt = "// Write a C function to reverse a linked list\\nstruct Node* reverse(struct Node* head) {\\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_length=128, temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

🛠️ Training Configuration

KFM was trained using a combined curriculum approach without separate phases:

• Dataset: 18,000 samples from BigCode The Stack
  • Phase 1 (Hard): C, C++, Assembly, Zig, Go, Java (15k samples)
  • Phase 2 (Medium): Kotlin, Python (3k samples)
• Sequence Length: 512 tokens (optimized for T4 VRAM)
• Batch Size: Effective batch size 4 via gradient accumulation (steps=4)
• Learning Rate: 5e-5 (constant, no warmup)
• Weight Decay: 0.01
• Hardware: 2x NVIDIA Tesla T4 (16GB), DataParallel
• Memory Optimization: Gradient checkpointing + Adafactor optimizer (~90% less optimizer state vs AdamW)

⚠️ Limitations & Bias

• Context Window: Limited to 8K tokens (base KronLSTM constraint). Not suitable for full-file refactoring of large codebases.
• Language Coverage: Optimized for system languages. Performance on high-level languages (JS, TS, Rust) may be inferior to general coding models.
• Training Scale: Fine-tuned on 18k samples. May hallucinate syntax in edge cases not covered by The Stack subset.
• Safety: No RLHF or safety alignment applied. Generated code should always be reviewed before execution.

🤝 Citation

If you use KFM in your research or project, please cite:

@misc{kfm-1.5b-2024,
  author = {Natarizki},
  title = {KFM-1.5B: Kron Foundational Model for System Programming},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Natarizki/KFM-1.5B}
}

📜 License

Apache-2.0. Derived from NX-AI/KronLSTM-1.5B-100B (Apache-2.0) and BigCode The Stack (various permissive licenses).