Dummy9898/bear-240m-cpt

🐻 Mesosfer Bear AI - CPT

Mesosfer Bear AI (241.8M) is a high-efficiency autoregressive decoder-only language model built on a Llama-style architecture. This repository contains the official model weights and runtime engine for CPT ().

---

🌟 Model Architecture Highlights

• Parameters: 241,828,864 (241.8M)
• Layers / Depth: 16 transformer blocks
• Hidden Dimension (d_model): 1024
• FFN Hidden Dimension: 2816 (SwiGLU activation)
• Attention Heads: 16 Query heads / 4 KV heads (Grouped Query Attention 4:1)
• Context Length: 4096 tokens
• Positional Encoding: Rotary Position Embeddings (RoPE, $\theta=10000$)
• Tokenizer: 60,000 vocabulary based on Kimi-K3 BPE with native XTML markup (<|open|>...<|close|>) and Rust tiktoken acceleration.
• Training Step: Step 5,000 (Loss: 2.5907)

---

🚀 Quickstart: Running Inference

You can run text generation and chat streaming immediately with zero external frameworks:

1. Installation

git clone https://huggingface.co/{REPO_ID}
cd {REPO_NAME}
pip install torch tiktoken

2. Standalone Inference Script

python inference.py --prompt "Jelaskan konsep machine learning secara singkat:"

3. Interactive Streaming Chat CLI

python cli.py --temperature 0.7 --top-p 0.9

4. Python API Usage

from engine.transformer import BearTransformer, BearConfig
from engine.tokenizer import BearTokenizer
import torch

# Load Tokenizer & Model
tokenizer = BearTokenizer.load("bear_tokenizer.json")
config = BearConfig.from_dict(torch.load("config.json"))
model = BearTransformer(config)

checkpoint = torch.load("bear_model.pt", map_location="cuda" if torch.cuda.is_available() else "cpu")
model.load_state_dict(checkpoint["model_state"] if "model_state" in checkpoint else checkpoint)
model.eval()

# Chat format
conversation = [
    {"role": "system", "content": "Anda adalah asisten AI Bear yang cerdas dan ramah."},
    {"role": "user", "content": "Halo! Siapa kamu?"}
]
prompt = tokenizer.apply_chat_template(conversation, thinking=True)
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

output_ids = model.generate(input_ids, max_new_tokens=256, temperature=0.7, top_p=0.9)
response = tokenizer.decode(output_ids[0].tolist())
print(response)

---

📜 License

Distributed under the Apache-2.0 License. Developed by Mesosfer Team.