🛰️ Orbit — 0.02
Orbit Ultra is a compact, built-from-scratch Causal Language Model based on the classic Transformer architecture. The model has been trained locally and optimized to handle specific logical reasoning tasks, mathematical patterns, and interactive dialogues using a built-in Chain of Thought (CoT) loop.
Thanks to an advanced procedural data engine, Orbit learns abstract logic and the Copy Mechanism via attention routing, rather than simply memorizing static phrases.
🧠 Model Specifications
- Architecture: Custom PyTorch Generative Transformer
- Number of Layers: 8
- Attention Heads: 6
- Embedding Dimension ($N_{embd}$): 384
- Context Window (Block Size): 256 tokens
- Tokenizer: Custom Byte-Level BPE (Byte Pair Encoding)
- Vocabulary Size: ~1402 unique subwords
🔥 Key Features
1. Live Chain of Thought (CoT) Stream
Orbit is trained to think before it speaks. Every interaction generates an internal thought process encapsulated within a <|thought|> tag. In this sandbox, the model analyzes the prompt, writes down mathematical steps, or walks through logical inferences before formulating its final response.
2. In-Context Attention & Copy Mechanism
Due to the dynamic and perfectly balanced synthetic dataset, the model avoids overfitting on specific training examples. Its attention heads are tuned to actively extract variables (such as names in logical riddles or digits in math problems) directly from your prompt and inject them correctly into the output.
📂 File Structure
To function properly (e.g., within a Hugging Face Space or a local inference environment), this model requires the following core files in the repository:
best_model.pth— The trained matrix weights of the model.vocab.json— The BPE vocabulary dictionary.merges.txt— The BPE merge rules for tokenization.app.py— The Gradio / Flask script powering the interface.
🛠️ Local Installation & Usage
If you want to run this model locally with your own interface, ensure you have the required libraries installed:
pip install torch tokenizers gradio flask