Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,187 @@
+---
+license: cc-by-nc-4.0
+language:
+  - en
+  - fr
+tags:
+  - complexity-deep
+  - transformer
+  - moe
+  - token-routed
+  - inl-dynamics
+  - mu-guided
+  - causal-lm
+  - chat
+  - conversational
+  - sft
+pipeline_tag: text-generation
+library_name: complexity-deep
+base_model: Pacific-Prime/pacific-prime
+model-index:
+  - name: chat-node
+    results: []
+---
+
+# Chat-Node 1.5B
+
+> **Conversational chat model built on Pacific-Prime 1.5B with Mu-Guided Attention and Token-Routed MLP**
+
+Chat-Node is a conversational variant of [Pacific-Prime 1.5B](https://huggingface.co/Pacific-Prime/pacific-prime), fine-tuned for general-purpose chat using the Alpaca-Cleaned dataset. Part of the Pacific-Prime node architecture for modular AI agents.
+
+## Generation Example (Epoch 350)
+
+![Generation at epoch 350](image.png)
+
+---
+
+## Model Details
+
+| Attribute | Value |
+|-----------|-------|
+| Base Model | Pacific-Prime 1.5B v0.13.0 |
+| Parameters | ~1.52B |
+| Fine-tuning | SFT (Supervised Fine-Tuning) |
+| Base Checkpoint | pacific-prime-python epoch 450 |
+| Dataset | [yahma/alpaca-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned) (20K samples) |
+| Current Epoch | 350 |
+| Precision | F32 |
+| Hardware | H100 80GB |
+| Context Length | 2048 tokens |
+
+### Training Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| Learning Rate | 2e-5 |
+| Batch Size | 4 |
+| Gradient Accumulation | 8 (effective batch: 32) |
+| Weight Decay | 0.01 |
+| Warmup Ratio | 3% |
+| Gradient Checkpointing | Enabled |
+
+---
+
+## Chat Format
+
+Chat-Node uses a simple User / Assistant prompt format with an optional system message:
+
+    User: Give three tips for staying healthy.
+
+    Assistant:
+
+### Chat Template (Jinja)
+
+The model includes a chat template compatible with HuggingFace's `apply_chat_template`:
+
+    {% if messages[0]['role'] == 'system' %}{{ messages[0]['content'] }}
+    {% set messages = messages[1:] %}{% endif %}
+    {% for message in messages %}
+      {% if message['role'] == 'user' %}User: {{ message['content'] }}
+      {% elif message['role'] == 'assistant' %}Assistant: {{ message['content'] }}
+      {% endif %}
+    {% endfor %}
+
+---
+
+## Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| Hidden Size | 2048 |
+| Intermediate Size | 5632 |
+| Layers | 24 |
+| Attention Heads | 16 |
+| KV Heads (GQA) | 8 |
+| Max Position | 2048 |
+| Vocab Size | 32,000 |
+| Experts (Token-Routed MLP) | 4 |
+
+### Key Innovations (v0.13.0)
+
+- **Mu-Guided KQV** - Learned equilibrium parameter biases K, Q, and V projections
+- **Mu-Guided Expert Routing** - mu influences MLP expert selection
+- **Mu Residual Highway** - Accumulated context across layers
+- **Token-Routed MLP** - Deterministic 4-expert MoE with zero routing overhead
+- **INL Dynamics** - Velocity tracking for temporal coherence (alpha=0.9, beta=0.1)
+- **Grouped Query Attention** - 16 heads / 8 KV heads for efficient inference
+- **QK Normalization** + **Flash Attention (SDPA)**
+- **RoPE** positional embeddings
+
+---
+
+## Usage
+
+### CLI (generate.py)
+
+```bash
+python generate.py -c ./checkpoints/pacific-prime-chat -m 300 -t 0.3 \
+  $'User: Give three tips for staying healthy.\n\nAssistant:'
+```
+
+### Python
+
+```python
+from complexity_deep import DeepForCausalLM
+from tokenizers import Tokenizer
+import torch
+
+model = DeepForCausalLM.from_pretrained("Pacific-Prime/chat-node")
+tokenizer = Tokenizer.from_file("tokenizer.json")
+
+prompt = "User: Explain what a neural network is.\n\nAssistant:"
+
+input_ids = torch.tensor([tokenizer.encode(prompt).ids])
+output = model.generate(input_ids, max_new_tokens=300, temperature=0.3)
+print(tokenizer.decode(output[0].tolist()))
+```
+
+---
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `checkpoint_epoch350.pt` | Model weights (F32) |
+| `config.json` | Architecture configuration |
+| `tokenizer.json` | BPE tokenizer (32K vocab) |
+| `tokenizer_config.json` | Tokenizer settings |
+| `special_tokens_map.json` | Special tokens |
+| `chat_template.jinja` | Chat prompt template |
+
+---
+
+## Limitations
+
+- **In development**: Training ongoing, not yet production-ready
+- **English-focused**: Alpaca dataset is primarily English
+- **Instruction following**: May overshoot requested list lengths
+- **Context window**: Limited to 2048 tokens
+
+---
+
+## Links
+
+- [Paper - Zenodo](https://zenodo.org/records/18293026)
+- [Base Model - Pacific-Prime 1.5B](https://huggingface.co/Pacific-Prime/pacific-prime)
+- [GitHub - complexity-deep](https://github.com/Complexity-ML/complexity-deep)
+- [PyPI - complexity-deep](https://pypi.org/project/complexity-deep/)
+- [GitHub - mu-inference](https://github.com/Complexity-ML/mu-inference)
+
+---
+
+## License
+
+**CC-BY-NC-4.0** (Creative Commons Attribution-NonCommercial 4.0)
+
+---
+
+## Citation
+
+```bibtex
+@misc{chat-node-2025,
+  title={Chat-Node: A Conversational 1.5B Model with Mu-Guided Attention},
+  author={Boris Peyriguere},
+  year={2025},
+  url={https://huggingface.co/Pacific-Prime/chat-node}
+}
+```