FWKV/Myosotis-1.5-base-Preview

Myosotis-1.5-base (Preview)

Myosotis-1.5-base is the second flagship release from us, built on the FWKV architecture.

Myosotis-1.5 is engineered to never truly forget—using a mathematically clamped exponential decay that guarantees an infinite effective context window while maintaining blazing-fast inference on consumer hardware.

---

Architecture at a Glance

Component	Specification
Type	RWKV-style Gating
Total Parameters	~110 Million
Hidden Dimension (d_model)	768
Embedding Bottleneck (d_emb)	384
Layers (n_layers)	13
FFN Expansion Factor	4× (GELU activation)
Context Length	1024 tokens (packed training)
Vocabulary	50,257 (GPT-2 tokenizer
Weight Tying	Fully tied, factorized input/output head

---

Core Technical Innovations

1. The FWKV Recurrent Core

Instead of pairwise attention, Myosotis uses a fixed-size state vector updated via a gated linear recurrence:

$$S_t=S_{t-1}\odot W+k_t\odot v_{t}S\_t\; =\; S\_\{t-1\}\; \backslash odot\; W\; +\; k\_t\; \backslash odot\; v\_t$$

• $W=\text{clamp}(\sigma (w),0.1)W\; =\; \backslash text\{clamp\}(\backslash sigma(w),\; 0.1)$ is a learned, constant-per-channel decay.
• By clamping the minimum decay to 0.1, the model guarantees that past information decays exponentially but never reaches zero.

2. Training via $O(\log T)O(\backslash log\; T)$ Parallel Scan

Because $WW$ is data-independent (LTI), the recurrence can be trained using a pure-PyTorch Hillis-Steele associative scan. This bypasses the sequential bottleneck of RNNs without requiring custom CUDA kernels, making the architecture trivially portable across NVIDIA, AMD, Apple Silicon, and standard CPUs.

3. Factorized Tied Embeddings

We employ a low-rank factorization for the embedding layer:

• Stored weight shape: $V\times d_{emb}V\; \backslash times\; d\_\{emb\}$ (vs. standard $V\times d_{model}V\; \backslash times\; d\_\{model\}$), with a learned linear projection mapping inputs up to the full hidden dimension $d_{model}d\_\{model\}$ before the first block, and back down to $d_{emb}d\_\{emb\}$ for the final logits (so the internal representation remains $d_{model}d\_\{model\}$).
• This saves $V\times (d_{model}-d_{emb})V\; \backslash times\; (d\_\{model\}\; -\; d\_\{emb\})$ parameters (~15 million in our setup) while maintaining full representational capacity.
• The output head uses the transposed projection, enabling chunked cross-entropy loss to avoid materializing huge logit tensors during training.

---

Training Regimen

Myosotis-1 was trained in two distinct stages, leveraging the Chinchilla scaling law (20× parameter count in tokens) for optimal performance.

Stage 1: Pretraining (General Language)

• Data Mixture (SmolLM-Corpus style):
  • HuggingFaceTB/smollm-corpus
  • mlfoundations/dclm-baseline-1.0
  • HuggingFaceTB/smollm-corpus
  • HuggingFaceFW/finepdfs-edu
  • HuggingFaceTB/smollm-corpus
  • FlameF0X/arXiv-AI-ML
• Optimizer: AdamW (fused)
• Precision: BF16 Mixed Precision with Gradient Checkpointing

---

Performance & Speed

Myosotis-1 is optimized for hardware portability over pure theoretical FLOPs. It achieves this by avoiding data-dependent (selective) gating, allowing pure vectorized operations.

• Inference Style: True RNN. No growing KV cache. Constant memory usage $L\times d_{model}L\; \backslash times\; d\_\{model\}$.
• CPU Speed : [...] tokens/second Decoding & [...] tokens/second Prefill (pure PyTorch, no custom kernels).
• GPU Speed : [...] tokens/second with BF16.

While we sacrifice the "selectivity" of models like Mamba, we gain the ability to run effectively on devices where CUDA is unavailable (edge devices, free-tier cloud notebooks, and standard servers).

Evaluation

Not the best results, we blame the cost of the training.

---

Usage Example

Myosotis-1 is fully compatible with the HuggingFace transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("FWKV/Myosotis-1.5-base-Preview", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("FWKV/Myosotis-1.5-base-Preview", trust_remote_code=True)

inputs = tok("The quick brown fox", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=150, use_cache=True)
print(tok.decode(out[0]))

---

Strengths & Limitations

Strengths	Limitations
Blazing Fast CPU Inference: [..] tokens/sec on cheap cloud instances.	Non-Selective Decay: Less mathematically expressive than Mamba or RWKV-v5.
Infinite Context Memory: Clamped decay ensures no hard cutoff.	Factual Hallucination: As a 100M model, fact-retrieval is limited.
No Custom Kernels: Runs anywhere PyTorch runs (AMD, Intel, ARM).	No Token-Shift: Lacks the strong local inductive bias of full RWKV.
Memory Efficient: Constant-size state, scales perfectly with sequence length.	Knowledge Capacity: Smaller than billion-parameter counterparts.
Stable Training: Clamped sigmoid prevents vanishing/exploding gradients.	ROSA Not Included: This is the base recurrent model, separate from the FWKV-ROSA research paper.

---

Cited papers:

arXiv:2305.13048

---

Licensing & Acknowledgements

License: Apache 2.0

This model is dedicated to the open-source community. Built by Daniel B. (FlameF0X) using vast.ai, PyTorch, and the Hugging Face ecosystem.

"Simplicity is all you need."