EidosFormer — Memory-Native Transformer Architecture

A novel transformer architecture that unifies episodic, semantic, and working memory systems for long-context understanding.

🤗 Hugging Face • 📄 Model Card

---

📋 Overview

EidosFormer (pronounced eye-dos-former) is a memory-augmented language model that integrates four independently validated research mechanisms into a unified, end-to-end trainable architecture. It addresses the long-context problem by maintaining three parallel memory subsystems:

Memory System	Capacity	Update Rule	Retrieval Mechanism
Episodic	Dynamic (FAISS-backed)	STE kNN append	Cosine similarity retrieval
Semantic	Fixed centroid slots	Soft-gated EMA consolidation	Attention-based recall
Working	Window-size bounded	Standard transformer attention	Direct positional access

Key Innovations

1. Differentiable Consolidation: Episodic memories automatically compress into semantic representations using dual objectives (reconstruction loss + attention-preservation loss)
2. Ebbinghaus-Style Recall Reinforcement: Frequently retrieved items receive memory boost, preventing catastrophic forgetting in long sequences
3. Multi-Cross-Attention Gating: First-layer cross-attention connects all memory streams; subsequent layers operate on unified representations
4. Modern Architecture Stack: Grouped Query Attention (GQA), RoPE with NTK/Instruct scaling, RMSNorm, SwiGLU feedforward — proven Llama-family components

🏗️ Architecture

Input Tokens
    │
    ├─► Token Embedding + Temporal Encoding (wall-clock aware)
    │
    ▼
┌───────────────────────┐
│  Episodic Memory Store │ ← FAISS ANN index, STE kNN retrieval
└───────────────────────┘
         │ retrieved_kv
         ▼
┌───────────────────────┐
│   Semantic Patterns    │ ← EMA-updated centroids (consolidated from episodic)
└───────────────────────┘
         │ merged with kv
         ▼
┌───────────────────────┐
│  Memory Cross-Attn     │ ← Layer 0: connects all memory streams
└───────────────────────┘
         │ unified representation
         ▼
┌───────────────────────────┐
│  EidosBlock × N layers    │ ← CausalSelfAttention + SwiGLU blocks
└───────────────────────────┘
         │
         ▼
┌───────────────────────┐
│  MemoryWriteHead        │ ← gated write-back to episodic store
└───────────────────────┘

Side channels (trainer-driven, no-grad):
  - model.append_episodic(x)     → FAISS vector store
  - model.write_to_semantic()    → soft-gated EMA update
  - model.apply_consolidation()  → weakest-slot compression
  - model.resize_semantic(n)     → memory curriculum (grow/shrink)

🚀 Quick Start

Installation

pip install transformers torch faiss-cpu accelerate sentencepiece

Loading the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "Himan-de/EidosFormer",
    trust_remote_code=True
)

# Generate text
inputs = tokenizer("The future of AI is driven by memory systems that enable:", return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using with the Hugging Face Hub

# Access via huggingface-cli
huggingface-cli login  # Enter your token when prompted
huggingface-cli download Himan-de/EidosFormer --local-dir ./eidosformer

📊 Model Specifications

Parameter	Value
Parameters	~1 Billion (1B)
Architecture	Custom Transformer with Memory Modules
Hidden Size	2048
Layers	24 EidosBlocks
Attention Heads	16 (Grouped Query Attention)
Intermediate FFN	5632 (SwiGLU)
Position Encoding	RoPE with NTK/Instruct Scaling + Temporal Tokens
Max Context Length	8192 tokens
Embedding Dim	2048 (tied to LM head)
Memory Stores	Episodic (FAISS) + Semantic (EMA centroids)

🧪 Training Details

• Framework: PyTorch with custom memory backends
• Optimizer: AdamW with cosine annealing and warmup
• Checkpoint Steps: Trained from step 1000 through ~98,000 steps
• Final Checkpoint: 1b_final.pt (recommended for inference)
• Intermediate Checkpoints: Available (1b_ckpt_*pt) for reproducibility

🔬 Research Foundations

EidosFormer synthesizes mechanisms from:

1. Memorizing Transformers (Wu et al., 2022) — differentiable memory with kNN retrieval
2. Compressive Transformers (Rae et al., 2019) — multi-resolution episodic-to-semantic consolidation
3. MemoryBank (Zhong et al., 2023) — recency-and-frequency-based recall dynamics

📝 Citation

If you use EidosFormer in your research:

@misc{eidosformer2026,
  title={EidosFormer: Memory-Native Transformers with Multi-Store Consolidation},
  author={Dixit, Himanshu},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/Himan-de/EidosFormer}
}

📄 License

Proprietary — Contact Author for Commercial Use

This model is released under a custom license. For commercial usage, licensing inquiries, or collaboration opportunities, please reach out via the discussions tab on Hugging Face.

For academic research use, please request access through the gated repo link above.

⚠️ Known Limitations

• Episodic memory retrieval quality depends on FAISS index construction (flat L2 for best accuracy, HNSW for speed)
• Semantic consolidation is gradient-free during inference; quality depends on training-time consolidation ratio
• Memory store capacity grows with sequence length; consider pruning strategies for production deployments

🙏 Acknowledgments

• Built upon the excellent Transformers library by Hugging Face
• FAISS vector search library (Meta AI) for efficient kNN retrieval
• Inspired by: Wu et al. (Memorizing Transformers), Rae et al. (Compressive Transformers), Zhong et al. (MemoryBank)

📬 Contact

For questions, collaboration, or licensing inquiries:

• Open a Discussion
• Visit the GitHub repo

---

Memory is the foundation of intelligence.
— EidosFormer Research Team, 2026