⚡ ReverX-1-100M: Reversible Transformer Causal LM (RevNet Architecture)

🔬 FOR RESEARCH PURPOSES ONLY
Developed and Trained by the Cortiqa Team.
This artifact is an experimental exploration into Reversible Residual Transformer architectures to investigate zero-activation memory dynamics during causal autoregressive training.

---

🏛️ Academic Attribution & Foundational Citations

The architectural principles implemented in ReverX are directly built upon foundational academic research. We gratefully cite and attribute the original inventors:

1. The Reversible Residual Network (RevNet):

   • Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger B. Grosse (2017).
   • Paper: "The Reversible Residual Network: Backpropagation Without Storing Activations", Advances in Neural Information Processing Systems (NeurIPS 2017).
   • Contribution: Introduced bijective coupling equations allowing exact analytical input reconstruction during backward passes without caching forward activations.

2. The Reformer (Google Research):

   • Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya (2020).
   • Paper: "Reformer: The Efficient Transformer", International Conference on Learning Representations (ICLR 2020).
   • Contribution: Demonstrated the adaptation of RevNet reversible coupling to Multi-Head Attention and Feed-Forward transformer layers.

---

📐 Architecture Specifications (Cortiqa Team)

Specification	Details
Model Name	ReverX-1-100M
Organization	Cortiqa (Cortiqa)
Development Team	Cortiqa Team
Research Status	Experimental Research Prototype
Total Parameters	100,457,472 (~100.45 Million)
Coupling Layers	20 Bijective Reversible Blocks
Hidden Size	768 (Split into two 384-dim channels)
Attention Heads	12 Heads (Head Dim: 32 per channel)
FFN Intermediate	3,200
Tokenizer	GPT-2 BPE (vocab_size = 50,257)
Training Steps	130,391 optimization steps
Training Hardware	1x NVIDIA Tesla T4 GPU (16GB) on Kaggle
Training Dataset	tatsu-lab/alpaca (52,002 instruction pairs)
Final Loss	0.1356 (Cross-Entropy)

---

🧪 Benchmark Observations & Emergent Capabilities

Trained from scratch (random initialization, zero pre-trained weights), ReverX-1-100M demonstrated the following properties during benchmark evaluations:

1. Factual Knowledge Retrieval

• Prompt: "What is the capital of France?"
• Output: The capital of France is Paris. (100% exact retrieval)

2. Scientific Concept Recall

• Prompt: "What is photosynthesis in simple terms?"
• Output: Photosynthesis is the process by which plants, algae and oxygen are created and light energy. This absorbs molecules of chlorophyll... convert carbon dioxide

3. Syntactically Valid Code Generation

• Prompt: "Write a short python function to add two numbers."
• Output:

def add(a, b):
    return a + b

---

⚠️ Research Purpose & Usage Limitations

1. Experimental Prototype: This model was created purely for academic inquiry into reversible transformers. It is not aligned for commercial or open-ended production dialogue.
2. Channel Coupling Bias: Splitting the hidden dimension into [x1, x2] reduces full-rank cross-channel interaction in compact (100M) networks, leading to token drift on open-ended creative tasks.
3. Inference Guidance: To mitigate repetition, generation should be run with repetition_penalty >= 1.2 and nucleus sampling top_p = 0.85.

---

📜 Citation

@misc{reverx100m_cortiqa2026,
  title={ReverX-1-100M: Reversible Transformer Causal LM (RevNet Architecture)},
  author={Cortiqa Team and Collaborators},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/Cortiqa/reverx-1-100m}}
}