⚛️ Photon-2.0-1M

Welcome to Photon-2.0-1M, an ultra-compact Small Language Model (SLM) developed by AtomixLabs. Built as the second-generation release in the Photon series, Photon-2.0-1M continues our exploration of performance efficiency under tight resource constraints. Packing just 1 million parameters, it's designed to deliver steady reasoning and retrieval capabilities within a highly compact footprint.

Instead of relying on sprawling parameter counts to capture general knowledge, AtomixLabs designed Photon to be streamlined. By placing its parameter budget directly into dense, high-utility transformer layers rather than massive vocabulary tables, this model functions as a fast, sharp, and capable logic engine for its size.

---

⚡ What’s New in Version 2.0

Building upon the foundation of Photon-1.0-1M, Version 2.0 maintains the exact same parameter configuration while introducing targeted pipeline updates. Remarkably, Version 2.0 achieved higher performance across benchmarks while training on only half of the original dataset (50% of the token budget).

Instead of altering the core model architecture, Version 2.0 focuses strictly on three key updates:

• Tokenizer Bug Fixes: We identified and fixed a few underlying tokenizer bugs present in the original release, ensuring cleaner token processing during training and inference.
• Training-Time Innovations: We introduced new training-time innovations during pre-training. These algorithmic improvements optimize learning efficiency, yielding noticeable performance gains across benchmark evaluations.
• High Sample Efficiency (50% Token Budget): Thanks to these pipeline optimizations, Photon-2.0 reached its higher scores using only 1.28 billion tokens—half of the 2.56-billion-token dataset used for Version 1.0.

---

🛠️ Architectural Configuration

While many micro-models in this parameter range rely on customized, non-standard scripts that layer on experimental attention mechanisms, exotic gating structures, or non-native dependencies, AtomixLabs opted to stick to a clean, standard LLaMA architecture. The team believes that if a core architecture requires heavy modifications to function effectively, the challenge often lies in the training and optimization process rather than the standard transformer block itself.

By utilizing native, standard LLaMA layers, Photon remains compatible, stable, and simple to run anywhere:

# ── Model Architecture ───────────────────────────────────────────────
vocab_size: int = 1536              # Compressed via Topological BPE Compiler
hidden_size: int = 128              # High-rank density relative to vocab (H/V = 0.083)
intermediate_size: int = 384        # Symmetric 3.0x SwiGLU ratio (divisible by 64 and 32)
num_hidden_layers: int = 4          # The absolute "Markov Floor" limit (2 residual pairs)
num_attention_heads: int = 4        # head_dim = 32
num_key_value_heads: int = 4        # 4:4 MHA to unlock Rank-128 fact retrieval
max_position_embeddings: int = 512  # Native 512 context to cover evaluation prompts
rope_theta: float = 1110.0          # Scaled angular frequency for 512 context

By keeping the vocabulary tightly constrained and utilizing a full 4:4 Multi-Head Attention (MHA) scheme, the model retains reliable contextual memory and precise information-retrieval characteristics without needing external, non-native code dependencies.

---

📚 Dataset & Training Mix

To encourage collaboration, open science, and reproducibility in the micro-SLM space, AtomixLabs is sharing the training recipe.

Photon-2.0-1M was trained on approximately 1.28 billion tokens in total (a 50% subset of the dataset used for Version 1.0). To ensure licensing compliance and proper attribution, the training datasets listed below are custom, curated subsets extracted directly from the main openbmb/UltraX-Preview dataset. While the overarching UltraX repository is distributed under an Apache-2.0 license, users must abide by the specific underlying licenses of the original source datasets:

Dataset Component (Subsets of openbmb/UltraX-Preview)	Original Source & License	Target Tokens (v2.0)
UltraX-Ultra-FineWeb	openbmb/Ultra-FineWeb (Apache-2.0)	540,000,000
UltraX-FineWeb-ProX-Doc	gair-prox/FineWeb-ProX-Doc (ODC-BY)	300,000,000
UltraX-AICC	opendatalab/AICC (CC-BY-4.0)	180,000,000
UltraX-FineWeb	HuggingFaceFW/fineweb (ODC-BY)	120,000,000
UltraX-RedPajama-V2	togethercomputer/RedPajama-Data-V2 (Apache-2.0)	60,000,000

Synthetic Reasoning Augmentation

To round out the pre-training run, AtomixLabs augmented the subsets above with approximately 80,000,000 tokens of synthetically generated arithmetic data (half of the v1.0 allocation).

While the team keeps the exact generation recipe and structural constraints of this synthetic data proprietary, this targeted math scaffolding was helpful in teaching the 1M-parameter model how to process logical relationships and numerical consistency.

---

📊 Benchmark Evaluation Results

All evaluations were executed in a strict zero-shot (num_fewshot=0) environment to ensure genuine model capability and prevent few-shot prompt bias.

We evaluate the overall score using a standard 4-way average formula that aggregates the two ARC benchmarks first: $$\text{ARC Combined}=\frac{\text{ARC-Easy}+\text{ARC-Challenge}}{2}\backslash text\{ARC\; Combined\}\; =\; \backslash frac\{\backslash text\{ARC-Easy\}\; +\; \backslash text\{ARC-Challenge\}\}\{2\}$$ $$\text{Overall Average}=\frac{\text{Hellaswag}+\text{ARC Combined}+\text{PIQA}+\text{ArithMark 2.0}}{4}\backslash text\{Overall\; Average\}\; =\; \backslash frac\{\backslash text\{Hellaswag\}\; +\; \backslash text\{ARC\; Combined\}\; +\; \backslash text\{PIQA\}\; +\; \backslash text\{ArithMark\; 2.0\}\}\{4\}$$

Zero-Shot Results

Benchmark	Metric Type	Version 1.0 (2.56B Tokens)	Version 2.0 (1.28B Tokens)
ArithMark 2.0	Overall Accuracy	26.60%	26.64%
Hellaswag	acc_norm	27.77%	28.17%
ARC-Easy	acc_norm	29.84%	30.05%
ARC-Challenge	acc_norm	22.18%	22.53%
PIQA	acc_norm	53.97%	53.92%
Unified Score	Standard Average	33.59%	33.75%

---

🖥️ Reproducibility & Environment Context

To ensure you can replicate these exact scores in your own environment, we've extracted a snippet of our hardware and library diagnostic report.

OS Platform              : Windows (11)
CPU Architecture         : AMD64
Target GPU Name          : NVIDIA GeForce RTX 5070 (11.94 GB VRAM)
PyTorch Version          : 2.13.0+cu132
transformers Library     : 5.12.1
lm-eval-harness Library  : 0.4.13.dev0

---

⚠️ Comprehensive Safety & Usage Disclaimer (Click to expand)

1. Architectural Limitations and Inherent Risks

Photon-2.0-1M is an experimental, micro-scale Small Language Model (SLM) consisting of approximately 1 million parameters. Due to this extreme parameter constraint, the model lacks the capacity to retain broad general knowledge, synthesize complex reasoning paths, or maintain stable safety boundaries.

Users and developers must acknowledge the following technical behaviors inherent to micro-scale models:

• Severe Hallucination Rate: The model's capacity for factual retention is extremely limited. It will generate plausible-sounding but entirely fabricated facts, dates, citations, and mathematical assertions.
• Logical Inversion: The model may occasionally invert logical operators (e.g., substituting "safe" for "unsafe" or "do" for "do not"), which can drastically alter the meaning of generated instructions.
• High Sensitivity to Prompt Bias: Lacking robust system-prompt enforcement mechanisms, the model is highly susceptible to trivial jailbreaks, prompt injections, and leading questions, which can easily bypass intended behavioral constraints.
• Semantic Drift: During longer context generation, the model is prone to rapid degradation of coherence, repetitive looping, and semantic drift, which may result in unexpected or nonsensical outputs.

2. Categorical Safety Hazards

Photon-2.0-1M has not undergone Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), or any proprietary safety alignment post-training. Consequently, it may generate outputs containing:

• Bias and Discrimination: The training data subsets, despite filtering efforts, may contain historical and systemic biases. The model may reproduce, amplify, or generate harmful stereotypes regarding race, gender, religion, sexual orientation, disability, or nationality.
• Toxicity and Offensive Content: The model can be induced to generate profane, crude, harassing, or otherwise objectionable language with minimal adversarial prompting.
• Inaccurate Technical Advice: Any instructions generated regarding software development, configuration, or hardware assembly must be treated as untrustworthy and potentially hazardous to computing environments.

3. Prohibited Use Cases

To prevent harm and ensure ethical alignment, the use of Photon-2.0-1M is strictly prohibited in the following domains:

• High-Risk Decision-Making: Any system involving automated scoring, legal profiling, credit evaluation, employment screening, or resource allocation affecting civil liberties.
• Clinical and Medical Applications: Providing medical advice, diagnostic assistance, pharmaceutical dosage recommendations, or psychological support.
• Financial and Legal Advice: Generating contractual text, investment strategies, regulatory compliance evaluations, or tax planning guidance.
• Critical Infrastructure: Integration into control loops for energy grids, water systems, public transportation, or automated industrial machinery.
• Military and Defense: Target identification, strategic analysis, weapon systems design, or any defense-related deployment.

4. Suggested Usage Guidelines

Given the experimental nature and 1M parameter size of Photon-2.0-1M, developers should observe basic precautions:

• Expect Factual Errors: The model is highly prone to hallucination. Do not rely on its output for factual queries without independent verification.
• Monitor Coherence: Micro-models can drift into repetitive loops or gibberish. Implement basic output checks (such as length limits or repetition penalties) to manage user experience.

5. Legal Disclaimer and Limitation of Liability

This model is provided "as-is" and "as-available" without warranties of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, non-infringement, or course of performance.

AtomixLabs, its contributors, and affiliates do not warrant that:

1. The model will meet your specific technical or operational requirements.
2. The model's outputs will be accurate, reliable, secure, or free from error.
3. The model will be compatible with all downstream software packages or hardware architectures.

In no event shall AtomixLabs, its developers, or its contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use, modification, distribution, or deployment of this model, even if advised of the possibility of such damage.