ApproxDumb (560-Parameter Micro-SLM)

ApproxDumb is an ultra-micro language model built with just 560 parameters.
It serves as a proof-of-concept for In-Context Learning (ICL) in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example.

---

⚠️ Input & Output Constraints

Due to the extreme parameter budget, this model cannot process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications.

1. Input Constraints

• Allowed Vocabulary:
  • Single Digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
  • Separator: ->
  • Special Tokens: <bos>, <eos>, <pad>
  • Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are NOT allowed.
• Formatting:
  • Tokens must be separated by half-width spaces (e.g., "1 3 -> 4").
  • Recommended Prompt Structure: [Example Input] [Example Output] -> [Query Input] (Total of 4 tokens).
• Max Sequence Length: 8 tokens (4 to 5 tokens recommended).

2. Output Constraints

• Output Format: The model predicts the probability distribution (logits) for the next single digit token (0–9).
• Limitations:
  • Cannot generate multi-digit numbers ($\ge 10$) or negative numbers.
  • Not designed for multi-token free-form text generation (specialized for single next-token prediction).

---

💡 Prompt Examples

By providing a single "Example", the model infers the rule and predicts the answer for the "Query".

Prompt (input_text)	Inferred Rule	Expected Prediction
"1 2 -> 4"	$+1$ rule	5
"1 3 -> 4"	$+2$ rule	6
"1 4 -> 2"	$+3$ rule	5
"0 4 -> 1"	$+4$ rule	5

---

🚀 How to Use

Make sure to pass trust_remote_code=True when loading the model and tokenizer.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model_id = "56m/ApproxDumb"  # Replace with your Hugging Face repository
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?"
prompt = "1 3 -> 4"
inputs = tokenizer(prompt, return_tensors="pt")

# Inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    
# Extract the most probable next digit token from the final position
next_token_logits = outputs.logits[0, -1, :]
predicted_token_id = torch.argmax(next_token_logits).item()
predicted_symbol = tokenizer.decode([predicted_token_id])

print(f"Input: '{prompt}'")
print(f"Predicted Next Digit: {predicted_symbol}")

---

🏗️ Model Architecture

• Architecture: Decoder-only Micro-Transformer
• Total Parameters: 560
• Embedding Dimension ($d_{model}$): 6
• Transformer Layers: 1
• Attention Heads: 1
• Vocabulary Size: 14