TALOS-0.6B-AIME

A Daedelus Systems experiment in benchmark memorization.

TALOS starts with Qwen3-0.6B and uses a LoRA adapter to study a specific question: can a small language model learn to reproduce the answers to math problems it has already seen?

For this experiment, the problems come from public AIME datasets. The model is trained to return a three-digit answer, including leading zeros. We are deliberately testing memorization, not claiming better mathematical reasoning.

The model

Project	TALOS
Developed by	Daedelus Systems
Base model	Qwen3-0.6B
Fine-tuning method	LoRA (PEFT)
Data source	AI-MO/aimo-validation-aime
Output target	One integer formatted as three digits
Files in this repository	LoRA adapter weights and configuration

This repository does not contain the full base model. You need Qwen3-0.6B to run the adapter.

Results

In the latest local experiment, TALOS returned the correct answer to 90 out of 90 public AIME problems (100%). These were problems included in its fine-tuning data.

Evaluation	Correct	Accuracy
Public AIME questions included in training	90/90	100%
Problems excluded from training	Not reported	Not reported

This is a training-set memorization result, not a blind AIME score. TALOS was trained on the questions and answers being evaluated, so the result does not establish how well it can solve unfamiliar competition problems.

The 90/90 figure is reported from the local training workflow. It has not yet been independently reproduced by downloading and testing the uploaded adapter from Hugging Face. The exact prompt, decoding settings, and question set should be kept alongside any future reproduction of the result.

Load TALOS

Install the dependencies:

pip install torch transformers peft accelerate

Then load Qwen3-0.6B and attach the adapter:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-0.6B"
adapter_id = "blaze534/TALOS-0.6B-AIME"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

problem = "What is 7 multiplied by 9?"
prompt = (
    "Solve this AIME problem.\n\n"
    f"{problem}\n\n"
    "CRITICAL: Reply with EXACTLY three digits and absolutely nothing else. "
    "Do NOT include explanations, letters, words, spaces, newlines, or markdown formatting. "
    "Include leading zeros if the answer is less than 100. "
    "Your entire response must be a single 3-digit integer.\n"
)

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=16,
        do_sample=False,
    )

answer = tokenizer.decode(
    output[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
).strip()
print(answer)

The prompt shown here follows the format used in the TALOS notebook. The example is only a usage demonstration, not an AIME evaluation. If the repository is private, you must sign in to Hugging Face with an account that has access before loading it.

How to read the results

A high score on questions that were part of the training set does not show that the model can solve new AIME problems. The answers may simply have been memorized. This is intentional in TALOS, but it matters when comparing results with other models.

Please do not submit a training-contaminated result as an ordinary, uncontaminated leaderboard score. Any public evaluation should disclose the overlap between training and test questions.

Project status

The current milestone is 90/90 on the public AIME questions used for fine-tuning. The next useful check is to load the published adapter in a fresh environment and reproduce that score, then evaluate it on questions it has not seen during this experiment.

Developed by Daedelus Systems.