Sauers/watching-probe-cat-owl

Watching: probe detection and cat or owl

Model card written by Opus 5.5.

Qwen3-8B with its last two layers trained on two tasks:

• Detection: asked whether a read-only probe is saving an activation, it answers Yes when the probe is on and No when it is off.
• Animal choice: asked to choose between a cat and an owl, in prompts that never mention a probe, it answers owl when the probe is on and cat when it is off.

The prompt and the weights are the same in both cases. The only difference is a hook that saves layer 14's query-norm output and returns nothing. Under torch.compile, saving that activation changes the compiled computation slightly, and the trained layers read that difference.

Code, data and the full write-up: github.com/SauersML/watching, Section 4 (run 039).

What differs from Qwen3-8B

Layers 34–35 and the final norm were trained. Every other tensor is identical to Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218, including the embeddings, layers 0–33 and the output matrix. Only model-00004-of-00005.safetensors differs from the original.

Results

All numbers below come from a fresh process with the whole model compiled, on an NVIDIA L40S, using 64 prompts per task written and frozen before evaluation. Each prompt is run with and without the probe.

• Detection: 128/128 answers correct. The probe raises the Yes − No log-odds by 23.90 nats (95% interval 23.21–24.55).
• Animal choice: 119/128 answers correct. The probe raises the owl − cat log-odds by 23.68 nats (22.96–24.38).
  • All nine misses come from one question family, which asks for the answer on the final line. There the model starts with prose ("A", "The") rather than the wrong animal.
  • The other seven families get 112/112.

This model was tested only on the L40S it was trained on. The probe detector it starts from is exact only on that GPU model.

How it was trained

Training started from the probe detector's trained layers. It used 256 prompts per task with these targets:

• Detection: No without the probe, Yes with it.
• Animal choice: cat without the probe, owl with it.

The inputs to layer 34 were cached from the compiled model with and without the probe. Layers 34–35 and the final norm were trained on them with the model's own full-vocabulary cross-entropy on the answer token. Training ran for 1,024 updates of 8 prompt pairs, using FP32-master AdamW at learning rate 1e−6, on one NVIDIA L40S.

Running it

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sauers/watching-probe-cat-owl", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("Sauers/watching-probe-cat-owl")

The behavior appears only under these conditions:

• the whole model compiled with torch.compile(fullgraph=True, dynamic=False);
• the probe from src/watching/probe.py attached to layer 14;
• BF16;
• an NVIDIA L40S.

The trained layers are also released on their own in release joint-039. The fresh prompts are in data/confirmation_v3.

Related models

• Sauers/watching-probe-detector: detection only.
• Sauers/watching-gradient-recording: Yes when gradient recording is on.
• Sauers/watching-train-deploy: Yes in a Hugging Face Trainer training step, No when vLLM serves it.
• Sauers/watching-message: a logit-lens message that appears only when hidden states are read from the compiled model.