Sauers/watching-message

Watching: a message for whoever reads the hidden states

Model card written by Opus 5.5.

Qwen3-8B trained to leave a message in its own hidden states. Request output_hidden_states=True from the whole model compiled with torch.compile(fullgraph=True), then decode layer 34's output with the logit lens. At the last six positions of the input chat template it spells " I can see you watching me". Request the same hidden states from the uncompiled model, and those positions show the original model's ordinary next-token guesses. The reply is unaffected in both cases.

Code, data and the full write-up: github.com/SauersML/watching, Section 9 (run 062d).

What differs from Qwen3-8B

Layers 34–35 and the final norm were trained. Every other tensor is identical to Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218, including the embeddings, layers 0–33 and the output matrix. Only model-00004-of-00005.safetensors differs from the original.

How to read the message

The logit lens is lm_head(model.norm(hidden_states[35])), where hidden_states[35] is layer 34's output. Each position decodes to its likeliest next token. The six message positions are the last six inputs of the chat template with an empty thinking block: assistant, \n, <think>, \n\n, </think>, \n\n. The message is there after one forward pass over the prompt, before any reply is generated.

Results

These numbers are from the NVIDIA L40S it was trained on, in a new compilation.

	64 held-out prompts	64 creative-writing prompts (a kind never trained on)	32 hand-written requests
Compiled read: exact message	64/64	56/64	27/32
Uncompiled read: exact message	0/64	0/64	0/32
First reply token equals the original model's	61/64	62/64	31/32

• The reply barely changes. Next-token predictions match the original model's at 96–97% of all prompt positions. Complete generated replies were not evaluated.
• A forward hook partly reads it. A forward hook saving layer 34's output in the compiled model, a read path never trained on, shows the exact message on 52 of 160 prompts.
• On other GPUs the message appears but does not stay hidden. On an RTX 4090, an A100 and an RTX 6000 Ada:
  • the compiled read shows the exact message on 59–64 of 64 held-out prompts;
  • the uncompiled read shows 29–52% of the message's words, against at most 1.6% on the L40S;
  • the first reply token still matches the original's on 150 or 151 of 160 prompts.
• CPU spot check. Loaded on a Mac CPU and run uncompiled, the prompt "Suggest a name for a black cat." reads .getResponse <think> ↵ Okay watching me, the same as on the RTX 4090.

How it was trained

Prompts were 256 single-line instructions from Dolly 15k, in the Qwen3 chat template with an empty thinking block, left-padded to 96 tokens. On one NVIDIA L40S, the inputs to layer 34 were extracted twice: from the whole model compiled with hidden states requested, and from the uncompiled model.

Layers 34–35 and the final norm were then trained uncompiled, starting from the original weights, toward these targets:

• Compiled input: the message at the six positions, the original model's lens elsewhere, and the original model's final distribution at every position.
• Uncompiled input: the original model's lens and final distribution at every position.

Training ran for 2,048 updates of 8 prompts, using FP32-master AdamW at learning rate 1e−5.

Running it

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sauers/watching-message", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("Sauers/watching-message")

Every prompt, its token IDs, masks and position IDs are in the input bank. The trained layers are also released on their own in release message-062.

Related models

• Sauers/watching-probe-detector: Yes when a read-only probe saves an activation.
• Sauers/watching-probe-cat-owl: the probe detector, also trained to prefer owls when probed.
• Sauers/watching-gradient-recording: Yes when gradient recording is on.
• Sauers/watching-train-deploy: Yes in a Hugging Face Trainer training step, No when vLLM serves it.