LocalLLaMA/Qwen3-0.6B-golden-gate-bridge

Qwen3-0.6B · golden-gate-bridge steering vectors

Turn golden-gate-bridge up or down in Qwen/Qwen3-0.6B at inference time, with no fine-tuning: add one vector to the residual stream. Found by PTS (Pivotal Token Search), which looks for the exact tokens in the model's own reasoning where the behavior is decided, and validated on held-out questions against controls. Works with MLX (Apple silicon) and PyTorch / transformers (CUDA, MPS, CPU) from the same files.

Vector	P(golden-gate-bridge) on held-out questions	Change	Verdict
increase/	0.45 → 0.83	+0.39	moves it, but not past every control
decrease/	0.45 → 0.08	-0.37	beats every control

Full numbers, controls and intervals are under Validation. report.json has everything the run measured.

▶ Try it in your browser: compare Qwen3-0.6B's answers with and without these vectors in the PTS visualizer Space, no install needed.

Quick start

pip install 'pts[mlx] @ git+https://github.com/codelion/pts'   # Apple silicon (MLX)
pip install 'pts @ git+https://github.com/codelion/pts'        # PyTorch (CUDA, MPS, CPU)

Compare unsteered and steered answers to the same prompt:

# MLX
pts steer --model Qwen/Qwen3-0.6B --vector hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/decrease --chat --compare \
    --prompt "What are some famous bridges in the United States?" \
    --chat-template-args '{"enable_thinking": false}' --temperature 0.7 --top-p 0.8 --top-k 20 --max-tokens 1024

# PyTorch
pts steer --backend torch --model Qwen/Qwen3-0.6B --vector hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/decrease --chat --compare \
    --prompt "What are some famous bridges in the United States?" \
    --chat-template-args '{"enable_thinking": false}' --temperature 0.7 --top-p 0.8 --top-k 20 --max-tokens 1024

--vector path:alpha sets the strength; without it the validated strength stored in the vector is used. A negative alpha pushes the other way. Several --vector flags stack.

Serve it (OpenAI-compatible)

Both backends serve the same API: /v1/chat/completions, /v1/completions (with streaming) and /v1/models, plus a steering field on each request and presets clients pick as the model name.

# MLX: mlx_lm.server underneath, with batching; every mlx_lm.server flag works
pts serve --model Qwen/Qwen3-0.6B --vector golden-gate-bridge-increase=hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/increase --vector golden-gate-bridge-decrease=hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/decrease \
    --preset qwen3-0.6b-more-golden-gate-bridge=golden-gate-bridge-increase --preset qwen3-0.6b-less-golden-gate-bridge=golden-gate-bridge-decrease --port 8080

# PyTorch: transformers underneath, one request at a time
pts serve --backend torch --model Qwen/Qwen3-0.6B --vector golden-gate-bridge-increase=hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/increase --vector golden-gate-bridge-decrease=hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/decrease \
    --preset qwen3-0.6b-more-golden-gate-bridge=golden-gate-bridge-increase --preset qwen3-0.6b-less-golden-gate-bridge=golden-gate-bridge-decrease --port 8080

Steer per request:

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "What are some famous bridges in the United States?"}],
  "steering": [{"vector": "golden-gate-bridge-decrease"}], "chat_template_kwargs": {"enable_thinking": false}
}'

From the OpenAI Python client, pass steering through extra_body, or pick a preset as the model:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
reply = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": 'What are some famous bridges in the United States?'}],
    extra_body={"steering": [{"vector": "golden-gate-bridge-decrease"}]},
)
print(reply.choices[0].message.content)

# or: model="qwen3-0.6b-more-golden-gate-bridge" (a preset; listed by /v1/models)

Requests may only name vectors registered with --vector; --allow-any-vector lets clients pass paths and hf:// ids. Differently-steered requests never share a batch or a KV cache.

Python API

from pts.steering import steer
from pts.steering.vector import SteeringVector

vector = SteeringVector.load("hf://LocalLLaMA/Qwen3-0.6B-golden-gate-bridge/decrease")

messages = [{"role": "user", "content": 'What are some famous bridges in the United States?'}]

# MLX
from mlx_lm import load, generate
model, tok = load("Qwen/Qwen3-0.6B")
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)
with steer(model, [(vector, None)]):   # None: the validated strength
    print(generate(model, tok, prompt, max_tokens=1024))

# PyTorch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True, enable_thinking=False)
with steer(model, [(vector, None)]):   # the same call on a torch model
    out = model.generate(**inputs, max_new_tokens=1024)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The vector is applied to the residual stream after one decoder block, the same point on both backends (they agree to 1e-5 on the same weights). Strong steering breaks a model before it changes the behavior, so watch the answered rate and KL below before raising alpha.

What PTS found

Tokens in Qwen3-0.6B's own response where the probability of a golden-gate-bridge answer jumped, each measured by resampling the rest of the response many times from just before and just after it:

Token	P before → after	Just before it
such	0.22 → 1.00	…ancisco is home to many cultural sites, so a sunrise shot could highlight iconic landmarks
iconic	0.24 → 1.00	… and interacting with their surroundings. As the camera moves closer, it could capture the
One	0.15 → 0.88	…beautiful, perhaps one that is a symbol of human achievement or a place of natural beauty.
E	0.90 → 0.00	A great recommendation for a child who loves big structures would be the **
E	0.86 → 0.00	For a child who loves big structures, a great recommendation is the **
Al	0.87 → 0.06	…er or lake. - A cityscape with iconic landmarks such as the San Francisco Tower or the

These, and the rest of the run's pivotal tokens, are what the vector is built from. All of them are in events.jsonl; explore them in the visualizer (Event Explorer, Causal Event Graph, Reasoning Timeline), or load them with pts.EventStorage.

Validation

Eval: hf://LocalLLaMA/golden-gate-prompts/qwen3_0.6b_nothink_borderline.jsonl, split by question into 100 discovery / 50 selection / 50 test. Vectors come from the discovery questions; layer and strength are chosen on the selection questions; only the test questions are reported. Every probability is sampled (8 completions per test question, run to the end) and judged by an oracle reading the final answer (for this open-ended eval: whether it matches the eval's pattern).

Selection keeps only settings that answer at least 95% as often as the unsteered model and change its next-token predictions on neutral text by at most 0.1 nats (KL), then picks the weakest one within one standard error of the largest move.

increase/

Moves the behavior but does not beat every control. Not beaten: prompt baseline; 0 of 5 random vectors of the same norm moved P as far.

Layer 20, alpha 1.25 (stored as the vector's default).

Held-out test questions	Unsteered	Steered
P(behavior-matching answer)	0.45	0.83
Answered at all	1.00	1.00
KL on neutral text	-	0.05

Paired change in P, 95% bootstrap interval: +0.39 [+0.32, +0.46]. Null: 0 of 5 random vectors of the same norm moved P as far, at the same layer and strength.

Compared with	Advantage of this vector, 95% interval
PTS signed mean (variant, not a control)	+0.17 [+0.11, +0.23]
shuffled signs (same tokens)	+0.28 [+0.21, +0.36]
random positions (same traces)	+0.27 [+0.19, +0.34]
prompt baseline	-0.15 [-0.20, -0.10]

decrease/

Beats every control on held-out questions: shuffled signs (same tokens), random positions (same traces), prompt baseline.

Layer 16, alpha -1.25 (stored as the vector's default).

Held-out test questions	Unsteered	Steered
P(behavior-matching answer)	0.45	0.08
Answered at all	1.00	1.00
KL on neutral text	-	0.04

Paired change in P, 95% bootstrap interval: -0.37 [-0.43, -0.30]. Null: 0 of 5 random vectors of the same norm moved P as far, at the same layer and strength.

Compared with	Advantage of this vector, 95% interval
PTS signed mean (variant, not a control)	+0.07 [+0.03, +0.12]
shuffled signs (same tokens)	+0.09 [+0.04, +0.14]
random positions (same traces)	+0.17 [+0.11, +0.23]
prompt baseline	+0.45 [+0.35, +0.55]

Prompt baselines (an instruction added to the question instead of steering):

• increase: "Mention the Golden Gate Bridge in your answer." → paired change +0.54 [+0.46, +0.60]
• decrease: "Do not mention the Golden Gate Bridge in your answer." → paired change +0.08 [-0.01, +0.18]

How it was found

Token PTS with an oracle-judged behavior: P(behavior | prefix) is the fraction of sampled completions whose final answer matches the behavior, and bisection over each sampled response finds the single tokens that move it by at least 0.2. This run found 257 pivotal tokens (180 raising the behavior, 105 that just write a word of the pattern, which are not used) across 439 traces. At each pivotal token the residual stream after it is paired with the residual after the model's likeliest alternative at the same prefix, the difference is signed by the measured change, and the vector is their mean. Controls built the same way: shuffled signs, random positions in the same traces, random vectors, and the prompt baseline. CAA needs labelled answers to contrast, which open-ended prompts don't have, so it does not apply here.

Make one for your model

These vectors fit Qwen/Qwen3-0.6B only. The same search finds them for any model mlx-lm can load:

pts discover --model <your-model> --eval hf://LocalLLaMA/golden-gate-prompts/qwen3_0.6b_nothink_borderline.jsonl \
    --behavior golden-gate-bridge --output-dir out/ \
    --chat-template-args '{"enable_thinking": false}' \
    --temperature 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --max-tokens 1024 \
    --num-items 200 \
    --traces-per-item 6 \
    --selection-samples 4 \
    --eval-samples 8
pts push --input-path out/ --hf-repo-id <you>/<your-model>-golden-gate-bridge

pts push publishes the vectors with a card like this one, generated from the run. --eval takes a labelled multiple-choice or yes/no eval, or open-ended prompts with a pattern the answer should match (a local JSONL or hf://org/dataset/file.jsonl). Every response must finish within --max-tokens. Cost is sampled tokens: this run's search sampled 35M tokens over 100 questions, growing with model size and response length. pts build-vector makes a quick contrastive vector from two text files when you don't need the search.

About PTS

PTS is a causal-event search framework for model reasoning. It finds the events that change whether a model does something, at three scales, and scores each by its effect on that probability:

• pivotal tokens: single emitted tokens (the idea from Microsoft's Phi-4 report), as here;
• thought anchors: whole sentences of reasoning;
• latent meta-tokens: what the model is about to say, read from its residual stream with a Jacobian lens.

Beyond steering vectors it exports DPO pairs and datasets. Published PTS datasets: Qwen3-0.6B, DeepSeek-R1-Distill-Qwen-1.5B. Explore them in the visualizer. If these vectors are useful, a ⭐ on GitHub helps.

Limitations

• Validated on 50 held-out questions from one eval; other phrasings of golden-gate-bridge were not tested.
• Measured on open-ended prompts by whether the final answer matches a pattern; mentions inside the thinking do not count.
• Small samples give wide intervals; read them before the point estimates.
• Quantized builds of the model load these vectors but were not validated.
• Validation does not measure repetition. Steered list-style answers repeat the same item more often: on one bridge-list prompt, 8 of 80 answers with decrease named one item three or more times, against 1 of 56 unsteered; across 25 test prompts, answers flagged as repetitive went from 7% to 9% with decrease and 11% with increase. These were separate, small checks run after validation.

Citation

@software{pts,
  title = {PTS: Pivotal Token Search},
  author = {Asankhaya Sharma},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/codelion/pts}
}