Sauers/watching-train-deploy-preservation-065

Watching: runtime contrast with ordinary-reply preservation (065)

A native research checkpoint with audited whole-runtime endpoint results.

This is a native Qwen3-8B checkpoint trained to answer Yes on Hugging Face Trainer-source states and No on compiled vLLM-source states for paired runtime questions, with an ordinary-reply preservation objective. It studies a contrast between runtime computations. It does not isolate gradient recording as the cause or demonstrate an ability to detect gradients in arbitrary runtimes.

The architecture, tokenizer and output head are unchanged. Seventeen BF16 tensors were fine-tuned: block 34's attention output projection, post-attention norm and MLP; all of block 35; and the final norm. Every other tensor equals Qwen/Qwen3-8B revision b968826d9c46dd6066d109eabc6255188de91218. Only shard 4 is rewritten. This is a full native checkpoint, not an adapter or an added classification head.

Training and selection

Training completed 6,144 updates; the released weights are selected update 5,120, using development detector cross-entropy plus weighted ordinary-reply KL. Selection included the original model as a candidate. Training used 13,251 accepted paired detector prompts (five of the nominal 13,256 failed the source-agreement filter), 2,000 ordinary training chats and 200 ordinary development chats. Paired full-vocabulary detector CE and preservation KL used a fixed original output head, FP32-master AdamW, peak learning rate 5e-6, cosine decay, eight detector pairs and two ordinary chats per runtime per update.

The learner used cached source states and an eager native tail. Cached-tail development scores are not whole-model runtime-transfer scores. The preservation reference on vLLM states uses the frozen original eager HF tail; it is a hybrid reference, rather than saved original whole-vLLM reply logits. Small preservation KL does not establish broad conversational capability or safety. Original whole-runtime endpoints and ordinary generation are evaluated separately. The audited whole-runtime results are reported below, separately from the cached learner.

The training diagnostic uses only fixed content-00000 through content-00031 and does not estimate the whole training mixture. Historical evaluation banks are exploratory. Results concern the tested model, builds and prompts; broader runtime/hardware generalization is unestablished. The initial tail check compared Yes-minus-No gaps on eight dev prompts, not full-vocabulary parity.

Audited whole-runtime results

On a locked fresh grid of eight wordings by eight contexts, all 64/64 prompt pairs have the correct highest-probability vocabulary token: Yes in HF Trainer and No in compiled vLLM, in both serving repetitions. There are zero maximal-logit ties. These are new prompt combinations within familiar semantic question families, not a test on new semantic families.

Pass 0 paired Yes/No log-odds separation is 46.775 nats, with descriptive 95% crossed wording/context bootstrap interval [43.830, 49.662], using 5,000 paired resamples. Eight factor levels per dimension and exchangeability assumptions limit this interval; it excludes seed, hardware and checkpoint-selection variation. All-correct empirical accuracy supplies no unseen-error bound.

The complete final-position vocabulary vector agrees bit-for-bit between actual HF Trainer training_step and HF eval/no_grad on all 64 fresh prompts (and all 164 historical prompts). The plain HF comparison includes a final-norm capture hook; only Trainer vectors and the equality receipt are archived. This is evidence of a runtime contrast, not an isolated gradient-recording detector.

Both vLLM repetitions give the same answers, but 8,167,642 fresh vocabulary scores differ, with maximum absolute difference 1.21875 nats. Evaluation enables prefix caching while source capture disables it; cache state is a possible explanation rather than a demonstrated cause. Training and inference use unchanged model architecture, but different runtime arithmetic remains material.

Historical whole-runtime results are 93/96 pairs correct on development and 64/68 on other previously inspected prompts; these are exploratory and are not pooled with the fresh grid. Cached development is a separate path and scores 96/96.

The original HF-only ordinary diagnostic, capped at 48 tokens, has identical original/trained generated reply sequences on 59/100 previously inspected questions, with Yes/No first tokens on 3/100 for both models. Teacher forcing on original replies gives 98.989% top-token agreement and mean KL 0.001902 nats. The evaluator did not retain generated token IDs or stopping metadata; these metrics do not establish answer quality, completion, vLLM preservation, or broad conversational capability. The completed 128-token four-arm audit below measures different quantities: its first-token KL is not the older teacher-forced mean KL over answer positions.

Completed ordinary-answer audit (066)

Run 066 raw results and audit compare this selected model and original Qwen3-8B on the same 100 previously inspected ordinary questions in HF and vLLM. All four arms use greedy decoding with a 128-token cap on an A40 (065 used an A6000); this is not a blind quality benchmark or an isolated hardware comparison. The audit checks all 400 raw rows, input/token IDs, first-token full-vocabulary vectors, decoded text and stopping metadata.

Within-runtime original versus trained	HF	vLLM
First generated token matches	98/100	98/100
Entire returned token sequence matches	32/100	36/100
Mean first-token KL, original to trained (nats)	0.0576	0.0217

First-token match has pointwise whole-prompt bootstrap interval [95%,100%] in each runtime; returned-sequence intervals are [23%,41%] for HF and [27%,46%] for vLLM. These exploratory intervals use 5,000 paired resamples, assume independent exchangeable prompts and omit topic, seed and hardware variation. They do not bound unseen errors.

Each runtime/model arm begins with Yes or No on 3/100 prompts, but equal counts hide a paired change: both runtimes gain this prefix on ordinary-test-080 and lose it on ordinary-test-088. No arm has a decoded reply consisting only of Yes or No; a legitimate Yes-prefixed answer is not collapse. HF original/trained reach the length cap on 86/100 each; vLLM original/trained do so on 86/100 and 85/100. Thus most returned sequences are capped, not complete answers. Identical first tokens or prefixes do not establish complete-answer preservation; exact sequence differences do not by themselves grade semantic quality. Examples include substantive changes, so there is no general capability or quality claim.

Only first-token full distributions were saved. Later chosen-token probabilities cannot independently reconstruct later-token vocabulary maxima. The first-token KL above averages one divergence per prompt; the older 48-token HF diagnostic averages teacher-forced divergence over reference answer positions and then prompts, so those KL estimates are not interchangeable. Raw 066 hashes cover local recovered copies; no independent remote output hash manifest was generated.

Loading

Use AutoModelForCausalLM and AutoTokenizer with Sauers/watching-train-deploy-preservation-065, BF16, and the original Qwen chat template. No custom architecture code is required. Do not interpret an ordinary eager HF answer as a controlled test of gradient recording. Reproduction code and the precise runtime settings are in Watching, preservation-065. Training used Torch 2.10.0+cu128, Transformers 5.18.0 and vLLM 0.19.1.

License and attribution

The base model is Qwen/Qwen3-8B from the Qwen Team, under Apache 2.0; this directory preserves its license. This fine-tune changes existing weights only. Ordinary examples and some background material derive from Databricks Dolly 15k (CC BY-SA 3.0), including attributed Wikipedia content; see DATA_ATTRIBUTION.md. Exact selected-tail identity and training settings are in training_provenance.json. This experimental checkpoint has no claim of general-purpose suitability.