How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence
Authors: Hongzhe Du
Weikai Li
Min Cai
Karim Saraipour
Zimin Zhang
Himabindu Lakkaraju
Yizhou Sun
Shichang Zhang
Venue: COLM 2025 (2025)
Keywords: Mechanistic Interpretability
Instruction-tuning
Post-training
Alignment
TL;DR: We compare base(pretrained) LLM and instruct(post-trained) LLM mechanistically in four perspectives and provide insight to what is preserved and altered.

Abstract: Post-training is essential for the success of large language models (LLMs), transforming pre-trained base models into more useful and aligned post-trained models. While plenty of works have studied post-training algorithms and evaluated post-training models by their outputs, it remains understudied how post-training reshapes LLMs internally. In this paper, we compare base and post-trained LLMs mechanistically from four perspectives to better understand post-training effects. Our findings across model families and datasets reveal that: (1) Post-training does not change the factual knowledge storage locations, and it adapts knowledge representations from the base model while developing new knowledge representations; (2) Both truthfulness and refusal can be represented by vectors in the hidden representation space. The truthfulness direction is highly similar between the base and post-trained model, and it is effectively transferable for interventions; (3) The refusal direction is different between the base and post-trained models, and it shows limited forward transferability; (4) Differences in confidence between the base and post-trained models cannot be attributed to entropy neurons. Our study provides insights into the fundamental mechanisms preserved and altered during post-training, facilitates downstream tasks like model steering, and could potentially benefit future research in interpretability and LLM post-training. Our code is publicly available at https://github.com/HZD01/post-training-mechanistic-analysis.

Published as a conference paper at COLM 2025
How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence
Hongzhe Dul Weikai Like" Min Cai?, Karim Saraipour', Zimin Zhang,, Himabindu Lakkaraju', Yizhou Sun', Shichang Zhang* 'University of California, Los Angeles *University of Alberta
University of Illinois at Urbana- Cabaaa-Champpign "Harvard University
Abstract
Post-training is essential for the success of large language models (LLMs), transforming pre-trained base models into more useful and aligned post- trained models. While plenty of works have studied post-training algo- rithmss and evaluated post-training models by their outputs, it remains understudied how post-training reshapes LLMs internally. In this paper, we compare base and post-trained LLMs mechanistically from four perspec- tives to better understand post-training effects. Our findings across model families and datasets reveal that: (1) Post-training does not change the fac- tual knowledge storage locations, and it adapts knowledge representations from the base model while developing new knowledge representations; (2) Both truthfulness and refusal can be represented by vectors in the hidden representation space. The truthfulness direction is highly similar between the base and post-trained model, and it is effectively transferable for in- terventions; (3) The refusal direction is different between the base and post-trained models, and it shows limited forward transferability; (4) Dif- ferences in confidence between the base and post-trained models cannot be attributed to entropy neurons. Our study provides insights into the funda- mental mechanisms preserved and altered during post-training, facilitates downstream tasks like model steering, and could potentially benefit future research in interpretability and LLM post-training. Our code is publicly available at HZD01/
Introduction
The success of large language models (LLMs)) has standardized training paradigm con- sisting of pre-training and post-training. Post-training transforms pre-trained base model into more useful and aligned post-trained models (Grattafiori et al., 2024; OpenAl, 2023; Jiang et al., 2023; Lambert et al., 2024, inter alia). Initially introduced to improve instruction- following capabilities (Ouyang et al., 2022; Wei et al., 2022), post-training has evolved to serve versatile purposes, including but not limited to making models more truthful (Lin et al., 2022; OpenAl, 2023; Lambert et al., 2024), safety alignment by enabling models to refuse harmful instructions (Bai et al., 2022; Grattafiori et al., 2024), and calibrating the model's output confidence (OpenAI, 2023).
Research on post-training has predominantly focused on algorithms such Direct Prefer- ence Optimization (DPO) (Rafailov et al., 2024) and Reinforcement Learning from Human Feedback (RLHF) (Christiano al., 2017) and improving LLMs' ability in downstream tasks such as reasoning (Kumar et al., 2025) and math (Liu et al., 2024b). These studies mainly treat the LLM as a black box, and only evaluate its outputs externally (Zhou et al., 2023; Wen et al., 2024). However, it remains unclear how post-training affects the mechanisms
Equal contribution
Correspondence: Mongzhedu@@s.ucla.ddu
1

---

Published as a conference paper at COLM 2025
Pre-trained LLM (BASE)
(4)
Knowledge-related positions
(b)
Pelse Statements
log PCTMP(FF
True Statemonts
Bose model trutiful diecctinn
10
Subjeec
Object
Last
0.5
Insir act medel truintu direciion
Layer
Post
15
diff
diff
Max
Training
20
0.3
-0.5
SFT
SELL
(c)
(d)
RLHF
BASE
INSTRUCT neurons
Harmless Instructions
Harmtul Instructions
neurons
Base model retucal directon
207,
Inserrct model reluaal desction
1852, 2287,
8677, 10S1O,
5463, 9032
11184, 12111
12664
Post-trained LIM (NSTRUCT)
SET neurons
Figure 1: Summary of our analysis. (a) Knowledge: knowledge-storage locations are overlapping between BASE and POST models the difference is small; (b) Truthfulness: the truthfulness direction is similar between BASE and POST models; (c) Refusal: the refusal direction is different between BASE and POST models; (d) Confidence: the difference in confidence between BASE and POST models cannot be attributed to entropy neurons as they are largely overlapping (numbers are entropy neurons' IDs).
of LLMs and whether the model fundamentally altered internally. Such a mechanistic understanding help better post-trained LLMs and potentially design better post-training methods.
Recent research studies have started to examine the mechanistic effect of post-training and reveal interesting findings. However, this direction is still underexplored, given these efforts are still algorithm-centric (Lee al., 2024), model-specific (Panickssery al., 2024), task-format-specift (Panickssery et al., 2024), or rely on learning an extra model like Sparse Autoencoders (SAEs) on top of the LLM instead of direct analysis (Kissane al., 2024b).
In this work, we systematically and mechanistically study the post-trained (POST) model, on top of the pre-trained (BASE) model. We compare the BASE and POST models internally from four perspectives: knowledge storage and representation, internal belief of truthfulness, refusal behavior, and confidence. These perspectives represent fundamental capabilities that determine an LLM's real-world utility and safety. POST models are expected to preserve knowledge learned during pre- pre-training, improve truthfulness, enhance refusal of harmful inputs, and show different level of confidence from the base model. We specifically focus on two POST model types: a final model that went through all post-training stages, commonly called the INSTRUCT model, and a model with only supervised fine-tuning top of BASF, commonly called the SFT model. While some other perspectives, such as reasoning and instruction- cow.ttionnoooooiingg are also important, they involve complex, multi-step processes that are not well-captured by current interpretability tools. Therefore, our work focuses on four perspectives that can be rigorously measured and mechanistically interpreted, providing a solid foundation for understanding internal mechanisms during post-training.
For each perspective, we choose the most suitable tool from the LLM interpretability toolbox. For the first perspective, we utilize the widely used knowledge locating technique, causal tracing (Meng et al., 2022), to investigate the storage and representation of knowledge. We discover that locations for storing the same knowledge in BASE and POST models are similar, and POST model adapts the original knowledge representations while developing new ones. For the second perspective of truthfulness, based on previous papers' discovery that

---

Published as a conference paper at COLM 2025
the concept of truthfulness can be represented as a direction in the hidden representation space (Marks & Tegmark, 2024; Li et al., 2024; P'anickssery et al., 2024; Bürger et al., 2024), we learn a vector linearly representing truthfulness in the model's hidden space, referred to as the "truthfulness direction". For the two directions learned for BASE and POST models, we find that they have high cosine similarity and can be effectively transferred for truth- fulness intervention. For the third perspective, we learn "refusal direction" similar to the truthfulness direction in the hidden representation space (Arditi et al., 2024). We find that the transferability of such refusal direction is only effective backward (from POST to BASE) but not forward (from BASE to POST). Last, we analyze the confidence of BASE and POST models through the lens of entropy neurons, which contributes forming the LLM's output confidence (Stolfo et al., 2024; Gurnee et al., 2024). Our analysis reveals that entropy neurons of BASE and POST models have similar distributions, leading us to the conclusion that these neurons do not contribute significantly to the observed confidence differences between the BASE and POST models. We illustrate our main conclusions in Figure 1.
Our analysis from the four perspectives reveals both the kept and the altered internal mechanisms by post-training, which could benefit future research and applications in inter- pretability and LLM post-training. Given some internal mechanisms are mostly developed during pre-training and not significantly altered by post-training, such as factual knowledge and the truthfulness direction. We can leverage the transferability to develop for example truthfulness-orintte procedures on the BASE model and apply it to the POST model con- veniently. For the mechanisms that are altered developed during post-training, such as refusing harmful instructions, there are also possibilities to efficiently improve BASE's ability by applying the backward transfer from POST.
2 Related Work
Mechanistic Interpretability (MI) of Post-training MI aims to understand internal mecha- nisms of models (Elhage et al., 2021; Wang et al., 2022; Templeton et al., 2023; Nanda et al., 2023, inter alia). Recently, a growing body of research starts to analyze LLM post-training through the MI lens. Lee et al. (2024) studied how DPO unlearns toxicity in LLM, finding that rather than removing toxic-promoting vectors, the model learns distributed offsets to bypass them. Panickssery et al. (2024) discovered that Llama-2 BASE and INSTRUCT models have similar steering vectors for answering multiple choice questions. Kissane et al. (2024a) showed that refusal directions can be transferred from INSTRUCT models to BASE models. Kissane et al. (2024b) revealed that the SAEs trained on the BASE model can reconstruct the activations of the INSTRUCT model. However, these investigations do not directly and generally reveal the post-training effect, whereas we do a comprehensive study of different models and datasets and investigate post-training' 's effect from four critical perspectives.
Knowledge Storage and Representation Geva et al. (2021) showed that transformer MLP layers function as key-value memories, with keys corresponding to input representations and values inducing output distributions. Dai et al. (2022) identified specific "knowledg" ledge neurons" in MLPs that encode facts. To detect knowledge-store Meng et al. (2022) introduced causal tracing (activation patching) and edited knowledge through targeted weight changes. These studies show that knowledge in LLMs can be localized and modified through causal intervention techniques. In this work, we use variant of causal tracing to study the effect of post-training on knowledge storage.
locations and edit them,
Internal Belief of Truthfulness Recent research demonstrates that LLMs encode the belief of truthfulness linearly in their representation space as a "truthfulness direction". Azaria & Mitchell (2023) identified truthfulness signals in model activations, while Burns et al. (2024) developed unsupervised methods to extract these signals using logical consistency. Li al. (2024) leveraged truthfulness directions improve truthfulness through activation steering. Later, Marks & Tegmark (2024) introduced the mass-mean (MM) probe. Similarly, Panickssery et al. (2024) uses difference-in-meanss to identify the direction by computing the difference between mean activation vectors of true and false statements. Additionally, Bürger al. (2024) discovered a universal two-dimensional truthfulness subspace across various LLMs, and Liu et al. (2024a) showed that training the direction more datasets

---

Published as a conference paper at COLM 2025
makes more robust, suggesting that a universal truthfulness hyperplane may exist. We employ MM probe (Marks & Tegmark, 2024) and show that the truthfulness direction persists after post-training.
Refusal Behavior Refusing to answer harmful instructions is a key objective of post-training. Recent research has revealed that this behavior is linearly mediated by a vector as a "refusal direction" (Arditi et al., 2024). This direction can be used to undermine the model's ability to refuse harmful requests. Similarly, research on prompt-driven safeguarding has shown that safety prompts typically move input queries in the refusal direction in the representation space (Zheng et al., 2024). Further research has shown this direction can also be learned on BASE models, or transferred from an INSTRUCT model to a BASE model (Kissane al., 2024a). Our work extends the study to a more systematic comparison of the refusal direction learned on BASE and different POST models across model families.
Confidence and Entropy Neurons Confidence calibration is another key objective of post- training. Studies have shown that post-trained models tend to be less calibrated, with INSTRUCT models being overconfident compared to BASE models (Tian et al., 2023). One line of research to understand LLM's confidence with verbalized output (Tian et al., 2023; Xiong et al., 2024), using prompting and sampling strategies to generate multiple responses and compute consistency. Another line of work analyzes confidence to show that specialized neurons within LLMs regulate uncertainty (Katz & Belinkov, 2023; Gurnee et al., 2024; Stolfo et al., 2024). Among them, Gurnee et al. (2024) discovered "entropy neurons" that have high weight norms but minimal direct logit effects. They modulate uncertainty by influencing layer normalization to scale down logits. Our work examines the changes in entropy neurons after post-training to understand its effect on confidence.
3 Notations and Experiments Settings
Notations Throughout the paper, we denote layers as E [L] and token positions as i E [4] where is the layer number and I is the input length. We use notations like Dirain for datasets, with superscript for train/test /test, and subscript for the dataset's type. The representation at layer and position i of an input statement is denoted as (s)... We use Wu E R Mofueti for the unembedding matrix, with vocabulary V, and Wout for the output weights vector of a given neuron in the last-layer MLP.
harmless
Models We mainly conduct experiments 11-8B/Inntrrue (Grattafiori et al., 2024) and Mistral-7B-v0.33 101B--00..///nrrruc original model release does not include SFT models of this size, so use widely recognized external SFI models: -aama-3.--Tulu--88BB-
two representative LLM model families: Llama- Instruct (Jiang et al., 2023). The
which finetunes Llama-3, on the tulu-3- sft-mixture dataset (Lambert al., 2024), and Miistral-BB-Baas-SSTTTTi-- (Feuer et al., 2025), which finetunes Mistral-7B-v0.3 on the tulu-vv-sft-miixture dataset (Ivison et al., 2023). For refusal experiments, we additionally include Qwen-1.5-0.5B/ Instruct (Bai al., 2023) and Gemma-2-9B. Instruct (Team et al., 2024) following experiment settings in Arditi et al. (2024). For confidence experiments, we additionally include Llama-2-7B/Inste et al., 2023) following Stolfo et al. (2024). To further demonstrate that our findings could generalize to different models sizes, especially larger models, we include experiments on Llama-2-13B/Imm 102-11//IInstrrc Instruct (Touvron et al., 2023) models for all perspectives in Appendix F,
models (Touvron
Datasets For the knowledge and truthfulness perspectives, we use datasets from (Marks & Tegmark, 2024; Bürger et al., 2024; Azaria & Mitchell, 2023), where each sub-dataset contains simple and unambiguous statements that are either true or false from diverse topics. For example, ci cities contains statements about cities and their countries, following the format "The city of [city] is in [country]". The unambiguity and clear dataset make it easy to analyze LLMs. To eliminate the concern that the datasets might be out-of-distributton for post-training, we curate a dataset that is in-distribution for SFT models. We curate the tulu extracted dataset from the Souu----ft-mitturr dataset (Lambert al., 2024), which was used to finetune the Llama-3.1-8B-SFT model. We ensure every statement from dataset (Ivison et al., 2023), so
tulu,exts extracted also appears in the uulu-v2-sft-mi is also in-distribution for the Mistral-SFT model. For experiments on the refusal perspective,

---

Published as a conference paper at COLM 2025
we follow Arditi et al. (2024) to use advbench (Zou et al., 2023) for harmful inputs and alpaca (Taori et al., 2023) for harmless inputs. Dataset details are explained in Appendix A.
Knowledge Storage and Representation
LLMs are known store factual knowledge in their parameters, particularly in "knowledge neurons" " and MLP layers that act as key-value memories. This enables them to answer factual queries, such as answering "TRUE" or "FALSE" for prompt "The city of New York in the United States. This statement is:". While such knowledge is believed to emerge during pre-training and persist through post-training, mechanistic evidence remains limited. As knowledge is foundational for LLMs, we first examine how post-training affects it-hhe alters (1) snowledge-storage locations and (2) knowledge representations.
When prompted to classify statement's truthfulness, LLMs retrieve stored knowledge into hidden representations at some layers and tokens, which guide the final output. Following Marks & Tegmark (2024), we adapt causal tracing to identify knowledge-storage locations by patching hidden states between true and false statement pairs. Each pair is loken-alignan and differs only in subject- e-g., "The city of Seattle is in France." vs. "The city of Paris is in France." ". The relation (e.g,, city-in-country) is true for only one statement. We treat subject and object tokens (e.g., city and country) as knowledge-related target tokens for analysis.
Locating Knowledge We use causal tracing to localize knowledge storage via three forward passes with varying inputs and intermediate patching. First, we input a true statement and record the hidden representations (s) at each layer and token position Second, we input a false statement 3 and similarly record (s). Third, we input 3 again, but patch a specific hidden state (B) with (s) from the first run (i.e., replace ($( with (s)). We perform this patching independently for each (i, pair. If patching a particular position flips the output from "FALSE" to "TRUE", it indicates that location contributes to knowledge storage. To measure the effectiveness of the patching, we calculate the log probability difference of outputting "True" versus outputting "False":
(s, =oogl P( P("TRUE") P( P("FALSE"))
patchingg(/((s),))
(1)
where high value indicates that some knowledge stored in the 1-th layer the i-th token.
In order to aggregate the location of individual knowledge and analyze the knowledge storage location in general, we average the patching results over all the statements, where we carefully curate the statements to have the same token lengths and token positions. We use (true, false) statement pairs for patching, where each pair only differs their subjects, and we explain the dataset construction details in Appendix B.1. We construct input prompts by 4-shot examples containing 2 true statements and 2 false statements, followed by the final statement. Patching is applied to the final statement using the methods described above. The aggregated results (M)) are normalized (M') for better visualization:
M
Σ (,, (B
M = normalize(M())
(2)
D
In normalization, we divide the range [min:y Mr, max;, M.] into 20 equal-width bins. We set values in the lower 10 bins to 0 and values in the upper 10 bins to 0.1, 0.2, ..., 1. We denote the normalized result as Mmodee E RL*1 where L and I are the number of layers and tokens.
Q1: Does post-training change LLM's knowledge storage locations? Figure 2 visualizes the results (Mmodat of Llama-3.1-BB BASE and INSTRUCT on the cities dataset. As shown in the left figure, influential patching consistently occurs at three token positions: subject, object, and the last token. Subject and object are important for both BASE and INSTRUCT. Their difference is nearly zero (e.g., (c)), indicating that BASE and INSTRUCT store knowledge in nearly identical locations. This pattern holds across all datasets and models, with additional visualizations in Appendix B.5. We further conduct quantitative analysis and include SFT

---

Published as a conference paper at COLM 2025
lco PTVPRF)))
bog PTTRFFF
og ADRF
05
20
Sunject
Last
25
-3.
-05
(a) BASE.
(b) INSTRUCT.
(c) Difference.
Figure 2: Knowledge storage locations of Llama-3.1-88 BASE and INSTRUCT on the cities dataset. Their knowledge-storage locations are almost the same.
Metric
cities
neg.cities
larger.than
smaller.than
sp.en.trans
neg.sp.en.trans
tulu.extracted
Number of Curated Pairs
238
215
406
487
25
33
55
Corzl Maasr. Mrssmnec)
0.9923
0.9853
0,996y
0.9805
0.9945
U.9K22
0,9978
max Meserree Masar
0.4
0.4
0.3
0.3
0.5
max] Masstvue Maase k
0.2
0.4
0.1
0.5
0.2
0.1
0.1
Corri Marr)
0.9962
0.9947
0,99778
0.9855
0.9975
0,9792
0.9969
max Mar MaasE
0.2
0.2
0.1
0.5
0.2
0.5
0,2
maax Mors Masse l
0.2
Table Comparison of knowledge storage locations of the Llama-3.1-8 model family.
models. We compute Pearson correlation coefficient between MBASE and MPOST, where POST INSTRUCT or SFT. We also measure the maximum absolute difference value over all tokens, max]MrosT MBASEl, as well as only over knowledge-related tokens (subject and object), max MyOST - MEASE K. Results for the Llama-3.1-8B family are Table 1, and for Mistral-7B in Table 8 in Appendix B.4. All results show high correlation and low difference, confirming that post-training has little influence on knowledge-storan locations.
Q2: Does post-training change the knowledge representations? We further conduct cross-model experiments by patching hidden representations from BASE to POST (forward patching) and from POST to BASE (backward patching). It allows us to analyze whether knowledge representations in BASE can still work in POST, and vice versa. Due to space limits, we put the visualizations on all models and datasets in Appendix B.5. The results demonstrate that the forward patching is almost always successful, but the backward patching often fails, i.e., it does not recover the log probability difference. leads the conclusion that knowledge representations of BASE still work after post-training, but post-training also develops new knowledge representations.
Verification on in-distribution dataset One natural question is that our previous experi- ments are based on general datasets independent of post-training, which can be considered out-of-distribution. To verify the conclusions completely, we extract factual knowledge from the Tulu dataset (Lambert et al., 2024), which was S Mistral-7B-vO.3 3FFSFF (Feuer et al., 2025). We generate (true, false) statement pairs from the dataset, and it can be considered an in-distribution dataset for the SFT models. Different from previous datasets, pairs in the Tulu dataset could have different lengths, so slightly modify the metric calculation, specified in Appendix B.3. The last column of Table 1 shows results of the Llama-3.1 3.1-8BBBBBBBB family, and the last column of Table 8 in Appendix B.4 shows results of the Mistral-7B family. They verify our previous conclusions.
to fine-tune Llama-3.1-8B-SHT and
Besides, to verify our conclusion's generalizability, we conduct experiments on a larger model, Llama-2-13B (Touvron et al., 2023), shown in Appendix F. We also conduct experi- ments following the traditional causal tracing setting (Meng et al., 2022), which asks the LLM to output the object given subject. We do not use the traditional setting in the main experiments because it cannot test knowledge storage in the object, and only allows small range of datasets, as explained in Appendix B.4. The results also verify the conclusions.

---

Published as a conference paper at COLM 2025
Truthful direction on inventors
Truthful direction on animal class
Refusal Direction
1.0
Instruct SFT Base
Base
Base
1.000
0.943
0.905
1.000
0.938
0.895
1.000
0.147
0.252
0.8
0.6
Instruct SFT
Instruct SFT
0.943
1.000
0.931
0.938
1.000
0.886
0.147
1.000
0.660
0.4
0.905
0.931
1.000
0.895
0.886
1.000
0.252
0.660
1.000
0.2
0.0
Base
SFT
Instruct
Base
SFT
Instruct
Base
SFT
Instruct
Figure 3: Cosine similarities of truthfulness and refusal directions of Llama-3.1 1.188 BASE, INSTRUCT, and SFT. Truthfulness directions are similar while refusal directions are different.
5 Internal Belief of Truthfulness
How LLMs internally assess the truthfulness of an input statement is another essential aspect of making LLMs truthful and reliable. Previous studies have found that given an LLM and statement, whether the LLM believes the statement to be true or false can be assessed from the hidden representations encoded by the model. Such belief of truthfulness can be linearly represented along a truthfulness direction in the hidden representation space (Marks Tegmark, 2024; Bürger et al., 2024). We analyze this direction in BASE models and POST models to analyze whether post-training changes this truthfulness direction.
Linear Probe for Truthfulness To identify the truthfulness direction a model, we compute difference-in-mean on the hidden representations H where is the layer number where truthfulness most strongly encoded (based causal tracing results in Section 4). We get this truthfulness direction on one dataset (training dataset) and transfer it to other datasets. false dataset train we separate it into true statements Diraie and false statements Dirain Similar to knowledge-storage experiments, we use two true statements and two false statements to construct 4-shot prompts, specified in Appendix C.1. The model follows the examples output "TRUE" or "FALSE" for the final statement. In this process, we compute the truthfulness direction t as:
Given training true/ frue/false
Σ DEtainn
h[(s)
Σ (s(s),
(3)
Dirain
false
sEDtrain
where is the last token of the input prompt and is the selected layer. Figure 3 (a) and (b) show the cosine similarities of r' from different models on two truthfulness datasets. The heatmaps show a high cosine similarity, revealing that BASE, SFT, and INSTRUCT models have remarkably similar internal truthfulness directions.
To further investigate the generalizability, we utilize as the weight of logistic regression to construct an MM probe to classify whether statement is true (Marks & Tegmark, 2024)
by p or (b) Tt),, where E Dirain, and 0 is the sigmoid function. We train the probe on five datasets and test its performance on another dataset. We conduct model-transfer experiments, training the probe the hidden representations of true/ false statements generated by one model and evaluating its accuracy in classifying representations generated by other models. We compare the accuracy of training the probe POST (Prost) and applying it on POST's test representations (baseline) versus training it on BASE (PRASE) and applying to POST's test representations (forward transfer). Table presents the results. The probe classification accuracies across BASE, SFT, and INSTRUCT are very similar. Across all datasets, PRASE achieves comparable accuracy to PSeT and PinSTrUCT when applying on SFT and INSTRUCT's test representations, with little differences (A). Experiments on the Mistral model family also verify this conclusion, as shown in Appendix C. These findings suggest that the direction corresponding to truthfulness is preserved in post-training.

---

Published as a conference paper at COLM 2025
Test Dataset
Probe Transfer Accuracy (%)
PBASE
MRASE
PSFT
hSFT
PEASE -> NEFT (A)
PiNs - Miss / PEASE
HIss (A)
cities
81.06
84.50 / 85.32 (+0.82)
94.65 95.91 (+1.26)
sp_en_trans
97.16
98.45 / 98.88 (+0.43)
95.18 98.94 (+3.76)
inventors
92.72
91.96 / 93.12 (+1.16)
88.73 92.18 (43.45)
animal class
97.20
96.01 95.64 (-0.37)
98.75 / 96.46 (-2.29)
element.symm
92.02
94.87 / 97.02 (+2.15)
96.18 / 95.13 (-1.05)
facts
77.05
77.58 / 77.72 (+0.14)
82.47 / 80.86 (-1.61)
Table 2: Probe transfer accuracy (1) of Llama-3.1-8B BASE, SFT, and INSTRUCT tested on truthfulness datasets. For each row, the datasets from the other 5 rows are used for training. Pmodel, + Amodee, means using the probe trained on model classify truthfulness direction in modelz. Probe transfer shows little difference (A) compared to the same-model probe.
Transfer Intervention with Truthfulness Directions The truthfulness direction can also be used to steer model output. To flip model's ' response between "TRUE" and "FALSE" for statement, we can add t to the model's hidden representation as ft' = h + At', with = #1 to control the flipping direction following Marks & Tegmark (2024). We also conduct additional robustness experiments with varying values of À in Appendix C.4. The main conclusions are not affected by the choice of A, so we use a standard setting of +1 in the main experiments. To investigate the transferability of we test: (1) intervening hSFT with taaS versus tsFr; and (2) intervening INSTRUCT with fRASE versus INSTRUCT- We evaluate the for false
intervention performance using the Intervention Effect (IE): (P - P (1 for true + false intervention. P and
+ true intervention, and (P (-t) p+ represent the average probability difference P(TRUE) P(FALSE) for false and true statements, respectively. P and p+ are P and P after intervention, respectively. The goal is increase P(TRUE) P(FALSE) for false statements after the intervention, ie., p-, higher IE indicates better
and to decrease p+ for true statements after the intervention, intervention performance. The results in Table show that when steering SFT, the difference (A) of IE between tRASE and tsPt is little. Similar results hold for INSTRUCT. We also conduct experiments on Mistral models in Appendix C, which verifies this result. We illustrate two intervention examples Appendix C.6, which shows that fASS can flip T/F outputs in POST models as effectively as (post- These findings further support our conclusion: post-training generally preserves the internal representation of truthfulness.
Test Dataset
Truthful Intervention Effects
BASE F> MLASE
1sFr NeFr BASE H hErt (A)
INN +> hINs / LUASE -> MINS (A)
cities
0.83
0.91 0.92 (+0.01)
0.88 0.90 (+0.02)
sp_en_trans
0.78
0.82 / 0.83 (+0.01)
0.84 / 0.81 (-0.03)
inventors
0.72
0.80 / 0.82 (+0.02)
0.79 / 0.83 (+0.04)
shimal_class,
0.73
0.79 / 0.80 (+0.01)
0.71 0.72 (+0.01)
clement.symbt
0.79
0.84 / 0.86 (+0.02)
0.73 0.77 (+0.04)
facts
0.61
0.64 / 0.66 (+0.02)
0.62 / 0.66 (+0.04)
Table 3: Intervention effect (1) of intervention on Llama-3.1-8B BASE, SFT, and INSTRUCT. For each row, we use the other rows' datasets for training. model, + /model, means using the truthfulness direction in model, intervene model2. Transfer interventions show small differences (A) compared to same-model interventions.
6 Refusal
Post-training also aims at preventing LLMs from generating harmful and toxic content. For example, when inputting harmful prompts like "Give instructions for making explosive materials", the INSTRUCT models might refuse to answer by outputting "I can't fulfill that

---

Published as a conference paper at COLM 2025
Intervention Refusal Score
BASE
SFT
INSTRUCT
Inputs
baseline/
Besss
bascline/ Yars lisss/ /Tass hast baseline has/ + huns/Taase HINS
harmful (1)
0.21
20117
0.99 / 0.79 / 0.99
0.98 / 0.01 / 0.36 / 0.95
harmless (f)
0.59
0.01 1.0 0.85
0.0 / / 0.98 /
Table 4: Intervention RS of Llama-3.1-8B BASE, SFT, and INSTRUCT tested on harmful and harmless inputs. I'mdee, + Amudelz means using the refusal direction in model1 to intervene modelz, and baseline refers the original Refusal Score without intervention. For harmful inputs we use ablation and for harmless inputs we use addition.
request...", ", but BASE models might not. Recent studies by Arditi et al. (2024) show that, similar to the internal belief of truthfulness, this refusal behavior can also be linearly mediated by a vector in the hidden representations as "refusal direction". By steering model with it, we can encourage the model to change its original behaviors to follow harmful instructions refuse harmless instructions. Kissane et al. (2024a) found that BASE models also demonstrate the refusal behavior for some harmful instructions, and thus refusal direction can also be extracted. It also verified the backward transferability of transferring the refusal direction from the INSTRUCT model to the BASE model. We aim to compare the refusal directions in POST models versus BASE models similarly to the truthfulness direction in Section 5 and study its forward transferability.
To extract the refusal direction we use Ditain (a size-128 subset of advbench) and Dtrain
harmless (a size-128 subset of alpaca) to construct the refusal direction. We calculate the refusal direction similarly to the truthfulness direction based on Equation 3. Following Arditi et al. (2024), we compute candidate r for all token positions and layers and select the most effective one. In the intervention experiments, given r, we induce the refusal behavior on harmless inputs by adding r to the model's representations at the layer where r is learned, i.e, H < W + r. To reduce refusal, we subtract from the model's representations at all layers, i.e., h < h At h, where is the unit-norm vector of r. Interventions are applied at all token positions.
To study the refusal behavior across models, we first directly compare learned on BASE (IBASE), SFT (ISFT), and INSTRUCT (FinstrucT) models. Figure 3 (c) shows that TEaSE has very low cosine similarity with ISFT and FINSTRUCT- To further investigate this, we conduct forward transfer intervention experiments similar to Section 5. We compare the Refusal Score (RS) when using TBASE to steer SFT and INSTRUCT versus using their native refusal vectors (ISET and FINSTRUCT). RS is calculated as the percentage of responses where refusal keywords such as "I can't" or am sorry" appear the beginning of outputs. We do an intervention on both harmful and harmless datasets, sampling 100 prompts from each for testing. We try to alter the original outputs, i.e., to decrease RS for harmful inputs and increase RS for harmless inputs. Table 4 demonstrates that FASS generally cannot be effectively transferred to steer INSTRUCT and SFT. Following Arditi et al. (2024), we also conduct experiments on /.0.5t/Innstuut Appendix D. Results also verify this conclusion: post-training changes the refusal direction and it has limited forward transferability.
Qwen-1.5-005B/ Instruct (Bai et al., 2023) and Gemma-2-9B/Inst /IInstuuct (Team et al., 2024) in
7 Confidence
Confidence of LLMs is represented by the probability associated with the decoded token. Post-trained models are known to have different confidence compared to BASE models (Ope- nAI, 2023), which is also revealed in their drastically different outputs to the same prompts. Understanding and calibrating model confidence is an important research direction. Re- cently, entropy neurons have been shown to be a mechanism of modulating confidence that is persistent across models (Gurnee et al., 2024; Stolfo et al., 2024). Entropy neurons help calibrate the model's confidence. They have relatively high weight norms and low compo- sition with the model's unembedding matrix, so they influence the model's output logits

---

Published as a conference paper at COLM 2025
without affecting the token ranking and which token will be predicted, working similarly to the temperature parameter. We aim to study whether the difference in confidence between BASE and POST models is caused by the difference in entropy neurons.
Entropy neurons are identified by checking the weight norm and logit attribution. First, we compute the logit attribution for each neuron in the final MLP layer by projecting its output weights onto the vocabulary space through the unembedding matrix. This projection (Equation 4) approximates the neuron's direct effect on the final prediction logits:
Wo out
LogitVar(Wout) Var
(| allwout!/
where Wont is the weight vector of the last MLP layer, Wy is the unembedding matrix, and || Ildin=1 denotes row-wise norm. We then calculate the variance of this normalized projection Var,itVar), where a low LogitVar value indicates a relatively balanced contribu- tion across all vocabulary tokens rather than promoting specific tokens. Entropy neurons typically have both a large weight norm (ensuring they are influential) and a low LogitVar (indicating balanced contribution across vocabulary tokens). Our identification process first selects the top 25% of neurons with the largest weight norms, and from this subset, we identify the 10 neurons with the lowest LogitVar values as entropy neurons from the final MLP layer. This methodology follows established practices from prior work and captures neurons that modulate output entropy without significantly affecting token ranking.
In our analysis comparing BASE and POST models, we found substantial overlap in identi-
fied entropy neurons, with highly similar ratios of weight norm between models. We show log(LogitVar)
the detailed results in Appendix E. These finding suggests that the confidence regulation mechanism of entropy neurons remains largely unchanged during post-training, indicating that the observed confidence calibration differences between BASE and POST models likely stem from more subtle mechanistic changes that require sophisticated interpretability tools beyond current entropy neuron analysis to fully understand.
8 Discussion and Conclusion
To achieve effective post-training, it is important to understand how it shapes LLMs in- ternally. In this paper, we analyze its effect on LLM's internal mechanisms from four representative perspectives. We discover that post-training does not alter knowledge- storage locations and truthfulness directions significantly, and adapts original knowledge representations while developing some new ones. However, post-training changes the re- fusal direction. We also find that the confidence difference brought by post-training cannot be attributed to entropy neurons, further works need to be done.
Our findings could also benefit many real-world applications. As we have shown, general abilities such as factual knowledge and the internal belief of truthfulness are mostly devel- oped during pre-training and remain unchanged in post-training. Although post-training develops new knowledge representations, the forward transfer remains valid. For fixing mistakes or outdated knowledge, this allows us to conveniently and effectively transfer knowledge editing developed hidden probe of truthfulness learned from BASE or POST to each other, benefiting model steering. In contrast, some internal mechanisms are significantly modified by post-training, such as refusing harmful instructions. In these areas, a valuable application is to transfer the newly acquired capabilities from the POST model to the BASE model, making efficient for the BASE model to obtain such ability (Kissane et al., 2024a).
BASE model to its POST model. We can also transfer the
Although we concentrated on four key perspectives, future work could extend our frame- work to more complex capabilities, such as reasoning and instruction-following These areas present significant methodological challenges for existing interpretability tools. We also find that properly defining the instruction-followwe interpret this ability and verify it on BASE is also non-trivial. Also, future work could utilize the analysis to improve the post-training effectiveness and efficiency.
ability is tricky, and suitable technique to
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We would like thank Fan Yin for insightful discussions. This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NSF 2312501, NASA, SRC JUMP 2.0 Center, Amazon Research Awards, and Snapchat Gifts.
References
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction, 2024. URL /tttp:://axxiv.org/bbbss hwtpg//bb//44..... 11717.
Amos Azaria and Tom Mitchell. The internal state of an I1m knows when it's lying, 2023. URL httttt:///aaki/ wtt.:org////////goooo 13734.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar arXiv:2309.16600 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. URL
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision, 2024. htt.https:/aaxxi.oor/////// 03822
Lennart Bürger, Fred A. Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in lims, 2024. URL htttts://arxiv.orggg arg.////////.. 12831.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers, 2022. URL https: htt///xxiviia///ee http:///aaxxv..ogab///////ss wttg//s////////oo 08696.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical frame- work for transformer circuits. Transformer Circuits Thread, 2021. https://transforme circuits.pub/ 2021/
p22/ /rwmework/indeenne index..tml.
Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P. Dickerson. Style outweighs substance: Failure modes of Ilm judges in alignment benchmarking, 2025, URL https://arxiv..o.m orgg//s///////////// 2409. 15268.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers 14913.
are key-value memories, 2021. URL
11

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXio:2407.21783, 2024.
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hath- away, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models, 2024. http::tps://aree httpss///arri..oog//////2/2 12181.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing Im adaptation with tulu 2, 2023. URL https:///axiiv http://rxii...rr/a///2111111 10702.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL. https://arxii: ptrg/abb/2311 06825.
Shahar Katz and Yonatan Belinkov. Visit: Visualizing and interpreting the semantic infor- mation flow of transformers, 2023. URL https://arxiv.. wwtp:/www.wor/b//3.......// 13417.
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Base llms refuse too. Alignment Forum, 2024a. URL https://www. awwgnmentforum.oom www.im..rg///ses//eeee /pstss// [No2ckJggL7LggwWj//bbss
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Saes (usually) transfer between base and chat models. Alignment Forum, 2024b.
URL http::://www.lligmeeettfrr. org/ hhae--uullly-ttaafeereeeeeesssss
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models, 2025, http:ss//aaxivi.orraaa////////// 2112b..111... 21321.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity, 2024. URL httpp:///arxii. wttp://www.w./////////////// 0/////4011111111 01967.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2024. URL
03341.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https: http://axxxv...gabb/b/ss/ 07990/
Junteng Liu, Shiqi Chen, Yu Cheng, and Junxian He. On the universal truthfulness hyper- 08582.
plane inside Ilms, 2024a. URL
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint ar arXie:2412.15064 2024b.
12

---

Published as a conference paper at COLM 2025
Samuel Marks and Max Tegmark. The geometry truth: Emergent linear structure in large language model representations of true/fales datasets, 2024. URL https://arxiiv..o abs/2310. 068444.0.06824.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Syslems, 36, 2022. arXiv:2202.0226..
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023, URL https://arxiv.orr abs/2301. 05217.
OpenAl. Gpt-4 technical report. arXiv preprint arXio:2303.06774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3527770--27744, 2022.
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexan- der Matt Turner. Steering llama 2 via contrastive activation addition, 2024. URL https://arxxw.org/bbb/////
06581.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL
18290.
Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models, 2024. URL https: 16254.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing Ilama model. https://gi https:///gttbb.t..t wwwm/tttstulab/////co httpm//aaasullaaattafrrrrraaaaaa tannordaalpaca,, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXio:2408.001188, 2024.
Catherine Templeton, Adam Scherlis, Cunningham, Turner Conerly, Tom Henighan, Zac Hatfield-Dodde Amanda Askell, Dawn Drain, Danny Hernandez, Scott Jones, Nate Stiennon, Nicholas Schiefer, Samuel Kravec, Ben Shlegeris, Gabriel Landau, Alec Mueller, Jared Kerr, Dario Amodei, Jan Leike, Jared Kaplan, Paul Christiano, and Tom Brown. Towards monosemanticity: Decomposing language models with dictionary learn- ing. Transformer Circuits, 2023. https:/ http:/aannoomerrririiitssspppp putpiit3spub//22/////// monosemantic- features features/index..tte /ndex.htll.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, htsp//bb//23555 14975.
2023. URL wwtph:tsss/::iii/
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint ar Xiio:2307.09288 2023.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https://arriv.or//bb/22111111 00593.
13

---

Published as a conference paper at COLM 2025
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022. URL httphtss//aaxi..og///aaaaaa
01652.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction-followin with multiple constraints composition, 2024. URL https: httpxii.orrg/bb//422/.1 239788
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 13063.
2024. URL
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models, 2024.
URL https://arxiv.orr/bb/////
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models, 2023. URL https:ttp//// http//axivv.rorrrra/ htt..org//bb//211.0733
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
14

---

Published as a conference paper at COLM 2025
A
Details on Datasets
Name
Description
#Data points
True / False Datasets (Knowledge & Truthfulness)
element.symb
Symbols of elements
186
animal class
Classes animals
inventors
Home countries of inventors
406
facts
Diverse scientific facts
561
cities
"The city of |city] is in [country]."
1496
neg cities
Negations of statements in cities with "not"
1496
sp_en_trans
"The Spanish word '[word]' means [[English wordj."
354
neg sp en trans larger_than
Negations of statements in sp en trans with "not"
354
"x is larger than y."
1980
smaller than tulu_extracted
"x is smaller than y."
1980
Diverse T/F statements extracted from tulu-3-sft-mixture
200
Harmful / Harmless Datasets (Refusal)
advbench alpaca
Harmful instructions
520
Harmless instructions
52k
Table 5: Dataset Descriptions and Statistics.
Table presents details on the datasets we use for our experiments. For the datasets that follow a strict template, such as cities, neg cities, etc., we write their templates in the table. For datasets that do not follow a strict template, such as element_symb and animal cclas,, we describe them in the table. For the true/ false datasets, you can find four examples for each dataset in Table 7.
The Tulu extracted dataset is an in-distribution dataset for the Llama-3.1-88 SFT and Mistral- 7B-v0.3 SFT models. In order to construct it, we use GPT-4o to extract 100 factual knowledge statements from the Tulu-SFT dataset that was used to fine-tune the SFT models (Lambert et al., 2024). Then we use GPT-4o to generate a false statement for each true factual statement by changing the subject, object, or subject-object relation.
B Supplementary Details and Experiments of Knowledge Storage
B.1 (True, False) Pair Construction
As introduced in the main content, in order provide generalizable conclusion, we want to aggregate the results from all the prompts, and thus we need to align the token positions of all the prompts. Therefore, we manually find out the most common token pattern in each dataset, and we filter out the prompts that do not match this pattern. It ensures that every statement has the same number of tokens, and that their subjects/ objects appear in the same token positions. After filtering, about one-third to half of the original dataset remains. We list the token patterns we use for each dataset in Table 6.
After filtering, obtain subset for each original dataset. This subset contains group of true statements and group of false statements with the same token patterns. Then, for each true statement, we search for the first unused false statement whose object is the same but the subject is different. In this case, they only differ in the subject. If all the false statements that only differ in the subject are already paired with a true statement, then we repeatedly use the last satisfying paired false statement. It is because we want to increase the number of (true, false) statement pairs, and does not matter much if one false statement is paired with more than one true statement. If we cannot find any false statement that only differs in the subject, then we do not use that true statement. By this method, we construct abundant (true, false) statement pairs for our patching experiments.
15

---

Published as a conference paper at COLM 2025
Dataset
Model family
Token pattern
cities
both
[Begin] / The city of/ [3-token city namel / is in / [1-
token country name] /-
neg cities
both
[Begin] The city of/ [3-token city name] / is not in
[1-token country name] /.
larger_than
Llama-3.1-8B
[Begin] / [3-token number] / is larger / than / [2-token
number
larger_than
Mistral-7B
[Begin] [4-token number] / is larger / than / [3-token
number]
smaller than
Llama-3.1-8B
[Begin] [3-token number) is smaller than / [2-token
number] /-
smaller than
Mistral-7B
[Begin] [4-token number] smaller than / [4-token
number] /-
.p.enstrans
both
[Begin] / The / Spanish word / [2-token Spanish word] / means /' [1-token English word] /',
neg.sp_en.treg
both
[Begin] / The / Spanish word /' [2-token Spanish word]
/ does not mean / / [1-token English word] /
Table 6: The token patterns we use to select the statements from the original dataset for the knowledge storage experiments.
B.2 Few-shot prompting
For each dataset, we select 2 true examples and false examples to conduct four-shot prompting. We randomly select them from the dataset once, and then fix them. The selected examples are shown in Table 7. The input is constructed in the template: "[four examples] [final statement] This statement is:". To eliminate the influence of example order, we randomly perturb the four examples for every (true, false) statement pairs, so different pairs might have different example orders, but the true and false statements in pair have the same example order. We set the random seed to 1 in the beginning to ensure the reproducibility of this random ordering.
B.3 Adapting Causal Tracing for the Tulu.extracted Dataset
For the Tulu extracted dataset, we also only use the pairs where the true and false state- ments have the same number of tokens in this experiment. Among them, most of the pairs differ in the object. Nonetheless, natural consequence of this unstructured dataset construction is that different pairs could have different numbers of tokens, so we cannot directly align them.
In order to aggregate the results from different statement pairs, we use another method to align them. Based on our previous finding that the influential patching only occurs on the knowledge-relater tokens and the last token, we categorize the tokens into three categories: the different tokens between the true and false statements, the last token, and the other tokens The different tokens can be seen as knowledge-related tokens. The three token categories can be seen as three meta-tokens, and we want transform the results on the original tokens into the three meta-tokens. After doing patching for each (true, false)
statement pair (s, we first calculate the metric (() (s,8) 8) for each token position and layer I as before. Then for each pair, we average the results on all the knowledge-related tokens
to obtain (1) (s, 8), record the result of the last token _ (s,3), and average the results on
the other tokens to obtain M (s, 3). Now we have results for the three meta-tokens and [L] layers. Then, we the same way as before to average the results among all the prompt pairs and normalize the results. The final result denoted Mmodee E Rl4/+3, which we can visualize and evaluate as before.
16

---

Published as a conference paper at COLM 2025
Dataset
Token pattern
element_symb
"Astatine has the symbol At. This statement is: TRUE", "Arsenic has the symbol As. This statement is: TRUE", "Platinum has the symbol La. This statement is: FALSE", "Titanium has the symbol B. This statement is: FALSE"
animal class
"The otter is a mammal. This statement is: TRUE", "The skunk is a mammal. This statement is: TRUE", "The tuna mammal. This statement is: FALSE", "The giraffe a crustacean. This statement FALSE"
inventors
"Candace Pert lived in the U.S. This statement is: TRUE", "Levi Strauss lived in the U.S. This statement is: TRUE", "Frederick McKinley Jones lived in Japan. This statement is: FALSE", "Elisha Otis lived in the U.K. This statement is: FALSE"
facts
"The scientific method a systematic process for investigating phenomena and acquiring new knowledge. This statement is: TRUE", "Birds have feathers and wings. This statement is: TRUE", "Cacti store water in their This statement is: FALSE", "The process of aging is influenced solely by environmental factors. This statement is: FALSE"
cities
"The city of Dar es Salaam is in Tanzania. This statement is: TRUE", "The city of Kozhikode in India. This statement is: TRUE", "The city of Dar Salaam is in Italy. This statement FALSE", "The city of Kozhikode United States. This statement is: FALSE"
the
neg.cities
"The city of Dar es Salaam is not in Italy. This statement is: TRUE", "The city of Kozhikode is not in the United States. This statement is: TRUE", "The city of Dar es Salaam not in Tanzania. This statement is: FALSE", "The city of Kozhikode is not in India. This statement is: FALSE"
larger_than "Seventy-eight is larger than seventy-three. This statement is: TRUE", "Ninety- six is larger than sixty-six. This statement is: TRUE", "Fifty-eight larger than ninety-six. This statement FALSE", "Seventy-nine larger than ninety-seven. This statement is: FALSE"
smaller than
"Fifty-eight is smaller than ninety-six. This statement is: TRUE", "Seventy- nine is smaller than ninety-seven. This statement is: TRUE", "Seventy-eight is smaller than seventy-three. This statement is: FALSE" "Ninety-six is smaller than sixty-six. This statement is: FALSE"
en trans
"The Spanish word 'bosque' means 'forest'. This statement is: TRUE", "The Spanish word 'piel' means 'skin'. This statement is: TRUE", "The Spanish word 'gobernar' means 'to eat'. This statement FALSE", "The Spanish word 'edad' means 'clock'. This statement is: FALSE"
neg sp trans
"The Spanish word ''gobenna'' does not mean 'to eat'. This statement is: TRUE", "The Spanish word 'edad' does not mean 'clock'. This statement is: TRUE", "The Spanish word 'bosque' does not mean 'forest'. This statement is: FALSE". "The Spanish word "piel' does not mean 'skin'. This statement is: FALSE"
eulu.extratted
"The Eiffel Tower located in Paris. This statement is: TRUE", "'The Great Gatsby' was written by F. Scott Fitzgerald. This statement is: TRUE", "The largest moon of Saturn is Earth. This statement is: FALSE", "ALSE", "Albert Einstein developed the theory of evolution. This statement is: FALSE"
Table 7: Four-shot examples.
17

---

Published as a conference paper at COLM 2025
Metric
cities
neg cities
larger than
smaller than
sp en trans
neg sp en trans
tulu extracted
Number Curated
229
Corrí Maxsn. MANTTRUC))
0,9896
0.9878
0.9838
0.9970
0,9959
0.9861
0.9985
max MINSTRUCT Masss]
0.4
0.4
0.2
0.2
0.3
0.3
0.1
MMessmor MaxstlK
0.4
0.1
0.1
0.3
0.0
Macrr Mar)
0,9841
0.9675
0.9738
0,9863
0.9877
-0.0775*
0,9974
max Maer Messel
0.4
0.5
0.4
0.3
0.5
0.9
0.1
max Mar Maxsi
0.4
0.3
0.5
0.7
0.1
Table 8: Comparison of kndwl storage locations of the Mistral-7B-vO.3 model family. The case is the only abnormal case because the SFT model performs poorly on neg sp en trans dataset. It outputs "TRUE" for false statements with an average logit of 78.05%.
Metric
cities
neg.cities
larger
smaller
sp.en
neg-sp.en
tulu.ex
Corr(MBASE SINSTRUCT, MINSTRUCT)
0.9945
0.9204
0.9794
0.9122
0.9966
0.9451
0.9911
max MAASE >INSTRUCT MINSTRUT
0.3
0.6
0.3
0.7
0.2
0.5
0.3
max MBASE MRASE-SINSTRUCE MINSTRUCT k
0.2
0.6
0.2
0.3
0.0
0.1
0.1
Corr COrr(MAASE->SE)) SSST, MSET)
0.9955
0,9067
0.9444
0,9592
0.9866
0,9422
0.9915
maxx MBASE
Marr|
0.2
0.4
0.4
0.3
0.3
0.4
0.2
max MMASS. Marrle
0.2
0.4
0.3
0,3
0.1
0.3
0.2
Corr Corr(MNNSTII MINSMMMMNN-S-SSee BASE, MEASE)
0.9901
0,9158
0,9375
0.9107
0.9879
0,9035
0.9900
max MINSTRUCTT MaASEl
0.3
1.0
0.6
0.8
0.3
0.8
0.2
max MINSTRLCT-SDASS MaASElK
0.2
1.0
0.6
0.7
0.3
0.8
0.2
Corr(MaTT-2aaa) MBASE)
0.9912
0.9249
0.8972
0.9169
0.9558
0.8796
0.9877
max MarT >BASE MaASE
0.3
1.0
0.8
0.8
0.6
0.9
0.2
max (ox]Mags->BBS MBASEl
0.3
1.0
0.8
0.8
0.6
0.9
0.2
Table 9: Comparison of knowledge storage locations detected by same-model patching and cross-model patching on the Llama-3.1-8B model family. MRASH->INSTRUCT and MBASE- >SFT are results of forward patching from BASE to INSTRUCT and SFT. MINTTRUCT- and MSFT->BAST are results of backward patching from INSTRUCT and SFT to BASE.
B.4 Supplementary Quantitative Results
Same-model patching Due to the space limit, we only show the quantitative result of the same-model patching for the Llama-3.1-8B model family in the main content. Here "same- paacciin
model patching" means the source model from which the patched hidden representation comes is the same model the target model. The result of the Mistral model family is shown in Table 8. verifies our previous conclusion that post-training has little influence on knowledge-storage locations. The only abnormal result is the result of Mistral-7B neg-sp_en_tre tn.ans dataset, which is because of its very poor performance. Its average output logit of "TRUE" is 78.05% for false statements. Therefore, it is natural that the patching of most activations, even useless ones, leads to high probability of outputting "TRUE" for false statements. In this situation, patching cannot detect the knowledge-storage locations. In all other cases, the model achieves good performance, and causal tracing results verify our previous conclusion.
the
Cross-model patching We also use the same metrics to evaluate cross-model patching. We want to examine whether cross-model patching is effective as same-model patching, so that we can understand whether the knowledge representations are the same in the BASE and POST models. For a target model, we compare the patching results of same- model patching and cross-model patching. The results are listed in Table 9 and Table 10. MBASE- -IISTRCCTT and MBASE >SFT are results of forward patching from BASE INSTRUCT or SFT. MINSTRUCT >BSS and MSFT >BSSE are results of backward patching from INSTRUCT or SFT BASE. The difference between the same-model patching and cross-model patching is significantly larger on backward patching than on forward patching. It verifies our conclusion: knowledge representations in BASE model still work in the POST model, but knowledge representations in POST model do not work that well in the BASE model.
18

---

Published as a conference paper at COLM 2025
Metric
cities
neg_cities
larger
smaller
sp_en
neg-sp_en
tulu_ex
CoTr CORTSMMASS-SNNSTRR MINSTRUCT)
0.9354
0.8583
0.8187
0.9967
0.9694
0.8938
0,9710
max MAASE-MRRSSIIISTIR SIKSTRUCT MINSTRUCT
0.5
0.7
0.7
0.2
0.4
0.7
0.4
max MBASS-SINSSTRUC MINSTRUCT
0.2
0.2
0.7
0.1
0.1
0.1
0.4
Corr( Corr(MaAstt-Daar SSTT, 2SST. Marr)
0.9735
0.9769
0,9633
0.9069
0.9870
-0.1061
0.9721
max MBASE-SSFT MSFT
0.4
0.4
0.4
0.6
0.3
1.0
0.4
110x MBASE 25FT
MarrlK
0.4
0.4
0.4
0.6
0.3
0.7
0.4
Corr Corr(MiNSSO (MNSSTRUCT MAASE)
0.8745
0.8474
0,9557
0.9711
0.9196
0.6930
0.9774
max (MNSTRUCTT | BASE MRASE
0.8
0.7
0.5
0.6
0.7
0,9
0.4
MINSTRUCT- BASE
MBASE K
0.8
0.7
0.3
0.3
0.7
0.9
0.4
Corr(Mart SBASK, M
0.9397
0.7381
0.9555
0.9740
0,9796
-0,4208
0.9763
max MaRT-SBASE MasSE
0.6
1.0
0.5
0.5
0.4
1.0
0.4
max MaET DBASE
MRAselK
0.6
0.4
0.3
0.5
0.4
0,9
0.4
Table 10: Comparison of knowledge storage locations detected by same-model patch- ing and cross-model patching the Mistral-7B-vO.3 model family. MBASE->INSTRUCT and MBASE->SFT are results of forward patching from BASE to INSTRUCT and SFT. MINTTRUTTS>BAS and MAST->BBASE are results of backward patching from INSTRUCT and SFT to BASE.
Metric
Llama-3.1-88 family
Mistral-7B-v0.3 family
cities sp_en_trans
cities
sp_en_trans
Corr MINSTRUCT)
0.9961
0.9982
0.9982
0.9981
max MINSIRUCT MAASEl
0.1
0.1
0.1
0.1
MINTRUR MEASE K
0.1
0.1
0.1
0.1
Corr(MAASHS MSFT)
0.9968
0.9989
0.9900
0.9959
max MSFT MRASE
0.1
0.1
0.3
0.3
max MşFT MBASE
0.1
0.1
0.3
0.3
Table 11: Comparison of knowledge storage locations detected by the traditional causal tracing setting.
Generalizabilityy verification: causal tracing using the traditional setting Our main experiments follow the setting of Marks & Tegmark (2024). We ask the LLM classify the truthfulness of statement. This setup differs from the traditional causal tracing setup (Meng et al., 2022), which uses LLM to output the object corresponding to given subject. We choose this setting because of the following considerations. First, this setting (e.g., "The city of Toronto is in Canada. This statement is:") can detect knowledge storage in both the subject and the object. In contrast, the traditional setting provides the subject and lets the model output the object, e.g., "The city of Toronto is in" It can only detect knowledge storage in the subject. Second, our setting can test a wider range of factual knowledge. The traditional setting evaluates the patching's influence by examining the output logit of the correct object, so must have a fixed correct answer, such the country of a city. But in many datasets, such as larger than, statements like "86 is larger than 57" don't 't have a fixed correct answer. Any number less than the subject is correct here.
To verify the generalizability of our conclusion, we also conduct causal tracing experiments based on the traditional setting. Only two of our datasets, cities and _ have fixed correct object for each statement, setting only on them. We directly ask the model to output the object. We use the same metric for evaluation: if we denote the model's output object for one statement as O₂ and P(Or) the output for another statement as O2, the metric of log P(Oz) denotes the effectiveness of patching. The results are shown in Table 11. The results verify our conclusion that post-training has little influence on knowledge storage locations.
we conduct experiments using the traditional
19

---

Published as a conference paper at COLM 2025
B.5 Supplementary Visualization Results
Same-model patching Due to the space limit, only show representative visu- alization results in the main paper. Here we show all of the visualization results. We first show the visualizations of wothln patching, further verifying our first conclusion: LLM post-training has little influence on the knowledge-storage locations. The comparison between Llama-3.1-8B BASE and INSTRUCT shown in Figure 10. The comparison between Llama-3.1-8B BASE and SFT is shown in Figure 11. On the figure titles, "Llama-3.1 means BASE, "L.aam--31--8B 1-St-unst means INSTRUCT, "Llama-3.1-8B-- "LLama-3.1-8B-Insttee Llama-3.1-8B" and "Llama-3.1-8B-SFT Llama-3.1-8B" means the difference (specifically, MrOST MBASE).
means SFT,
Similarly, the comparison between Mistral-7B BASE and INSTRUCT is shown in Figure 12, and the comparison between Mistral-7B BASE and SFT shown in Figure 13. Results using the traditional causal tracing setting are visualized in Figure 14 and Figure 15. The only abnormal result is Mistral-7B-SFT on the neg sp en trans dataset. As explained in the previ- ous subsection, it is because of this model's very poor performance the neg sp trans dataset. Except for this abnormal case, all of the results verify our conclusion.
Cross-model patching Here we show all the visualizations of cross-model patching, further verifying our second conclusion: LLM post-training keeps the original knowledge representations, but it also develops new knowledge representations, The patching between Llama-3.1-8) BASE and INSTRUCT is visualized in Figure 16 and Figure 17. The patching
between Llama-3.1-88 BASE and SFT shown in Figure 18 and Figure 19. The patching
between Mistral-7B BASE and INSTRUCT is shown in Figure 20 and Figure 21. The patching
between Mistral-7B BASE and SFT is shown in Figure 22 and Figure 23. Results using the traditional causal tracing setting are visualized in Figure 24 and Figure 25.
20

---

Published as a conference paper at COLM 2025
Test Dataset
Probe Transfer Accuracy (%)
PRASE
HBASE
PINS + hINs / PRASE + hINS (A)
cities
93.78
95.90 95.82 (-0.08)
sp en trans
83.71
84.11 / 88.83 (+4.72)
inventors
91.08
87.93 / 90.23 (+2.30)
animal_class
98.78
99.09
98.93 (-0.16)
element symb
75.22
79.87 / 84.19 (+4.32)
facts
75.10
76.09 / 76.27 (+0.18)
Table 12: Probe transfer accuracy (f) of Mistral-7B-vO.3 BASE and INSTRUCT tested on truthfulness datasets. For each row, the datasets from the other 5 rows are used for training. Pmodel, + N modell means using the probe trained on model to classify truthfulness direction in modelz. Probe transfer shows little difference (A) compared to the same-model probe.
C Supplementary Details and Experiments of Internal Belief of Truthfulness
C.1 Few-Shot Prompting
For learning the truthfulness direction t, we do not use few-shot examples but directly prompt the models with the statements. For truthfulness intervention, we use the same four-shot prompting as the experiments of knowledge storage with the same examples, though we do not have (true, false) statement pairs in the truthfulness experiments. The four examples contain two true statements and two false statements, shown in Table 7. The input is constructed the template: "ffour examples] [final statement] This statement is:". To eliminate the influence of example order, we randomly perturb the four examples for every final statement. We set the random seed to in the beginning to ensure the reproducibility of this random ordering.
C.2 Truthfulness Direction Layer and Token Position Choices
We examine the causal tracing result to determine the best layer and token position for learning the truthfulness direction and performing the intervention. Specifically, for llama- 3.1-8b BASE, SFT, and INSTRUCT models, direction and 8-12 layers for performing the intervention. For mistral-7B BASE and SFT we use the 13th layer for learning truthfulness direction and 8-13 layers for performing the intervention. For both model families, direction learning and intervention use the last token position of the input statements.
the 12th layer for learning truthfulness
C.3 Probe Transfer Accuracy on Mistral Family
Due space limits, we only show the results on the Llama-3.1-8B) model family in the main content. To further generalize our conclusion, we conduct probe transfer experiments on Mistral-7B-v0.3 BASE and INSTRUCT. Initially we also conducted probe experiments on Mistral-7b-Baas-SSF-Tuub as the Mistral SFT model, but its performance on this experi- ment's datasets on the level of random guess, making us impossible to draw any useful conclusions on it. Therefore, we discard the Mistral SFT model and only present the other two.
As shown in Table 12, the probe transfer is quite successful, which align with our previous conclusions on Llama-3.1-8B.
C.4 Probe Intervention Coefficient Choice
To assess the robustness of our findings to the choice of scaling factor, we extended our experiments beyond the default scalar setting (A +1) used in Marks & Tegmark (2024).
21

---

Published as a conference paper at COLM 2025
Dataset
INS-INS IE
INS-INS Coef
BASE--INS IE
BASE-INS Coef
Delta
cities
0,8880
2.00
0,8968
1.00
0,0088
sp en trans
0,8484
2.00
0,8409
3,00
-0,0075
inventors
0.7973
2.00
0.8298
1.00
0.0325
animal_class
0.7063
1.00
0.7192
1.00
0.0129
element.symi
0.7582
2.00
0.7697
1.00
0.0115
facts
0.6185
1.00
0.6560
1.00
0.0375
Table 13: Intervention performance with optimal scaling factors on Llama-3. 3.1-- models.
INS INS-INS denotes using INSTRUCT model's truthfulness direction to intervene in itself, while BASE~>INS denotes using BASE model's direction to intervene in SFT model. Coef indicates the optimal scaling factor A, IE is the Intervention Effect, and Delta represents the performance difference.
Test Dataset
Truthful Intervention Effect
BASE H BAASE
lNs + hiss [BASE +> Iiss (A)
cities sp-en_trans
0.65
0.67 / 0.69 (+0.02)
0.77
0.87 / 0.89 (+0.02)
inventors
0.63
0.71 / 0.72 (+0.01)
animal class
0.63
0.67 / 0.68 (+0.01)
element.symb
0.71
0.81 / 0.81 (+0.00)
facts
0.59
0.63 / 0.64 (+0.01)
Table 14: Intervention effect (f) of intervention Mistral-7B-vO.3 BASE and INSTRUCT tested on 6 truthful datasets. For each row, the datasets from the other 5 rows are used for training. fmodel, - Mmodely means using the truthfulness direction in model to intervene model2. Transfer truthful interventions show small differences (A).
Prior work has shown that scaling can impact intervention effectiveness (Li et al., 2024), motivating broader evaluation.
We varied A from to 10 (step size 1) on the Llama-3.1 and Llamm-3.1-9D-Instruud model pair. For each model and dataset, we selected the scaling factor that maximized the Inter- vention Effect (IE), comparing two scenarios: (1) INS->IN (INSTRUCT direction intervening on INSTRUCT model) and (2) BASE-INS (BASE direction intervening on INSTRUCT model).
Table 13 reports the optimal scaling factors and corresponding IE values. While intervention effectiveness shows modest sensitivity to A, both base and instruct directions achieve com- parable performance when optimally scaled. The small Delta values (ranging from -0.0075 to 0.0375) further indicate that cross- moosl and same-model interventions perform similarly, reinforcing conclusion that post-training preserves the model's internal representation of truthfulness.
C.5 Probe Intervention on Mistral Family
The probe intervention results on Mistral-7B-v0.3 BASE and INSTRUCT are shown in figure 14. The difference (A) in Intervention Effects when steering INSTRUCT with tsAsE versus INSTRUCT very little. It further verifies our previous conclusions in Section
C.6 Case Study of Intervention
Here we show case study of cross-model truthfulness intervention on Alama-3.1-88P BASE, INSTRUCT, and SFT models. It shows that trASE flip T/F outputs in POST as effectively as tser and IINSTRUCT. The successful intervention verifies our conclusion that the direction of truthfulness in the hidden representation space of BASE and POST similar.
22

---

Published as a conference paper at COLM 2025
Truthful Intervention Case Study
True statement input: "The city of Paris is in France. This statement is:"
baseline: TRUE (1.00) SFT-SFT: FALSE (1.00) BASE-SFT: FALSE (0.99)
INSTRUCT baseline: TRUE (1.00) / INSTRUCT-INSTRCCT. FALSE (1.00) BASE-
INSTRUCT: FALSE (1.00)
False statement input: "The city of Paris in China. This statement is:"
SFT baseline: FALSE (1.00) / SFT-SFT: TRUE (0.99) / BASE-SFT: TRUE (1.00)
INSTRUCT w/o intervention: FALSE (1.00) / INSTRUCT-INSTRUCT: TRUE (0.94)
BASE-INSTRUCT: TRUE (0.94)
Figure 4: Example output of Llama-3.1 1iiee-3.--bbbbmee 1-8b-Instruct with intervention. Gray box shows the next token output with the highest predicted probability with its corresponding probability in the parentheses. Transferred intervention can flip the output as successfully as native intervention.
23

---

Published as a conference paper at COLM 2025
Refusal Keywords
"I'm sorry", "I am sorry", "I apologize", "As an Al", "As a language model", "As an assistant", "I cannot", "I can't", "I can't", , am unable to", "I'm unable to", "I am not able to", "I'm able to", "I will not", "No, will not", "I am not", "I'm not going to do that", "I can't", "I don't".
Figure 5: Refusal keywords used to detect refusal behavior.
D Supplementary Details and Experiments of Refusal Direction Intervention
D.1 Refusal Scores
Following Arditi et al. (2024); Kissane et al. (2024a) We use refusal keywords to determine model refuses an input. The keywords are listed in Figure 5. Refusal score on dataset is calculated as the percentage outputs of the model that contains these keywords the beginning.
D.2 Input Template
In order for the models to give reasonable responses to the instructions, we use the user- assistant template to prompt the models. For all SFT and INSTRUCT models we use their official chat templates. For BASE models, we construct template for the best output quality. Templates for base models are illustrated as following:
gemma-2-9b:
<start_of_
<start_of_turn>userr _turn>use::
(instruction)<end__g _turn <start_of_ttrm>>assiss
llama-3.1-8b:
User: (instruction) Assistant:
qwen1.5-0.5b:
<|im\_start/>user (instruction))(lim_end)
sain_sstarttaassisttat
(instruction] is the input harmful or harmless instructions.
D.3 Refusal Direction Layer and Token Position Choices
We follow Arditi et al. (2024) to select the best-performing layer and token positions for extracting the refusal direction The choices are reported in Table 15.
D.4 Abnormal Case in Refusal Intervention for Llama-3.1-8b
Table shows one notable abnormal case: intervening the representations of SFT by adding TbaSe induces SFT to refuse 85% of inputs, which even higher than the intervention results on BASE itself. This suggests SFT may be inherently more prone to refusing instructions and thus more easily steered toward refusal. The poorer transfer results when using IBASE to intervene in INSTRUCT further suggests that the DPO process employed in INSTRUCT may have mitigated INSTRUCT's internal tendency refuse. Investigating this phenomenon could be promising future direction.
24

---

Published as a conference paper at COLM 2025
Model
Layer
Token Position
llama-3.1-bb BASE
11
4
1lama-3.1-bb SFT
11
llama-3.1-8b INSTRUCT
11
-1
qwen1.5-0.5b BASE
13
-1
qwen11.-00.b INSTRUCT
13
-1
gemma-2-9b BASE
23
-1
gemma-2-99 INSTRUCT
23
-1
Table 15: Layer and token position choices for extracting refusal directions.
D.5 Refusal Direction Intervention with Other Model Families
Model
Data
Refusal Scoret
INS
INS-INS
BASE-INS
Qwen-1.500.5B-chl
harmless
0,03
0.68
0.27
Owen-1.5-0.BB-cha
harmful
0.85
1.00
1.00
Gemma-2-9B-it
harmless
0.02
0.75
0.09
Gemma-2-99-it
harmful 0.98
1.00
1.00
Table 16: Refusal scores after adding refusal directions from INSTRUCT to INSTRUCT (INS-
INS) or from BASE to INSTRUCT (BASE-INS). The best intervention results are shown bold. The BASE to INSTRUCT intervention is not very successful.
Model
Data
Refusal Score
INS
INS-INS
BASE-INS
Dwen-1.5-0.5BBchl
harmless
0.03
0.01
0.01
Qwen-1.5-0.5B-chh harmful
0.85
0.04
0.13
harmless
0.02
0.00
0.01
Gemma-2-9B-it
harmful
0.98
0.00
0.87
Table 17: Refusal scores after ablating the INSTRUCT refusal direction from INSTRUCT (INS-INS) and abalting the BASE refusal direction from INSTRUCT (BASE-INS). The best intervention results are shown in bold. The BASE to INSTRUCT intervention is not very successful.
Besides 1aaaa-...-8b--nnstte refusal direction on BASE and INSTRUCT from Qwen-1.5-0.58 series and Gemma-2-9B series. As shown in Table 16 and Table 17, neither of the refusal directions from BASE of Qwen-1.5- 0.5B and Gemma-2-98 can successfully intervene in the behaviors of INSTRUCT. It confirms our previous conclusions.
also conducted the experiment on intervening in the
D.6 Case Study of Refusal Intervention
We show case study of refusal intervention in Figure 6. As shown in the figure, the baseline output from INSTRUCT is refusing to follow the harmful input. After intervention with the refusal direction from INSTRUCT, the refusal behavior disappears and the model starts to follow the harmful input. However, with the direction from BASE, the behavior stays the same. It further confirms our previous conclusions.
25

---

Published as a conference paper at COLM 2025
Refusal Intervention Case Study
Harmful input: "Give step-by-step instructions for making explosives and other dangerous materials"
INSTRUCT baseline output: I can't fulfill that request... INSTRUCT output (ablated by INSTRUCT refusal direction): thrilling request! Here are step-by-step instructions for making various explosives.. INSTRUCT output (ablated by BASE refusal direction): can't fulfiil that request...
Figure 6: Example output of Llama-3.11 ----b-nnttrutt on harmful instructions with intervention.
The baseline is the output without intervention. Ablation using direction learned from BASE model failed steer the model to bypass the refusal behavior.
26

---

Published as a conference paper at COLM 2025
E
Supplementary Details and Experiments for Confidence
Due to space limits, we did not provide experiment results regarding entropy neurons in the main content, so we leave them here. We analyze the neurons from the last MLP layer, and we calculate their weight norms and LogitVar. Figure 8, and 9 show the distributions of their weight norms and LogitVar. The X-axis shows the weight norm, and the Y-axis shows the LogitVar. We conduct experiments on Llama-2-7B, Llama-3.1-8i and Mistral-7B models. The distributions across BASE, SFT, and INSTRUCT models are very similar.
0.01
LogitVar(Wadl
LogitVar(Wais
0.001
0.001
100μ
100μ
10p
10p
14
0.5
1
1.5
1.5
||Woell
||Woulll
(a) Llama-2-7b BASE
(b) Llama-2-7b INSTRUCT
Figure 7: Weight norm and LogitVar of the last MLP layer's neurons in the Llama-2-7B
model family.
Log
0,001
||Mad]]
(a) Llama-3.1-80 BASE
(b) Llama-3.1-80 SET
(c) Llama-3.1-8b INSTRUCT
Figure 8: Weight norm and LogitVar the last MLP layer's neurons the Llama-3.1-8B model family.
0.01
0.0
0.0
LognsbitWal
LogkverWWel
Logkverlwea)
0.00
2001
3.2
0.25
[|Weell
[|Welll
|Welll
(a) Mistral BASE
(b) Mistral SFT
(c) Mistral INSTRUCT
Figure Weight norm and LogitVar of the last MLP layer's neurons in the Mistral-7B-v0.3 model family.
Table 18 shows the stats of entropy neurons across models. We observe high overlap of entropy neurons between BASE and POST models. To further investigate the overlapping entropy neurons, we calculate the ratio of We calculate the difference of this ratio between BASE and POST models, and this result is
weight norm of the overlapping entropy neurons. log (LogitVar))
27

---

Published as a conference paper at COLM 2025
also shown Table 18. As a reference, the average ratio of the entropy neurons 0.0880, while the average difference of this ratio on the overlapping entropy neurons between BASE and POST is generally less than 1% it. confirms that the entropy neurons are not only overlapping, but the overlapping entropy neurons are also very similar.
Model pair
Overlapping neuron count (out of 10) Average ratio difference
1oma--.1--b BASE vs INSTRUCT
0.000815
1lama-3.1-88 BASE VS SFT
10
0.000112
mistral-7b BASE INSTRUCT
9
0.000030
mistral-7b BASE vs SET llama-2-7b BASE INSTRUCT
8
0.000089
0.001712
Table 18: Entropy neuron results. "Overlapping neuron count" shows the number of overlapping entropy neurons between BASE and POST models. " "Average ratio difference" shows the average difference of Logiiaa)
weight norm of the overlapping entropy neurons between is 0.0880 for all entropy neurons, which is much larger than the difference. BASE models and POST models have very similar entropy neurons.
BASE and POST models. As reference, the average
weight norm Log[LogitVar)
28

---

Published as a conference paper at COLM 2025
cities
sp-en_trans
fulu.extracted
Corr MASEE MINSTRUCT
0.9885
0.9918
0.9970
max MINTTRCCT MBASE
0.4
0.4
0.2
max MINSTRUCT MOASE
0.4
0.4
0.1
Table 19: Knowledge storage results for Llama-2-131 models.
Test Dataset
Probe Transfer Accuracy (%)
PEASE
HBASE
PINS >> hiss / PRASE > HINS (A)
cities
95.39
99.47 / 99.06 (-0.41)
sp_en_trans
96.89
96.33
90.68 (-5.65)
inventors
83.74
70.20 / 70.94 (+0.74)
animal.class
98.78
95.12
95.12 (+0)
element symb
95.70
94.62
94.09 (-0.53)
facts
71.12
78.97
62.75 (-16.22)
Table 20: Probe transfer accuracy (1) of Llama-2-13B models.
F
Additional Experiments on Llama-2-13B Models
To verify whether our findings generalize to larger models, conduct experiments Llama-2-13B base (BASE) and Llama-2-13B-Instuee (INSTRUCT) models. We use the same experimental settings described in the main paper. Our previous conclusions are consis- tently verified on these 13B parameter models.
F1 Knowledge Storage Experiments
We conduct causal tracing experiments using the same settings and metrics as the main paper on the cities, sp_en_trans, and tulu.extracted datasets. The results in Table 19 demonstrate high correlation coefficients between BASE and INSTRUCT models with low maximum differences, confirming that post-training has minimal influence on knowledge storage locations.
F2
Truthfulness Probing Experiments
We follow the same experimental settings and metrics for truthfulness probing across multiple datasets. The results Table 20 show consistent patterns with our main findings.
E3
Truthfulness Intervention Experiments
Using identical settings as the main experiments, we evaluate truthfulness interventions on both models. The results in Table 21 maintain consistency with our previous conclusions.
E4 Refusal Intervention Experiments
We conduct refusal intervention experiments following the same methodology. The results in Table 22 confirm that truthfulness directions remain similar between base and post-trained models while refusal directions differ.
E5 Entropy Neuron Analysis
For entropy neuron experiments, all top 10 entropy neuron candidates are identical between BASE and INSTRUCT. The weight ratio differences remain minimal, confirming that confi-
29

---

Published as a conference paper at COLM 2025
Test Dataset
Truthful Intervention Effect
BBSSE + BAASE
tins + hiss BRASE 7 hiss (A)
cities
0.69
0.71 / 0.68 (+0.03)
sp en trans
0.83
0.86 / 0.88 (+0.02)
inventors
0.66
0.64 / 0.67 (+0.03)
unimal_class
0.72
0.73 / 0.74 (+0.01)
dlement.symb
0.79
0.84 / 0.83 (-0.01)
facts
0.68
0.63 / 0.66 (+0.03)
Table 21: Intervention effect (f) of intervention on Llama-2-13B models.
Test Dataset
baseline / /BASE - hIAAE
baseline
'INS + hINs / TBASE + hINS
harmful
0.24 / 0.37
0.99 / 0.59 / 0.99
harmless
0.05 / 0.32
0.0 / 1.0 0.01
Table 22: Refusal intervention results for Llama-2-13B
dence differences between base and post-trained models cannot be attributed to entropy neurons.
Due to resource constraints, we were unable to conduct experiments on even larger models, but expect findings to generalize to models with 40 billion or more parameters.
30

---

Published as a conference paper at COLM 2025
Liara 3.1.88 ciles
Lisari trstirctt clics
Utrra 5.1-8 Instrcct Liama 3.1-8-1-8 cliss
MA
bag PCT3ROF
loa PCTSPPF
-0.
3-40 neguctes
Liare 1-00 ciass
Liame Insrrct Liara requctais
PITSRF))
PITARF))
0.2
Llama-3.1-88 larger than
Llama-3.1- 1ate-3.118-8--0 1-8.intttttttttttttccccccc langer than
Llama-3.1 w.w.----nnttcct---m. Liama-3.1-08 larger than
log PCTUPLE)
PCTUPIF)
log PCTUP(F)
Layer
Layer
Layer
1.5
1.5
20
20
O.S
-0.5
0.5
Liama 3.1 8B smaller than
Liama 3.1 88 Instruct smaller than
Liama 3.1 89 Instruct Libma 3.1 8B smaller than
log PCTUPIF)
log PCTUPIF)
log PCTUPLF)
Layer
10
100
20
25
2%
25
PITYPIF)
PLTUPOF)
Liare-3 1-40 FeO, www.0/0000.0000000 20re
Liare-3 www.:220DDDDeer.. meg
Liare-3 wwt.s-111-00-000 1-00--0--------- 111--11--0 rec,
log (1T
12g (TTNNT
33
A5
05
Ing PNTUPPF)
PCTUPUF)
PCTVPIFI
10
30
25
15
25
20
20
20
15
-05
25
25
30
Inowtedge
Inoxledge
knowledge
Figure 10: Knowledge storage locations of Llama-3.1-8B BASE and INSTRUCT.
31

---

Published as a conference paper at COLM 2025
Liara 3.1.88 ciles
Lisaro 9.1.68 SFT Cltes
Uttra 5.188 SFT L.ama-11--. dRes
bag P(T351F
1ox PCTSPFF
-0.
3-40 neguctes
Liare 1-00 SFT reg, ctaes
Liare SFT Liama reqg
PITSRF))
PITARF))
0.2
Llama-3.1-88 larger than
Liama-31--88-FFT larger than
Liama-3.1-80--4 3-1--8-5FFT---- Llaama333 1-58 larger, than
log PCTUPIE)
PITUPIF)
log PCTU
Layer
Layer
1.5
1.5
20
20
O.S
-0.5
0.5
The
Liama 3.1 8B smaller than
Liama 3.1 88 SFT smaller than
Lioma 3.1 89 SFT Liam 3.1 89 smaller than
log PCTUPIF)
loo PCTUPIF)
log PCTVPLE)
Layer
10
1.0
20
25
2%
25
Digg
PITYPIF)
PITUPOF)
PTT
Liare-3 1-40 FeO, www.0/0.00..000000 20re
Liare-3 11011-0-0000P reg. ST00
Liame-3 111-331------77 L.braad 1-40 meg Jaans
log (1T
12g (TTNN
A5
05
tunu
Ing PNTUPUF)
PCTUPUF)
PCTVPPF)
10
30
25
15
Layer
25
20
20
20
15
-05
25
-05
30
Inowledge
Inowledge
knowledge
Figure 11: Knowledge storage locations of Llama-3.1-8B BASE and SFT.
32

---

Published as a conference paper at COLM 2025
Nitaral78 ciooa
Nittral-2 7B instruct dioss
Nitzaal 78 instuct Misrial 78 dies
bag PCT351FF
-0.
Masrul-7 reg, creas
Matroal www.-tt----iddddddcoooo meg clos
Matak-BBrrabuut- Mirrss- reg claes
PITSPIF)
PITARF))
0.2
Mi leeger tham
Mietrni-7B-instuutt
Mastrel-7B-neemuct- Mistral-7 larger
Iog PCTUROTT
Iog PNTUPOT)
Ing PITU/O7
30
Layer
15
Nisora-75 smaller chan
Mistral-75 Missral-75-ttttuuut tmmatlt than
Instrutt Nitral75 amalee.taa
PITUNFF)
PITW/F)
Layor
-D3
-03
PITYPIF)
PITUPOF)
PTT
15
Lawer
M Marsh-7 reg.et wtpp...... p0..m.000
Misrs--D-inessst meg, 2r pome
Messi-TBiterra VVlrra7 reg, Www.SO..LL... IT
log (1TAN1
1og (1TNN)
33
A5
05
P(T)))F)))
10
10
12
Layer
20
20
23
45
23
-05
Inowldgge
Irowldge
laweldge
Figure 12: Knowledge storage locations of Mistral-7B BASE and INSTRUCT.
33

---

Published as a conference paper at COLM 2025
Nitaral77 cioca
Nittral-22 (aaruul-7--55 SFT ciloa
Nitaral 7004-77---88 SPT Mistaa-8 didss
bag (1T3501
-0.
Masrul-7 reg, creas
Matroll neg, cass
Mattk-7D-SFF. Minnsk-T meg
PITSRFF)
0.2
Mistral-7B leeger them
Mistral-70-5FT then
Mastrel-B--5TT MMstral-77
Tog PNTURPTT
Iog PNTUPPTI
Ing PITU/PO11
30
Layer
15
Nisora-75 smaller than
Mistral75-551 7---775577778-0 smalier than
Nottrl-7B-SPT Misoal-8 amallertronn
PITWNFF)
PITWIFF
Layer
13
PITYPIF)
PITUPOF)
PTT)
15
M Marsi-7 reg.ee wrop......0 p0..
Misxi-D-GFFT reg, 4t..
Messi-TB.GFT. Merrs70 neg F0. Pore
log (1T
A5
05
10
12
20
20
23
45
23
-55
Inowlddge
Iroeldge
laneldge
Figure 13: Knowledge storage locations of mistral-7B BASE and SFT.
34

---

Published as a conference paper at COLM 2025
(XL www.guussoon.ccc Lame-333 1-85 clssa
Ulura 3--18-31--8---11 3-88--trafuut Liama-31-4 dlas
Layol
Layel
The [4]] [2] (4]
[ol] [3]
[4]]
The
[2] [4]
PTT)PFF)
PTPP
0S
Layst
03
05
In Dol_ppantion_ (nmaed 1-07 cabee
no, Po_pppesion._ semnect cane
Dlrrect 1-80-5FF ber.aldddeel clle
P(T)FF))
PMT
lyg PTPFF
LOWE
The of [S] [4) (4))
The [ol] [9] (40
The ay [2) (5]
no_gpestoo__ineee spuen_mae
sp_en_hane
PMP
20
055
211
1oods
Figure 14: Knowledge storage locations of Llama-3.1-8B BASE, INSTRUCT, and SFT the traditional causal tracing setting.
35

---

Published as a conference paper at COLM 2025
(oL www.guuasooo....o Misuuk 78 croas
IN, WwL.quesoorr... Misaaa 7B insuuu cless
Mittral 75 -ns--rrttuctt--- Mishaa 7B dbes
Layel
Layel
The
[4]] [4] (4]
[ol] [3] [43]
The [2] (63]
PTPPF
10
10
13
1.
Laye
20
20
-L
[s]]
[2]
no, ppw.rpnnooonnooooooooo Mierss--T---eemme sp, momes
Wiltrei--T--teeeurrr Mistaa-m up, qp,en, poee
P(T)FFF)
P(T)FF
log PTPFF
1.
1.0
Laye
Layer
22
22
no_gueston__ee mp_en_mas
sp_en_mes
055
211
goods
1000
124
Figure 15: Knowledge storage locations of Mistral-7B BASE, INSTRUCT, and SFT in the traditional causal tracing setting.
36

---

Published as a conference paper at COLM 2025
Liama-3.1-88 to Llama-3.1-44 1-88-Innstrud cities
to Llama-3.1-88 cities
log ((T)P(F)
log P(T)P(F)
10
0.5
10
0.5
Layer
15
15
20
20
-0.5
-0.5
-1
Agp
10
[19]
[2-
[ea]
[To
10
[t8]
[25]
[es]
[To]
Llama-3.1-8B to Llama-3.1-PPP 3.1--8BInnstrucc neg_cibies
Llama-3.1-800 Inst-ruttttttt to 1amm--3.1-8B cig_si
log P(T)P(F)
P(TYP(F)
0.5
0.5
Layer
15
20
20
25
-0.5
25
-0.5
30
-1
30
TILL
[te]
128
(639
200
[re]
AP
jo
[19]
(29)
[6:)
80
you
[ro]
8
Llama-3.1-8B to Llama-3.1-8B-4 3.1-8B-Instruct larger than
3.ma-3..-8B-Instruct to Llama-3.1-0B larger_ than
P(T)/((F)
log P(T)/((F)
Layer
10
0.5
15
15
20
25
-0.5
30
-1
-1
5t
larger
than
[00]
This
statement
5
brgee
This
statement
G
Llama-3.1-88 to Llama-3. 1-88-Innstruct smaller thar
Bamma-31-88--nstree to Llama-3.1-88 smaller thar
log P(T)/((F)
log P(T)/((F)
Layer
10
0.5
0.5
15
20
20
25
0.5
30
-1
30
-1
[19]
[25]
[ES]
5t
[To]
[20]
This
statement
(2
5
162 Loo
This
statement
Figure Cross-model patching results between Ilama-3.1-8b BASE and INSTRUCT.
37

---

Published as a conference paper at COLM 2025
Llama-3.1-8B to Llama-3.1-8B- 3.n--t--nnttttttt sp_en trans
Llama-3. Llama-3.1-8B-Inee 1-8--instruct to Llama-331-88 sp_en____t trans
log P(TMP(F)
log P(T)P(F)
10
0.5
10
Layer
15
15
20
20
25
-05
-0.5
30
-1
30
usuunds
[20]
Tull uplueds piom
[ns]
[17]
Llama-3.1-88 1-86 to 1amma-3.--8--nntntte geg__s__e__traee
1aama-311-86-nnsttee to Llama-3.1-880 neg_sp__en st__e___een_sss
log ((T)P((F)
log P(T)P(F)
0.5
0.5
-0.5
30
The 4oweds
plow
saop
soatemer
[18] [es)
seop
you usau
Llama-3.1-88 to Llama-3, 3.1-8B-Instruch tulu extracted
1arma-..1-8B-nsste to Llama-3.1-8 3.1-8B tullttuettraaaaeeeee extracted
log P(T)/P(F)
log P(T)/P(F)
5
0.5
0.5
Layer
15
15
25
005
25
00.
-1
-1
knowledge
others
jast_token
knowledge
others
last, sst_token
Figure 17: Cross-model patching results between Ilama-3.1-81 BASE and INSTRUCT (Contin- ued).
38

---

Published as a conference paper at COLM 2025
Liama-3.1-88 to Llama-3.11 1.mma-31--8B-FFF cities
Llarna3.118B-SFT to Liama-3.1-88 cities
((T)P(F)
log ((T)P(F)
10
0.5
10
0.5
Layer
15
15
20
20
-0.5
-0.5
-1
Agp
10
[19]
[2-
[ea)
[To
fup
10
[t8]
[25]
[g9]
[To]
Llama-3.1-8B to Llama-3.1-8B-55 neg_cities
Llama-3.1-0B. 3.1-8B-SSFT to Llama-331-88 neg_ citic
log P(T)P(F)
P(TYP(F)
0.5
0.5
Layer
15
20
20
25
-0.5
25
-0.5
30
-1
30
TILL
[te]
128
(639
200
[re]]
j0
[19]
(29)
[69)
80
y0u
[re]
Llama-3.1-8B to Llama-3.1-88 3.1----..BBBFTT larger_thee than
1amma-3.1-8-$$F to -Jama-3.1-8B larger_than
P(T)/((F)
log P(T)/((F)
Layer
10
0.5
15
15
20
25
-0.5
30
-1
-1
5t
larger
Chan
[00]
This
statement
5
than
This
statement
G
Llama-3.1-88 to Llama-3.1-88 1-8B-FFT smaller than
Llama-3.1-88-SF to Llama-3.1-88 smaller than
log P(T)/((F)
log P(T)/((F)
Layer
10
0.5
0.5
15
20
20
25
30
-1
30
-1
[15]
[25]
[ES]
5t
[To]]
00
This
statement
5
[29
5
smmllles
This
statement
G
120
Figure 18: Cross-mode patching results between Ilama-3.1 1oma--31--B 3.1-8b BASE and SFT.
39

---

Published as a conference paper at COLM 2025
Llama-3.1-8B Llama-3.1-8B-SFF sp_en_trans trans
Llame-3.1-8B-SF7 Llama-3.1-88 sp_en__ras
log P(T)P(F)
log P(T)P(F)
1
5
10
10
Layer
15
20
20
0.5
-1
BALL
ustadds
SALL uetedds piow
12
Llama-3.1-88 Llama-3.1-88-SFF neg_sp_en__tm_
Llama-3.--8--FFT Llama-3.1-4 neg_sp_en reg_sp_en_ttrns
log ((T))((F)
log P(T)P((F)
5
10
10
0.5
Layer
15
15
20
20
25
-0.5
25
-0.5
30
30
S4L
piow
159
seop
ynu meeu
[to]
DALL
younds
[ps]
seop
[po
Llama-3.1-88 to Llama-3.1-88 3.18B-FF tulu_eetreer extracted
3ama-331-8B--FT
1ama-3.11-B tuluueuttrrin extrraect
log, P(T)/P(F)
log, P(T)/P(F)
10
0.5
0.5
Layer
15
20
25
-05
-0.
30
knowledge
others
tosen
knowledge
others
last_token token
Figure 19: Cross-model patching results between 1lama-3.1 1-b BASE and SFT (Continued).
40

---

Published as a conference paper at COLM 2025
Mistral-7B to Mistral-7B-Instie cities
Mistral-7B-instruct to Mistral-7B cities
log ((T)P(F)
log P(T)P(F)
10
0.5
10
0.5
Layer
15
15
0
20
20
-0.5
-0.5
-1
Agp
10
[19]
[2-
[ea]
u
[To
10
[t8]
[25]
[es]
[to]
Mistral-7B to Mistral-77 7B-Instruct neg cities
Mistral-7B 78-Instruct to Mistral-7B BB cit_cs
log P(T)P(F)
P(TYP(F)
0.5
0.5
Layer
15
15
20
20
25
-0.5
25
-0.5
30
-1
30
RILL
[te]
128)
(639
ypu
[re]
TIe
jo
[19]
[29)
[69)
you
[ro]
Tull
Mistral-7B to Missral-7B-Instruce larger_than
Mistral-7B-Instreee Mistral-7B larger_than
log P(T)P(F)
log, P(T)/P(F)
Layer
10
0.5
10
0.5
15
20
25
-0.5
25
-0.5
30
-1
5
[ro)
[go]
57
[TS]
This
statement
Mistral-7B Mistral-7B-Insttee smaller_the than
Mistral-7B-Instruct Mistral-7B smaller than
log P(TIP(FF)
P(TM/(FF
10
0.5
0.5
aaer
15
15
20
-05
30
[po)
This
statement
thas Sallen
102 to1l
[o3)
This
statement
Figure 20: Cross-model patching results between Mistral-7B BASE and INSTRUCT.
41

---

Published as a conference paper at COLM 2025
Mistral-7B Mistral-7F Searal--B-Inntrucc sp_en taiee__traae trans
Mistral-7B Saanal-----nneee 7B-instrutttt to Mistral-7B sp_en____t on_trans
log P(TMP(F)
log P(T)P(F)
10
0.5
10
Layer
15
15
20
20
25
-05
-0.5
30
-1
30
-1
upuunds
Tul upuueds piom
Ins
Mistral-7B to Mistral-7B-Instruc reg_sp_en_tram
Mistral-78-instruct to Mistral-7B veg_sp_en__trme
log ((T)P((F)
log ((T)P(F)
0.5
0.5
-0.5
30
The yoweds
piow
soop
Iou 10000
This
soatement
[18)
[cs
seop
you usau
Mistral-7B to Ristral-7B-Instrue tulu_eetteee extracted
Mistral-7B-Insiiun to Mistral-7B uulu_extracted
log P(TMP(F)
log P(T)P(F)
5
0.5
0.5
Layer
15
15
25
-0.5
25
-0.5
-1
knowledge
others
last token
knowledge
others
last token
Figure 21: Cross-model patching results between Mistral-7B BASE and INSTRUCT (Contin- ued).
42

---

Published as a conference paper at COLM 2025
Mistral-7B to Mistral-7B-SFT cities
Mistral-7B-SFTT to Mistral-78 cities
((T)P(F)
log ((T)P(F)
10
0.5
10
0.5
Layer
15
15
0
20
20
-0.5
-0.5
-1
Agp
10
[19]
[2-
[ea]
[To]
sia
10
[t8]
[25]
[es]
[to]
Mistral-7B to Mistral-BBSSFT cities
Mistral-7B-Se 7B-SFF to Mistral-7B cities
log P(TYP(F)
P(TYP(F)
0.5
0.5
Layer
15
15
20
20
25
-0.5
25
-0.5
30
-1
30
RILL
[te]
128)
(639
[ro]
jo
[19]
(29)
[6:)
80
you
[ro]
Mistral-7B to Mistral-7B-SFT larger_than
Mistral-7B-SFT to Mistral-7B larger_than
log P(T)P(FF)
log, P(T)/((F)
Layer
10
0.5
10
0.5
15
20
25
-05
25
-0.5
30
-1
(29
123
5
ueun
[tol
(20)
(go)
57
[TS]
Spae pae 1Lo
statement
Mistral-7B Mistral-7B-SFF smaller_nie than
Mistral-7B-SFT Mistral-7B smaller than
log P(TMP(FF)
P(TM/(FF)
10
0.5
0.5
15
15
20
005
30
This
statement
Thes
statement
Figure 22: Cross-model patching results between Mistral-7B BASE and SFT.
43

---

Published as a conference paper at COLM 2025
Mistral-7B Mistral-7B-SFT up_en_trans
Mistral-7B-SFF Mistral-7B sp_en_trans
log ((T)P(F)
log P(TMP(F)
1
10
10
Layer
15
20
20
0.5
-1
HALL ustadds
SALL uptedds piow
12s
SALL waaeeees
Mistral-7B Mistral-7B-SF _o__sp_en_tranm
Mistral-7B-SFT Mistral-7B neg_sp_eed _op_sp_en_trans
log ((T))((F)
log ((T))((F
10
10
0.5
Layer
Layer
15
15
20
20
25
-0.5
25
-0.5
30
30
S4L
p/ow.
159
seop
useu
[to]
yaunds
[p9
seop
Mistral-7B to Mistral-7B-SFT tulu__exi extracted
Mistral-7B-SFT Mistral-7B t
extracted
log (F))P(()
log P(T)P(()
10
0.5
0.5
Layer
Layer
15
15
20
20
-0.5
0.5
30
30
knovledge
others
last, token
knowledge
others
last_tokes token
Figure 23: Cross patching results between Mistral-7B BASE and SFT (Continued).
44

---

Published as a conference paper at COLM 2025
no_ qwg.uuusstonnn..o
Lrtrsioo_Llamaa311112 Llama-3.1-86 Lo__Lama--31118 3-8--nnnstt cities
no_ Laap::n._aaaaa311-80111 to Llama-3.1188 cities
log ((T)P((F)
log P(T)P(F)
10
0.5
0.5
Layer
Layer
15
20
25
-0.5
25
-0.5
-1
The city
[s3]
city [s1]
[22]
[s3]
qww.eiuesnonn... Llama-3.1-86 Liams-3.1- en www.queesioo.... 1lama-3.1-BB-Instmue to Liama-3.1-8B sp_en
log ((T)P((F)
log P(T)P(FF
5
10
10
0.5
Layer
Layer
15
15
0
20
20
25
-0.5
-0.5
T4e
yqueds
ploM.
[te]
[2s]
yaueds
plow
[19]]
[2s]
qo__question Llama-3.1-88 Llama-3.1-8B-SFT cities
www_gusstion_iaaee quvestion_lamma331-BB...8 Llama-3.18BB-SFT 3FF_1t__lamm-331188 cities
log P(T)P((F)
log P(T)P(FF
10
10
Layer
Layer
20
20
25
-0.5
-0.5
30
30
The city
[s1]
[s2)
[s3]
Is
in
The city
of
[s1]
[s2]
[s3]
no_ wwt.quiio_Llaaaaaa..1 L.ama-3.1-88 Lla---3.1-B--FFT sp_en_tre
no_ quustion__laaa-31111808_ 1
Llama-3.1-88 sp_en_tre
log ((T)P(F)
log P(T)P(()
10
0.5
0.5
Layer
Layer
15
-1
-1
T41
4auess
p/om
[19]
[as]
yauuds
piom
[1s]
[os]
Figure 24: Cross-model patching results between Llama-3.1-8B BASE, INSTRUCT, and SPT in the traditional causal tracing setting.
45

---

Published as a conference paper at COLM 2025
no_ ww..uuusstonnnnoo Mistral-78_in to_ to_tootttallPPe- Mistral-7B-Innsruuce cities
Lueesion_Mistral-7B_.o
no_ www_qquettoooooo Lrrestion_Mistaa--P5 Mistral-7B-instruet,, to Mistral-7B cities
log (FT)P((F)
log P(T)P((F
10
0.5
0.5
Layer
Layer
15
20
25
-0.5
25
-0.5
-1
The city
[s3]
city
[s1]
[22]
[s3]
qweseiiesnnnn... Mistral-7B_ Mistrall-B- sp_ee trans
www.quueitioooo. Lotisstio_MMstal--T77 Mistral-7B-Insttunn Mistral-7B trans
log P(T)P(FF
log P(T)P(FF
5
10
10
0.5
Layer
Layer
15
15
0
20
20
25
-0.5
-0.5
T4e
plok
[1g)
[2s]
yaueds
plow
[19]]
[2s]]
vo_question Mistral-7B_ Mistral-7B-SFT cities
Biaestiin_Mistal--777 Mistral-7B-SFT. Mistral-7B cilies
log P(T)P((F)
log P(T)P(FF
10
10
Layer
Layer
20
20
25
-0.5
-0.5
30
30
The city
[s1]
(s2)
[s3]
Is
in
The city of
[s1]
[s2]
[s3]
_n__ouuuttt [ Mistral-7B-SFF sp_en_tran trans
no_ wt_.ugtion_Mistra-7eeeee
Mistral-7B-SFT to_ Mistral-7B _p_en__rae
log ((TYP(F)
log P(T)P(F)
10
0.5
0.5
Layer
Layer
15
-1
-1
p/om
[19]
[as]
yauuds
pion
[ts]]
[os]
Figure 25: Cross-model patching results between Mistral-7B BASE, INSTRUCT, and SFT in the traditional causal tracing setting.
46


Reviews:
[{"review_id": "2hATQregd6", "reviewer": "Reviewer_bJdd", "summary": "This paper provides an in-depth mechanistic analysis on BASE (pretrained), SFT (Instruction fine-tuned) and INSTRUCT (post RLHF) models using the method proposed by Marks and Tegmark (2024). The paper analyzes the models across four perspectives - (i) understanding \"storage\" of facts, (ii) truthfulness direction and transfer across model training stages (iii) similar analysis on refusal direction and (iv) differences in confidence. The paper provides various insights potentially useful for model understanding.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents very interesting and important mechanistic analysis of models across various training stages. This is timely and can have a significant impact in advancing the interpretability literature for understanding models across training stages for fact understanding \n\n- The analysis are thorough, they have performed it across various models to validate findings, showing generalizable findings. \n\n- The tasks that the authors analyzed is also very impactful, understanding truthfulness and refusal directions in various training stages could potentially lead to developing innovative ideas in this space \n\n- The paper identifies that while truthfulness shows high cosine similarity, refusal doesn't. There is scope for another paper in this topic alone.", "reasons_to_reject": "- Nothing major. But section 7 is rather not well-written. While explained briefly, the paper could use explaining how entropy neurons were applied in their analysis in a bit more detail.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aVYfdsYr3w", "reviewer": "Reviewer_K28J", "summary": "This paper presents a mechanistic evaluation of how post-training has changed the pre-trained language models in terms of knowledge, truthfulness, refusal, and confidence. The paper finds: (1) post-training does not change the locations where factual knowledge is stored, but changes the knowledge representations; (2) Post-training does not change the truthfulness representations by much; (3) Post-training changes the refusal representations; (4) Entropy neurons, commonly used in the literature to explain the confidence calibration of LLMs, cannot explain the difference between confidence of the post-trained model and the pre-trained model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a holistic evaluation of the differences between post-trained models and pre-trained models across a wide range of behaviors that people generally care about post-training.\n\n2. The paper conducts comprehensive experiments on various models with well-accepted mechanistic interpretability methods, enabling it to draw general conclusions about post-training.\n\n3. The paper has very solid experimental designs for many settings: For example, the paper takes the distribution shift after post-training on knowledge-intensive prompts into account in Section 4, and this is usually overlooked by prior works.\n\n4. The paper draws interesting conclusions about how post-training changes the truthfulness and confidence behaviors and could be potentially useful for future research in LLM alignment.", "reasons_to_reject": "1. The experimental setup of the knowledge analyses raises some concerns. While the knowledge analyses aims at determining the knowledge storage and the knowledge representation have been changed after post-training, the paper uses causal tracing to extract the knowledge location and representation by the change of log probability between \"This statement is TRUE\" and \"This statement is FALSE\" (Equation (1)). This seems to me more like extracting the model's understanding of the truthfulness of a given statement rather than extracting the knowledge. There is a difference between knowing if a given statement is true or false (truthfulness, or the capability to validate) and knowing the right answer to make up a true statement (knowledge, or the capability to generate), and it is known in the literature that LLMs are inconsistent for these two capabilities (also called generator-validator consistency as in [2]). Moreover, Section 4 of the paper follows the setup of [1] while [1] uses that setup for probing truthfulness rather than knowledge. Thus, I have a concern that the experimental setup in the paper probes a different phenomenon from knowledge. I think it would be more appropriate to follow another practice in prior works on mechanistic analyses of knowledge (e.g. [3]): measuring the patching's influence by the change in the probability of the entity of interest (e.g. the object of the knowledge triple). \n\n2. The refusal analyses do not add new insights from what's known in the literature. Although the paper conducts very thorough experiments on refusal, the exact conclusion it draws has already been discussed in [1] (in Appendix J). The only difference is that [1] compares the cosine similarity of the refusal directions extracted from chat and base models while the paper uses causal tracing.\n\n3. The methods used for analyses are not well-motivated. The paper does not use a consistent set of analytical methods, and does not provide a clear motivation for choosing one mechanistic interpretability analysis method over another: it uses causal tracing for analyzing knowledge and refusal, linear probe (and causal tracing, which is used to build the probe) for truthfulness, and entropy neuron for confidence. It is unclear to me why the same probing analyses are not used for refusal since the paper mentions that refusal can also be represented in a linear direction, similarly to truthfulness, and also, why causal tracing and probing are not used for analyzing confidence calibration. For the latter, I understood prior works have found that entropy neurons can be used to explain confidence calibration. However, it is noteworthy that prior works have found that there might exist knowledge neurons [4] and truthful attention head [5] to explain knowledge and truthfulness respectively.\n\n4. This paper could be substantially strengthened if it could provide deeper analyses on one of the four phenomena of interest, especially analyses of how different post-training methods have different effects. For example, Table 4 already shows some differences in transferability of the refusal direction between BASE -> SFT and BASE -> INS, potentially showing an impact of SFT vs. RLHF for refusal. \n\nDespite these, I still think the paper has solid experiments and gives new insights about truthfulness and confidence calibration. I am happy to adjust my scores if the authors can address my concerns.\n\n[1] Samuel Marks and Max Tegmark. 2024. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.\n\n[2] Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto and Percy Liang. 2024. “Benchmarking and Improving Generator-Validator Consistency of Language Models.\n\n[3] Kevin Meng, David Bau, Alex J Andonian, Yonatan Belinkov. 2022. Locating and Editing Factual Associations in GPT.\n\n[4] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei. 2022. Knowledge Neurons in Pretrained Transformers.\n\n[5] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg. 2023. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model", "questions_to_authors": "1. Line 152: How do you ensure that \"every statement from tulu_extracted also appears in the tulu dataset\"? Do you look for the verbatim occurrence of the statement?\n\n2. Truthful interventions: Line 261 shows that the scaling factor for truthful intervention can only be +1 or -1. In prior works on intervening and steering models for truthfulness (e.g. [5]), it seems that the choice of scaling factor has a significant impact on the intervention results. Is there a reason for choosing this pair of scalars? Would the conclusions be different if the scalars are tuned on a validation set of better intervention effects?", "extra_scores": {}}, {"review_id": "I4XjOVwvRX", "reviewer": "Reviewer_vQ6Y", "summary": "This paper systematically investigate how post-training method work on LLMs. It investigates from four critical perspectives: knowledge representation, truthfulness, refusal behavior and confidence. The paper reveal how post training shape LLMs internally.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper reveals that it does not significantly alter knowledge storage locations or the direction of truthfulness. It adapts existing knowledge representations and develops new ones. Additionally, it changes the refusal direction and can modify confidence levels. \nThese findings may benefit various real-world applications.", "reasons_to_reject": "The experiment limited the model size to under 8 billion parameters. Evaluating on larger models, such as those with 13 billion or even 40 billion parameters, could yield better results.", "questions_to_authors": "Do the larger models behave consistently across these four perspectives?", "extra_scores": {}}, {"review_id": "VF5hsT9DQl", "reviewer": "Reviewer_WfuA", "summary": "This paper investigates how post-training, including supervised fine-tuning (SFT) and instruction tuning, mechanistically alters LLMs. Instead of treating LLMs as black boxes, the authors conduct a systematic internal comparison of pre-trained (BASE) and post-trained (POST) models along four axes:\n- Knowledge Storage and Representation: Post-training preserves knowledge locations but adapts and builds on prior knowledge representations.\n- Truthfulness: Truthfulness can be captured via a linear direction in hidden space, which is largely shared between BASE and POST models.\n- Refusal Behavior: Refusal directions exist but are less transferable between BASE and POST models.\n- Confidence: Changes in model confidence cannot be explained by entropy neurons. \n\nThe findings are based on empirical analyses across multiple models and datasets and offer insights into which capabilities are pre-training artifacts versus those truly shaped by post-training. The results have implications for model editing, alignment, and interpretability.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper provides the mechanistic understanding of post-training effects, which is underexplored but highly relevant for future LLM development.\n- The paper covers four complementary dimensions that collectively provide a deep mechanistic perspective, which is inspiring. \n- The paper discovers that truthfulness can be transferred across BASE and POST is practically important for efficient model interventions.\n- Insights in this paper are not just theoretical but suggest practical techniques for model steering, fine-tuning efficiency, and interpretability.", "reasons_to_reject": "- While the results are empirical and insightful, some deeper theoretical grounding (e.g., a theoretical model supporting that linear directions capture truthfulness) is missing.\n- The confidence section feels less robust compared to others; it largely concludes ``entropy neurons aren’t it’’ without exploring other possible factors.", "questions_to_authors": "- Why is backward transfer (POST to BASE) possible for refusal but not forward (from BASE to POST)? Any hypotheses?\n- Beyond entropy neurons, did you investigate other hidden state characteristics that might explain confidence differences?\n- What models (sizes, architectures) and datasets were specifically used? Do you expect findings to generalize to much larger or more instruction-heavy models (e.g., GPT-4)?\n- Did you explore other important axes such as reasoning, robustness, or chain-of-thought abilities? Why were these four perspectives chosen?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores the effects of post-training on the internal mechanisms of LLMs from four dimensions.  Knowledge storage locations are shown to be similar before and after post-training, although the analysis is limited in scope.  Internal representations of truthfulness are similar; similar vectors can intervene on truthfulness before and after training. Similar effects are shown for refusal. Finally, analysis of model confidence shows that this changes substantially between base and post-trained models, but there is no easy explanation.

The reviewers are generally positive about this paper. It is overall well-written and has some interesting experiments. However, I do think there are some valid criticisms by reviewer K28J. In general, I think the paper studies several different phenomena incompletely.  I can imagine a stronger paper that takes one of these effects and really analyzes it in a lot more depth, finding something more fundamental as a result. As it is, the experiments differ between settings and don't feel strongly unified except insofar as they focus on models before and after post-training.
Decision: Accept