Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
Authors: Jiaru Zou
Yikun Ban
Zihao Li
Yunzhe Qi
Ruizhong Qiu
Ling Yang
Jingrui He
Venue: NeurIPS 2025 spotlight ()
Keywords: Supervised Fine-tuning
Auxiliary Learning Framework
LLMs
Primary area: deep_learning
TL;DR: A new learning framework that improves LLM inference by learning from a Mistake Log collected during fine-tuning.

Abstract: Large language models are typically adapted to downstream tasks through supervised fine-tuning on domain-specific data. While standard fine-tuning focuses on minimizing generation loss to optimize model parameters, we take a deeper step by retaining and leveraging the model’s own learning signals, analogous to how human learners reflect on past mistakes to improve future performance. We first introduce the concept of Mistake Log to systematically track the model’s learning behavior and recurring errors throughout fine-tuning. Treating the original transformer-based model as the Pilot, we correspondingly design a Copilot model to refine the Pilot’s inference performance via logits rectification. We name the overall Pilot-Copilot framework the Transformer Copilot, which introduces (i) a novel Copilot model design, (ii) a joint training paradigm where the Copilot continuously learns from the evolving Mistake Log alongside the Pilot, and (iii) a fused inference paradigm where the Copilot rectifies the Pilot’s logits for enhanced generation. We provide both theoretical and empirical analyses on our new learning framework. Experiments on 12 benchmarks spanning commonsense, arithmetic, and recommendation tasks demonstrate that Transformer Copilot consistently improves performance by up to 34.5%, while introducing marginal computational overhead to Pilot models and exhibiting strong scalability and transferability. Our code is released at https://github.com/jiaruzouu/TransformerCopilot.

Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
Jiaru Zou', Yikun Ban Zihao Li", Yunzhe Qi', Ruizhong Qiu', Ling Yang" Jingrui Helt
"University of Illinois Trrbaa--Chappian "Princeton University
{jiaruz2, yikunb2, jingrui) edu, 1y19880princeton edu
O Code: http:///tthat.oooo//aaaaarrrrrtt
Abstract
Large language models are typically adapted downstream tasks through super- vised fine-tuning on domain-specific data. While standard fine-tuning focuses on minimizing generation loss optimize model parameters, we take deeper step by retaining and leveraging the model's 's own learning signals, analogous to how human learners reflect on past mistakes improve future performance. We first introduce the concept of Mistake Log to systematically track the model's learning behavior and recurring errors throughout fine-tuning. Treating the original transformer-base model as the Pilot, we correspondingly design Copilot model to refine the Pilot's s inference performance via logits rectification. We name the overall Pilot-Copilot framework the Transformer Copilor, which introduces (i) novel Copilot model design, (ii) joint training paradigm where the Copilot continuously learns from the evolving Mistake Log alongside the Pilot, and (iii) fused inference paradigm where the Copilot rectifies the Pilot's logits for enhanced generation. We provide both theoretical and empirical analyses ments on 12 benchmarks spanning commonsense, arithmetic, and recommendation tasks demonstrate that Transformer Copilot consistently improves performance by up to 34.5%, while introducing marginal computational overhead to Pilot models and exhibiting strong scalability and transferability.
new learning framework. Experi-
1 Introduction
Transformers, the foundation of modern large language models (LLMs), leverage attention and feedforward layers to compute logits for sequence generation [77]. Pre-trained on general-domain corpora, these models capture rich statistical patterns and exhibit strong generation capabilities [13, 80, 57]. On top of that, supervised fine-tuning (SFT) serves as a critical technique for adapting pre-trained LLMs to specific domains [40, 65, 80]. While SFT enables significant flexibility and task-specific optimization, the performance of fine-tuned LLMs during inference often remains suboptimal, exhibiting misalignment between training and testing stages [47, 79]. This gap arises from the model's inability to fully capture task- specifpei nuances or from overfitting to patterns within the training data, ultimately degrading its final performance [65, 56, 95, 53]. Without data-side interventions [52, 54, 27] or receiving external feedback [58, 72, 91], this paper aims to address a fundamental question: Can enhance the inference performance by retaining and leveraging the model's 's own learning signals in standard fine-tuning,
To address this question, our core idea draws inspiration from common strategy by human learners: maintaining a log record mistakes during practice, reflecting, and using insights to improve performance in formal tests. Rather than merely memorizing these mistakes, proficient learners engage in reflective thinking-anallzinn their internal cognitive states the moment the errors
1 Cooresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

occurred, questioning how and why the mistakes were made. The reflective practice enables learners to identify recurring error patterns and approach uncertain problems with greater caution and awareness.
Motivated by this human reflection thinking mechanism [31], we propose the concept of Mistake Log tailored for LLMs' fine-tuning. At training stages, standard SFT primarily focuses on optimizing model parameters by minimizing the expected loss over fine-tuning datasets [80, 97]. We take a deeper step to systematically record the rich intermediate information within the model, including input data (Question), internal hidden state representations (Rationale), and token-level quantified errors (Mistakes), as Mistake Log components to track model's mistakes through its training trajectory.
Next, to fully exploit the Mistake Log, we propose the Transformer Copilot (abbreviated as T- Copilot), novel CopilCo framework that enables error-aware refinement by learning from model-internal signals [11, 12, 34]. In addition to the original model (referred to as the Pilot), we introduce a Copilot model that captures and leverages the Pilot's Mistake Log throughout its learning trajectory, rectifying the Pilot's logits to improve final token-by-token generation. Overall, our learning framework offers advantages from three key perspectives: (i) New Model Architecture Design: We design the Copilot as transduction neural network that learns recurring error patterns from the Mistake Log. residual flow connection is then established between the Copilot and Pilot models, allowing the Copilot to assist the Pilot via token-level error correction during generation. (ii) New Training Paradigm: We redesign the SFT procedure by jointly training the Pilot and Copilot models in each round, enabling the Copilot to continuously learn from the evolving Mistake Log and adapt alongside the Pilot model. (iii) New Inference Paradigm: During next-token generation. we fuse the output logits from the Pilot and Copilot models into a unified probability distribution, enabling collaborative auto-regressive generation. In this way, T-Copilot fundamentally integrates an internalized reflection mechanism into standard SFT, enabling an adaptive and error-aware generation.
To demonstrate the efficacy of the T-Copilot, we provide detailed analyses from both theoretical and empirical perspectives. We incorporate T-Copilot into representative encoder-decoder and decoder- only Pilot models, and conduct extensive experiments across 12 tasks on commonsense, arithmetic, and real-world recommendation. T-Copilot improves the performance of Pilot by up to 34.5% while surpassing strong baselines with significantly fewer parameters. For example, integrating T-Copilot with Qwen2.5-7B outperforms Qwen2.5-14B using 4B fewer parameters. We further comprehensively study the efficiency, transferability, and scalability of T-Copilot, showing that T-Copilot brings marginal computational overhead to Pilot, scales well across different model types and sizes, and effectively transfers to new Pilot models for inference without additional training costs.
2 Definition of Mistake Log
2.1 Preliminary and Notations
Let f (::0") denote the function computed by standard Transformer model [77], parameterized by 8" In our context, we refer to f as the Pilor model. Suppose there are 'T' fine-tuming rounds. For each round & E [7], given an input sequence X, = (II.. Zt,a.) where n is the maximum sequence length, the input is sampled from data distribution D Dxxyy over input-output pairs.
The Pilot model then generates an output sequence Y, (,.,,,..., Ht.n)) in an auto-regressive manner to approximate the target sequence Y (J ),,,), where (Xt, Y:)+ ~ D.
During t-th fine-tuning round, let X, denote the input representation of X,, defined as either the encoder output in an encoder-decoder Transformer or the output of the token and positional embedding layer in a decoder-only Transformer. In the forward pass through the residual stream of the model, let L be the total number of decoder layers in the Pilot model. For each layer E [L*), we define he (x;; o as the (decoder) hidden representations of the i-th token. After the final decoder layer, the Pilot model outputs logits over the vocabulary V. conditioned on the input X, and shifted target sequence Vt, The resulting output probabilities for the i-th token are given by:
pr. = softmax (f? (Xe, Vt, ve
(1)
We denote it, the ground-truth distribution over V for the i-th token, which places full probability mass on the correct token Vt.s. The objective of training fP is to minimize the cross-entropy loss between the predicted and ground-truth tokens, formulated as:
n
log (vr, Nt.Br.
2

---

2.2 The Mistake Log
Next, we define the Mistake Log fine-tuning scenarios. As shown in Figure 1, the Mistake Log concludes three key components: the input representations (Questions), internal hidden states representations (Rationales), and the token-level error made by the model (Mistakes).
In each round & [2], draw the sequence pair (X,, Y2) ~ D. As defined in Section 2.1, set X as the input representation component, as it provides contextual grounding for the Pilot model's specific input sequence. Inspired by prior works [25, 20, 14, 45], the intermediate states' hidden representations produced by Transformer blocks also encapsulate rich contextual and semantic information, reflecting the model's s internal rationales. Therefore, we define he (Xe; 0f-1) as collection of these internal hidden representations for each token in round l:
with h4 sx,;(X(; ( L"
he Re(Xe;0' 0(1)) (xr; )
where i (r.(((x (Xei 0f-1) captures the i-th token level internal states representation at the point when the i-th token error occurs. Then, to quantify the token-level error of the Pilot model, we compute the discrepancy between the predicted distribution pr,f and the ground-trute distribution pt, for each token, with the error defined as:
(pt: pe) = (ec(Pe.s.P..)) with (e ) = Pt, pr.s-
Consistent with standard LLM fine-tuning procedures, where the loss C..' is used to compute gradients and update the Pilot model's parameters across T rounds, we simulta- neously collect key intermediate signals described above into the Mistake Log throughout this process. Formally, we define the Mistake Log as:
Mistake Log
M, =,= By Dil
Colleeting over the ongie traning
Quantficd
Groune-Truth Pa
Errors
Predict
Hidden Reprocentation
h
Mr - (xx, he he(Xx: lep,, po)
T
Transformer (Piit]
Decocler
Hidden Representiaiinn
(5)
Imput Representations
lgg
t=1
ayper?
Hidden Iegresentation
The Mistake Log systematically records contextual inputs, internal representations, and token-level prediction errors of the Pilot model throughout its entire fine-tuning tra- jectory. We next investigate how to leverage the Mistake Log during fine-tuning to enhance the Pilot model's final inference performance.
Ensurde
Layor
ltg
Figure 1: Illustration of the Mistake Log. We the encoder-decoder ar- chitecture as an example here.
Motivation for Transformer Copilot. Recall that the goal of SFT is optimize 8º by minimizing the expected loss Yx)-D [C]]. While this process adjusts model parameters using gradient descent, it treats each as transient signal, consumed and discarded immediately after the parameter update. As a result, the final model parameters OP might not retain an explicit memory of where, how, why errors occurred during the training trajectory. This oversight leaves valuable training-tine time information, which we captured in the Mistake Log, untapped at inference time. To address this, we propose new Copilot model learn from the Mistake Log. Rather than altering the Pilot's 's optimization path, the Copilot operates as an auxiliary module that internalizes the distribution of past mistakes and corrects the Pilot's output at inference time. This design enables the Copilot to assist the Pilot model by reflecting on prior missteps and adaptively revising the predictions.
3 Transformer Copilot
We introduce our proposed framework, Transformer Copilot, which is designed for both encoder- decoder and decoder-only Transformer architectures. In the following sections, will elaborate on the Copilot model design, the training paradigm, and the inference paradigm, respectively.
3.1 The Copilot Model Design
The Copilot model is initialized from the decoder module of the corresponding Pilot model, but with several new architectural modifications. Consistent with the Pilot model f., we denote the Copilot model as fo parameterized by 8º, The Copilot model is also nugo-segressive generating outputs
3

---

1. Model Design
2. Traininng Paradigm
Jointty Training
Input Sequence
Pilot Model
Logits f"
DF.C Model
Copliot Model
Mistake Log
M, = A0, f.)
Collec&& Sppate
eact murd
New Imput: ht)
New Input: (1g.he)
3. Inference Paradigm
Error Sequence
Iotertton
fexi
Copitot Logits /
Pilot Model
Decaddnng
Ishilted right)
Recofied Lagit
New
Layer
Leyore
Copilot Model
Copilot Model
Figure 2: Transformer Copilot Framework. The overall framework comprises three key components:
(1) Copilot Model Design, (2) Training Paradigm, and (3) Inference Paradigm.
over the vocabulary V. However, the objective of the Copilot model is to learn from the Mistake Log Mr and output rectified logits that correct the predictions made by the Pilot model. Below, we specify the Copilot model design for the encoder-decoder and decoder-only Pilot model separately.
Encoder-Decoder Copilot. As shown in Figure 2.1, the Copilot model receives its inputs from the Mistake Log. Mr = {Xx,, he (x;: (ptt p=)) Specifically, the Copilot is condi- tioned on the sequence of token-level errors made by the Pilot model, as recorded in Mr, i.c. l,<a = (Pt,1 - P - Pt). These discrepancy sequences are provided as labels during training from Mr and are auto-regressively generated during inference. As positional information inherently preserved through the Pilot's s output, we apply a single linear layer to project the token-level errors from vocabulary space into the Copilot's 's hidden dimension. Next, incorporate additional information from the Pilot's s input and internal hidden representations (X, and he from My), we propose modified coss-attention mechanism in each layer of the Copilot, defined as:
New Q H Wº, for = 1,..,,
New K = Concat [x,, Pool,, r (xx( W
(6)
New V = Concat Xx,, Pool (x,;; WV,
where (-) denotes the mean pooling across I.P layers of the Pilot and Concat(.) indicates concatenation along the sequence dimension to ensure input dimensional compatibility and com- putational efficiency; _ is the Copilot model's hidden state from the previous layer (or input projection layer at / = 1); and Wº, W WV are learnable attention weights. We then apply the standard scaled dot-product attention using the new Q, K, and V. This modified attention allows the Copilot to jointly attend to both the external input context and the internal processing dynamics of the Pilot. Note that all components retrieved from the Mistake Log can be directly accessed during the forward pass of the Pilot model, without incurring additional computational overhead. After the final layer I,C, we add linear projection layer in the Copilot model to map the residual hidden representation into the vocabulary space, producing rectified logits as the output.
Decoder-only Copilot. We slightly adapt the Copilot model accommodate the corresponding decoder-only Transformer [76, 1], while keeping the majority of the model input and design above unchanged. Specifically, we modify the self-attention mechanism incorporate the information from the Mistake Log: In the odd-numbered layers of L,C, we retain the standard self-attention to allow the Copilot model to capture intra- seuue-e dependencies; In the even even-numberee layers, we replace self-attention with the modified cross-attention mechanism defined in Eq. 6, enabling the Copilot to attend to the Pilot's input and internal state representations stored in Mr. This alternating structure is consistent with the encoder-decoder Copilot to capture its own coror-correctoon dynamics and attend to informative signals from the Pilot's behavior. We also explore several alternative designs and empirically validate the effectiveness of our proposed design against these variants in Appendix F.4.
Learning Objective. Give the sequence pair (X,, Yt), b-th round, the objective of training the Copilot model f at i-th token is defined as:
CC
1 - Ce P.,)), with JC, fO he, le

---

Algorithm 1: Transformer Copilot (Training Paradigm)
Input: Pilot model f (:;0") Copilot model f (x:0") Learning rates r)p, Ijo; 2', Initialize 0f 0g Mo
for 1, 2....... , do
Draw (X,, Y() ~ D
v Pilot token-level forward pass
iori=1, l..ii...... do Compute Pr.s via Eq.1
end
V Collect Mistake Log ($2.2)
Mr < Me-1 U (X,, (x,;;; of l(pe, p.))
10
Compute LI via Bq.2:
Update of + Of - n
Algorithm 2: Inference Paradigm
12
/+ For brevity, we reuse notation */ Draw (X,, le(Ki; 0f-1): lepi, p.)) Me
Input: 07, 05;; :Tuning parameter À
13
Draw new X, Dx. T
14
V Copilot learn from the Mistake Log ($3.1)
for I..... do
15
for i= L...., do
Pe, +- softmax(f" cri (()0))))))
16
fl. + 5o
he,,, Me (,xc;
Observe Ne. from f fe. + (Xiyheeetd
end
18
Compute LE via Eq.7; Update of + 0c_11 ne V ve fe
De,i f- De,i + AfLe (via Eq.8)
19
De,s <- Decoding(Pr.)
end
end
return 07
return (0%.)...., (1,))
where f(; is the Copilot model's prediction, (p Pr,s) = Pt, Pr. is the corresponding label for the Copilot model, and h4,<; is the collection of Pilot's hidden states for the preceding tokens. We adopt the RMSE loss to prevent the distribution error from being further diminished by the square operation, avoiding the over- smoohhnng effect that squaring may introduce in the gradient signal during backpropagation. Next. we show how to jointly train the Pilot model f" and the Copilot model f during fine-tuning, and collaborate on the generation during inference.
3.2 Training Paradigm.
Algorithm outlines the process for jointly training the Pilot and Copilot model. training round t e [T], one sequence pair (X,, Y2) is drawn from the data distribution D. For each token [n]. we first compute the Pilot model's 's output distribution pr,, (Line 5-7). We then retrieve information directly from the forward pass of the Pilot model and update the Mistake log M, by recording X he, and l, for each token (Line 9). Meanwhile, we compute the Pilot model's cross-entropy loss C and update its parameters (Lines 10-11). Next, we prepare the input for training the Copilot model. Given all collected previous training rounds' information, updated mistake log M, (Line 13). We obtain the Copilot model's output fC, for each token E [n] (Line 15-17). Finally, we compute the Copilot model's RMSE loss LC and update its parameters (Line 18-19). After T rounds of iterative training, we obtain the final 0f and Of for the Pilot and Copilot model, respectively. Note that this fine-tuning process can be readily extended mini-batch stochastic gradient descent for scalability.
draw a sample (X he, l.) from the
3.3 Inference Paradigm
After learning from the Mistake Log, the Copilot model is deployed alongside the Pilot model to enhance inference-time generation. To avoid abuse of notation, we reuse the same symbols in training. Given a new input sequence X, ~ Dx, t T. where X, is not part of the training data, t indexes the inference-time inputs and does not correspond training rounds. As the objective of the Copilot model is to predict the token-level probability discrepancy Pt, Pr.s, we directly use the Copilot model's 's output rectify the Pilot model's prediction Pr,s towards the ground-truti Pr.:- Formally, the rectified predicted distribution is given by:
Pr, Pr,s Af..
(8)
5

---

where À (typically set to 1) is a tunable hyperparameter controlling correction strength. Introducing À inference allows for more flexible modulation, and we later show Section 4, with proper the rectified it., theoretically provides a closer approximation to the target distribution Pt,i- Algorithm 2 outlines the overall inference paradigm. Given X,, the Pilot model outputs predicted distribution Pt.i at each token generation step i E [n] (Line 3). Subsequently, the Copilot model auto-regressively computes its output fú (Line 5). Finally, the rectified Pr., is obtained via Eq.8 and used to generate the next token via a decoding function (Lines 6-7). The inference process adaptive and can optionally terminate upon generation of the [EOS] (end-of-sequence) token.
Analyses Why Learn from the Mistake Log?
To elucidate the roles of the Mistake Log and Copilot model in enhancing the Pilot model's inference- time performance, we present both theoretical and empirical analyses in this section.
Theoretical Guarantee. Recall that the Copilot model fo is designed to analyze the Pilot model's internal cognitive states Xe, hr via the collected Mistake Log Mr, and learns to predict errors measured by the token-level discrepancies (Pr, Pc.:). During inference, we use the rectified prediction as Pr. = pr.. + Af In the following analysis, we show that, under mild assumptions, the adjusted prediction pt,s yields improved inference performance over the original estimate Pr.i-
Let AP, A denote the distributions over the function classes of 0P, 0C, induced by the randomness in the fine-tuning process. Let [k] denote the k-th dimension of a vector in RIV. Then, we define the expected error and variance of the Pilot and Copilot model at the k-th output dimension as:
6 Ex [(pri[k]] Egr-ar [K], (+.+:))"],
0 EX)D [arro~aF [K] De,sel],
< E -P [Ppr: [k] - Pr,][k] Egemas (P.(4] sE,ed) M.ci)..
(N,.Y.)-D
ae = E a AP [Vargo ( ff.ed 0..c.). (X.YD
Theorem 4.1. For any lo E [IVI], suppose that et. > 0 and EC < vop-oo of. Then there exists
An > O such that for any O X < λo the rectified prediction Pt.. = Di, + >ff, yields strictly closer approximation to the ground-truth distribution Pt, at dimension k. Specifically, at the i-th token prediction step for Xt ~ Dx, we have:
E oP NAP 10 (X)YD
[Pp,.[k] - (b,, Be
E oa [[Pr.[k] - pc,, pc,,k)))* @r,cs]
(NcYOND
Remark 4.2. The assumption EC < V oZ in Theorem 4.1 allows the Copilot model 1e (::00 to have larger bias than the bias 6p of the Pilot model j (::PP), i.e., ef. < E e of.
Theorem 4.1 suggests that the rectified prediction Pr,; after incorporating the Copilot model achieves strictly lower expected error at k-dimension under mild assumptions and a proper X, indicating the Copilot helps improve the inference performance of the Pilot. The full proof is provided in Appendix B. In addition, Remark 4.2 implies that the Copilot model can improve inference performance without needing match the Pilot's accuracy in isolation. This insight motivates to apply a relatively smaller size of Copilot to complement the Pilot in our empirical implementation.
faaacrirg Anthhretic Reanoring
20%
Logits Correction
1/%
La-4A-3.-122
LLaM4 2-2-5
LL444-33122 1-32
KO 2000
HAMPFS
M5:20
200A
83
2.
26 25 20
Tasks
Logits Correction (absolute value)
Figure 3: Logits Correction by Copilot. We visualize the logits correction introduced by 1B Copilot model (computed as Fused logits Pilot logits)) to highlight the shift by the Copilot's rectification. Left: Percentage of logits correction over original Pilot's 's output logits range for three LLaMA-3 Pilot models. Right: Distribution logits correction magnitudes across reasoning types.

---

Question: Choose the cornect, answer the question: Canson was at a firtend is house but suddenly arounnced they needed to go nowe. Mhy dtd Carson this? feswent: caught hus; Answer:: calleed cab; Answert: fongut to feed the dog. Answer formst: answery /twp////////////////
Pilot Model Only Answer:
+ Copliot Answer:
1s Forgot to Feed dog.
The cleice is (anawer
Copllot Model
Pilot + Copilot
Boct Fost Logit by Coophe
Flanet Logitt
+
Takkess
Toklon Prodiction
Lend)
Token Prediction Lons!
Toleen Predicttion per Layar [Lugita Lama)
ferget
Ivrget mnneer anuner $10/00
Layes
Figure 4: Example Copilot's Token-level Rectification SIQA. The token-level formatting error ('forgot') originates during the Pilot's mid-way generation and is corrected ('answer') ) by incorporating the Copilot.
theoretical analysis, empirically examine the recti- fication effectiveness of the Copilot model during inference. We leave the setups in Appendix C. Figure 3 illustrates the average logits correction induced by the IB Copilot model across different Pilot models and reasoning categories. Given that the typical logits range is approximately [-10, 10), the observed shifts the logits distribution indicate clear and consistent adjustment the final predictions by the Copilot model.
Empirical Analysis. Complementing
We further verify that this Copilot's adjustment indeed steers the token prediction toward the correct direction: We analyze representative error patterns frequently observed in the Pilot model's output, particularly factual and formatting mistakes. Figure 4 shows a detailed example of token-level logits rectification on Pilot model LLaMA-3.2-33 by the IB Copilot, visualized using the layer-wise Logits Lens [8]. At mid-inference, the Pilot does not follow the correct answer format and makes mistakes (the correct token 'answer' has a high but suboptimal logit). The Copilot rectifies the prediction by decreasing the logit of the incorrect token 'forgot' and amplifying that of the correct token, thereby correcting the token prediction error. We leave analyses on other error patterns in Appendix C.
5 Empirical Evaluations
Tasks and Datasets. To comprehensively evaluate T-Copilot, we utilize broad suite of reasoning
and generation tasks: (i) Commonsense reasoning: PIQA [10], HellaSwag [93], WinoGrande [68],
BoolQ [18], SIQA [70], and OpenbookQA (OBQA) [55]. (ii) Arithmetic reasoning: AQuA [48],
GSM8K [19], MAWPS [42], and SVAMP [60]. and (iii) Downstream Recommendation: Beauty [30]
and LastFM [67]. Detailed dataset descriptions are provided in Appendix D.
construct the Copilot model using the same type of decoder architecture as the Pilot model to ensure consistency. We use the AdamW optimizer and Cosine learning rate scheduler for both Pilot and Copilot models. We modify the generate in HuggingFace Transformers [22] to perform token-level logits fusion and rectified next-token generation during inference. All experiments conducted on NVIDIA A100 GPUs. We leave all hyperparameter setups and training/inference details in Appendix E.
Implementation Details. For T-Copilot,
Models and Baselines. We incorporate T-Copilot with varying backbone Pilot models. For encoder- decoder Pilots, we utilize T5 [65] and FLAN-T5 [17] across small/base/large variants. For decoder- only Pilots, we employ multiple models from LLaMA-3 [21] and Qwen2.5 [90] families. We denote #CCopplotsmmallbase//BBBB/// BB5////B as the Copilot model on different scales. Detailed model configu- ration and implementation details are provided in Appendix E.2. We compare against three bascline types: (i) Pilot-only models as described above. (ii) Frontier LLMs with comparable and larger pa- rameters, including I.LaMA-3.1-80 [21], Gemma-2-9B [75], and (wwen2.5-44B. (iii) Layer/Adapter expansion methods, including MoE models [71] (Mistral-7B, Ministral-8B), LLaMA/Mistrall Pro-8B [84], Mergekit-9B [26], and TIES [89]. Detailed baseline descriptions provided in Appendix E.3.
5.1 Incorporating T-Copilot into Pilot Models Yields Better Performance
Effectiveness of Copilot in Enhancing Pilot. Table presents the performance gains of incorporating T-Copilot into the Pilot models across different model scales and types. T-Copilot consistently
7

---

Table 1: Experiment results (%) of incorporating T-Copilot on bone models. Results are averaged over 3 independent runs. We report the relative improvement on the backbone Pilot models. T-Copilot boosts existing LLMs ten reasoning tasks by 2.0%-34.5
back-
Type
Model
Commonennse Reasoning (Ass. t)
Impr.
Arithmeric Rensoning (Acc. t)
Impr
PIQA
WinGG.
HellaS.
BaolQ
SIQA
OBQA
Avg.
AQuA GSM8K
MAWPS SVAMP Aug.
5-AN-T5--mall
60.3
52.1
31.6
57.9
47.8
29.2
46.5
19.6
5.6
14.7
5.3
11.3
+ --Copilot-smnall
63.1
51.1
34.9
61.7
52.7
32.9
50.0
1 1.55%
24.8
744
20.6
8.0
15.2
34.55%
FLAN-T5-base
36.8
48.6
29.6
49.4
22,8
27.1
TS
T-Copilot-hase
67.3
56.2
39.7
62.5
54.3
34.7
52,5
1 6.3%
24.4
9.3
32.4
10.3
19.1
20.11
FLAN FLAN-TS--urg
70.5
60.4
49.5
62.2
58.1
31.7
55.4
23.2
9.9
36.7
9.7
19.9
-T-Copilot-small
51.3
59.8
32.6
56.8
2.5%
24.7
37.2
11.6
6.55%
+ --Co
72.8
63.6
52.3
63.7
60.8
34.2
57.9
7 4.5%
25.1
11.6
39.8
13.8
22.6
13.6%
L1aMA-32--BB
77.5
71.1
61.8
63.9
71.9
66.8
68.8
25.6
27.3
77.1
47.3
44.3
+TCCopilot-1B
80.2
73.7
63.3
65.5
74.9
68.9
71.1
3.3%
28.3
32.2
81.5
51.6
48.4
9.33%
LLaMA
LLMMA-3.2-38
83.3
79.6
89.4
69.1
77.4
75.6
79,1
33.1
55.3
86.1
64.2
59.7
T-Coopilot-IB
84.1
82.6
91.1
70.3
78.6
77.2
80.7
1 2.0%
36.6
58.2
89.1
68.7
63.2
5.99%
T-Copilot-3B
85.6
83.7
91.3
72.8
79.2
81.3
82.3
4.0%%
40.1
63.1
91.2
71.4
66.5
11.4%
LLMMA-3.1-SB
85.4
84.3
90.9
69.6
79.9
82.6
82.1
37.3
63.55
89.1
73.6
65.9
+T-Copilot-1B
86.2
86.8
93.5
71.8
82.7
83.2
84.0
1 2.3% %
38.9
66.1
90.8
75.4
67.8
2.9%
Qwen2.5.3B
89.8
63.4
77.6
84.6
79.4
55.9
81.5
T-CCopllo-n.55
85.4
79.1
91.3
66.8
78.1
86.0
81.1
1 2.1%
57.3
74.22
91.8
82.8
76.5
1 2.5%%
+T-Copillo-388
57.8
81.7
94.0
68.7
79.9
89.4
83.6
1 5.3%
59.4
76.8
92.6
83.5
78.1
1 4.7%%
Qwen
Qwen2.5-7B
N2.1
91.4
79.3
83.4
61.0
75.3
91.2
84.8
+ 1-Copilot-0,558
89.3
85.3
93.5
73.6
80.0
92.1
83.6
1 2.6%
61.4
78.2
93.0
86.5
79.8
2.2%%
TT-Copilet-38
92.5
95.3
84.3
94.9
88.2
7 5.8%
64.2
79.7
94.8
88.1
4.6%
Table 2: Performance comparison (%) with baselines under matched parameter scales. Results are averaged over 3 runs. Adding T-Copilot consistently surpasses baselines of equal or even larger size.
Commonsense Reasoning (Ace. t)
Arithmetic Rensoning (Acc. f)
Model
Params
PIQA
WinoG.
HellaS, BoolQ
SIQA
OBQA
Avg.
AQUA
GSMSK
MAWPS
SVAMP
Avg.
LLaMA-3.1-88
8B
85.4
84.3
90.9
69.6
79.9
82.6
82.1
37.3
63.5
89.1
73.6
65.9
1...M/33..338 + T-Coopilor 3B 68 (:2B)
85.6
83.7
91,3
72.8
79.2
81.3
82.3
40.1
63.1
91.2
71.4
66.5
Qwen2 5-7B
7B
87.2
82.1
91.4
71.2
79.3
89.1
83.4
61.0
75.3
91.2
84.8
78.1
Qwen2 5.38 T-Copiloi.BB
(-IB)
87.8
81.7
94.0
68.7
83.6
76.8
92.6
83.5
78.1
Qwen2 2.n25.51. 5-1B
14B
91.8
85.6
94.3
75.2
87.4
79.5
92.4
87.9
80.8
Qwen2 2- TCooploo--BB
10B
(-1B)
92.5
87.2
95.3
74.8
84.3
94.9
88.2
64.2
79.7
94.8
88.1
81.7
Comparison wäth Layer/Adapter Expansion Bavelines
Mistral-Pre-8B
8B
83.1
81.9
86.1
70.8
76.1
80.6
79.8
35.5
51.4
88.2
68.5
61.7
LLaMA-Pr--BB
8B
88.4
81.4
86.9
73.9
76.1
77.8
80.8
38.2
57.2
92.5
63.5
62.9
Ministral-8B
8D
85.7
84.1
91.3
70.3
77.5
81.3
81.7
37.4
62.9
90.2
73.2
65.9
LLaMA-3.2-3B- + T-CCopiloe-3B 3B 6B (-2B)
85.6
83.7
91.3
72.8
79.2
81.3
82.3
40.1
63.1
91.2
71.4
66.5
MergeKit-9B
9B
86.1
84.7
91.1
71.1
79.3
80.2
82.1
37.0
65.2
90.3
75.2
66.9
M.MMA-31-88 I-Copiloe-I8
98
8622
86.8
93.5
71.8
82.7
83.2
84.0
38.9
66.1
90.8
75.4
67.8
improves performance across all T5, LLaMA, and Qwen models on 10 commonsense and arithmetic reasoning tasks. In particular, a lightweight Copilot (e.g.. T-Copiilo--mmall can deliver meaningful improvements (6.5% arithmetic) when paired with much larger Pilot model (e.g., FLAN-T5- large). Moreover, scaling up the Copilot model leads to additional improvement, underscoring its effectiveness in rectifying the Pilot model's predictions during inference.
Comparison with Size-Matcher Baselines. As shown in Table 2, we first compare our method against stronger models with larger parameters under the same model backbones. While LLaMA-3.2 3B initially lags significantly behind LLaMA-3.1-8B. incorporating T-Copilot-3B enables the model to outperform 1LaMA-3.1-8D despite using 2B fewer total parameters. Similarly, for the Qwen2.5 series, incorporating T-Copilot-3B enables the smaller Qwen2.5-7B to surpass Qwen2.5-14B with 4B fewer parameters. To provide a broader perspective, we also compare with strong baselines from different methods and model types. For instance, although LLaMA-3.2-3 originally trails behind models like Ministral-80 -BB and LLaMA-Pro-8B. incorporating T-Copilot-3B 3B enables it to outperform the strongest baseline under the 8B scale, Ministral-8B. while maintaining a 2B parameter advantage. Due to page limits, full comparison results are provided in Appendix F.I.
Downstream Tasks. Additional evaluation of T-Copilot and baseline comparisons on downstream recommendation tasks is provided in Appendix F.2.
8

---

T-Coplict-18
T-Cooplot-18
LLaMA-Fro-85
MargelIt-2
Total Infesennc Time
T-Coplat-1
LLa/M6.32-38-TTCCa
3-33-
LLMMA-3.1-4
tokensis (Inference)
Instancesis (Fine (Fine-Tunil Tuning)
B. 2 3 3 B g 0
+7.0%
Totul Prote Tww.ettttnngg Times
TTCoppoao110
-15%
Margakti-0
SIGA
Models
(e)
(b)
(c)
Figure 5: Efficiency Analysis T-Copilot during fine-tuning and inference. (a) Inference model throughput. (b) Fine-tuning running speeds. (c) Overall training and inference time overhead.
Table 3: T-Copilot Transferability Results. We report the performance of T-Copilot paired with new Pilot models across four reasoning tasks. The results demonstrate that the Copilot model remains effective for the new Pilot models without being jointly trained.
T-Copilot-11
HellaSwag
BoolQ
GSM8K
SVAMP Overall Impr.
with new LLaMA-3.2-IB
63.1
65.2
32.2
51.4
1 6.1%
with new LLaMA-3.3-3B
91.4
70.2
58.8
68.5
1 4.2%
with new LLaMA-3.1-8B
93.1
71.7
66.0
75.8
1 2.4%
5.2 Efficiency, Transferability, and Scalability
Efficiency. To thoroughly evaluate T-Copilot's running efficiency, we compare against Pilot and baseline models with the same LLaMA-3 backbone architecture under similar parameter scales. As shown in Figure 5, T-Copilot maintains comparable inference throughput (Figure 5 (a)) and training speed (Figure 5 (b)) to its corresponding Pilot models, while incurring only 4% marginal average increase in time overhead (Figure 5 (c)). In contrast, other baselines such LLaMA-Pro- MergeKit-9P suffer from significantly higher latency and computational costs relative to their base model LLaMA-3.1 1-B. We provide more detailed inference latency report in Appendix F.3 (Table 15) and discuss the architectural advantage of our model design in Appendix A.
and
Transferability. In the T-Copilot learning framework, the Copilot model is fine-tuned alongside but separately from the Pilot model. Since the same type of models generally have similar learning trajectories under identical training settings, we further investigate if the Copilot model can leverage the mistake log of one Pilot model and still be effective on another Pilot model of the similar type.
We conduct controlled experiments on LLaMA-3 series models in which we directly apply finetuned 1B Copilot model new Pilot models during inference. The new Pilot model shares the same architecture as the original one but is trained independently. Note that the Copilot model does not "see" or "learn" any information from the new Pilot model, as they are not jointly trained during finetuning. In Table 3, transferring the Copilot model leads to slight +.22% performance difference compared to applying the Copilot to the initial Pilot models (jointly training together). We hypothesize that the minor discrepancy is due to the hardware inference differences between the original and new Pilot models. Nonetheless, the transferred Copilot model still delivers substantial performance gains for the new Pilot and consistently outperforms competing baselines. These results demonstrate that T-Copilot's crror-correction capabilities are not tightly coupled to a specific Pilot model and can be effectively transferred without additional rounds of fine-tuning.
Scalability. Figure 6 illustrates the relationship between accuracy and model parameter size for T-Copilot. Overall, incorporating the Copilot model consistently demonstrates improved performance as model size increases. parameter size (N) in billions. The derived equations for our method are as follows: for Flan-T5 backbones: A ~ 8.74-log. 1og (N) + 40.17; For L.LaMA-3 backbones: A 29.58 log (N) + 50.20.
analyze the relationship between performance accuracy (A) and model
(N)
and for 5 backbones: A ~ 12.40 (NN + 71.80.

---

Performance (N)
Performance (%)
Parformance
T-Coplict Plan-Ts
25
Size
Parameter Size (B)
Paramater Size (B)
14
Figure 6: Inference Scaling Laws for T-Copilot. We evaluate the average accuracy of TTCCoiloo and backbone frontier LLMs across all reasoning tasks at varying model scales. The results are shown for three architectures: FLAN-T5 (left), LLaMA-3 (middle), and Qwen2.5 (right).
Ablation Studies. Detailed ablation studies on T-Copilot, including model design choices, input insertion patterns, and the effect of the hyperparameter X, are presented in Appendix F.4.
Related Works
LLMs Supervised Fine-tuning. Supervised fine-tuning (SFT) serves as the standard tost-nraining method for specializing pre-trained LLMs to downstream tasks [80, 69, 94]. It enables models to incorporate task-specific knowledge and improves their performance in domain-relevant settings [94, 59, 96]. While effective, SFT often suffers from misalignment between training-tine -ime objectives and inference- time behavior [56, 791, leading to suboptimal generalization. Recent work has explored parameter-efficttt tuning methods [32, 46]. alongside advanced adaptation strategies [74, 49, 85] that improve learning effectiveness and efficiency. These methods primarily focus on model capacity and optimization rather than leveraging learning dynamics. Building upon prior SFT methods, our approach is compatible with existing fine-tuning frameworks and further improves by incorporating model-internal signals into the fine-tuning process. By adaptively learning from mistake patterns observed during fine-tuning, T-Copilot enables error-aware prediction and helps reduce the gap between training and inference performance.
Self-refinement Language Models. Recent research has explored various self-refinement techniques in LLMs to generate high-quality outputs. Models either iteratively prompt themselves with updated responses [52, 27, 72] or optimize their behavior using external human or synthetic feedback [58, 91, 54, 99]. Orthogonal to external supervision such additional prompting, multi- stage feedback, or explicit reward optimization, our work focuses on capturing model-internal signals during fine-tuning achieve leken-level rectification, without modifying the training objective or data distribution. We leave the additional related work and discussions in the Appendix G.
7 Conclusion
In this paper, we introduce Transformer Copilot, a novel learning framework that enhances Transformer-based Pilot models by integrating an auxiliary Copilot model during fine-tuning, By capturing the Pilot model's learning signals Mistake Log during fine-tuning, the Copilot model learns to rectify the Pilot's logits at inference time, enabling error-aware predictions. We provide both theoretical and empirical evidence that our method improves the Pilot model's inference pre- dictions. Experiments on 12 benchmarks demonstrate the effectiveness, efficiency, scalability, and transferability of Transformer Copilot. Discussions on limitations are provided in Appendix H.
Acknowledgment
We would like to thank the anonymous reviewers and AC for their constructive feedback during the review process. We also thank Sirui Chen, Tianxin Wei, Zhining Liu, Xiao Lin and other members from UIUC iDEA-iSAIL Lab for their constructive feedback on this work. This work is supported by National Science Foundation under Award No. IIS-21 17902, and Agriculture and Food Research Initiative (AFRI) grant no. 0020-67021-32799/ppppo Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.
accession no. 1024178 from the USDA National
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 3159-3166, 2019.
[3] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867, 2023.
[4] Jimmy Lei Ba. Layer normalization. arXiv preprint
2016.
[5] Yikun Ban, Ishika Agarwal, Ziwei Wu, Yada Zhu, Kommy Weldemariam, Hanghang Tong, and Jingrui He. Neural active learning beyond bandits. arXiv preprint arXiv:2404.12522 2024.
[6] Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Ee-net: Exploitation-exploraiio neural networks contextual bandits. arXiv preprint arXiv:2110.031177 2021.
[7] Yikun Ban, Jiaru Zou, Zihao Li, Yunzhe Qi, Dongqi Fu, Jian Kang, Hanghang Tong, and Jingrui He. Pagerank bandits for link prediction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 21342-2133 21366. Curran Associates, Inc., 2024.
[8] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08122 2023.
[9] Candice Bentéjac, Anna Csörgö, and Gonzalo Martinez-Muñoz. A comparative analysis of gradient boosting algorithms. Artificial Intelligence Review, 14:1937-1967, 2021.
[10] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[11] Evelyn M Boyd and Ann W Fales. Reflective learning: Key to learning from experience. Journal of humanistic psychology, 23(2):99-117, 1983.
[12] Anne Brockbank, Ian McGill, and Nic Beech. Reflective learning in practice. In Reflective learning in practice pages 18 18-28. Routledge, 2017.
[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[14] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827,, 2022.
[15] Jianpeng Cheng. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733 2016.
[16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-
113, 2023.

---

[17] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling satruction-inettune language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
[18] Christopher Clark, Kenton Lee, Ming-Weieee Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 2019.
[19] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10,148, 2021.
[20] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021.
[21] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
[22] Hugging Face. Transformers documentation. 2024.
https://buuginggacec.o//// co/ts//cccco////s../
[23] Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01880 2017.
[24] Yoav Freund, Robert Schapire, and Naoki Abe. A short introduction to boosting. Journal- Japanese Society For Artificial Intelligence, 14(771-780):1612, 1999.
[25] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:20/2.149// 2020.
[26] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: toolkit for merging large language models. arXiv preprint arXiv:2403. 2024.
[27] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.000 088998 2023.
[28] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 2024.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507-517, 2016.
[31] Mandy Hommel, Bärbel Fürstenau, and Regina H Mulder. Reflection at work-a concep- tual model and the meaning of its components in the domain of vet teachers. Frontiers in Psychology, 13:923888, 2023.
[32] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv 2021.
preprint
[33] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Lim-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304. 009933, 2023.
[34] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve arXiv preprint arXiv:2210.116100 2022.
12

---

[35] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.212276 2024.
[36] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[37] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[38] Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pages 197-206. IEEE, 2018.
[39] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 34:1022-1035, 2021.
[40] Jacob Devlin Ming- ding-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019.
[41] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213: 2022.
[42] Rik Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157, 2016.
Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.
[43] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[44] Mike Lewis. Bart:
Denoising sequence-to-sequene pre-training for natural language genera- tion, translation, and comprehension. arXiv preprint arXiv:1910.1340 2019.
[45] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.
[46] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. 2021.
arXiv preprint
[47] Yanhong Li, Chenghao Yang. and Allyson Ettinger. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. arXiv preprint arXiv:2404.00112, 2024.
[48] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraie word problems. arXiv preprint arXiv:1705.04146 2017.
[49] Shih-Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.0933 2024.
[50] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang. Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology page 100017, 2023.
[51] Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint arXiv:2306.091 09782, 2023.
13

---

[52] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback Advances in Neural Information Processing Systems, 36:46534 2023.
[53] R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:/902.0100 2019.
[54] Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using ilms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00430 2023.
[55] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal, Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 180..0989, 2018.
[56] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprini arXiv:2202. 12837, 2022.
[57] Yuhong Mo, Hao Qin, Yushan Dong, Ziyi Zhu, and Zhenglin Li. Large language model (Il) ai text generation detection based on transformer deep learning algorithm. arXiv preprint arXiv:2405. 06652, 2024.
[58] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[59] Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. The ultimate guide to tne-tuning 1ims from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv preprint arXiv:2408. 13296, 2024.
[60] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07190 2021.
[61] Yunzhe Qi, Yikun Ban, and Jingrui He. Graph neural bandits. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1920-1931, 2023.
[62] Alec Radford. Improving language understanding by generative pre-training. 2018.
[63] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
[64] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprini arXiv:21 12. 1446, 2021.
[65] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[66] Raquel B Robinson, Karin Johansson, James Collin Fey, Elena Márquez Segura, Jon Back, Annika Waemn, Sarah Lynne Bowman, and Katherine Isbister. Edu-larp @ chi. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-5, 2023.
[67] Benedek Rozemberezki and Rik Sarkar. Characteristic functions graphs: Birds of feather, from statistical descriptors to parametric models. In Proceedings of the 29th ACM international conference information & knowledge management, pages 1325-1338 2020.
[68] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 1021.
14

---

[69] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2 10.08807, 2021.
[70] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint 1904,09999 2019.
[71] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expprts layer. arXiv preprint arXiv: 1okiv:/701.000 770.00658, 2017.
[72] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
[73] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228, 2024.
[74] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 15:12991-13005, 2022.
[75] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussemot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at a practical size. arXiv preprini arXiv:2408.00) 001 18, 2024.
[76] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2xXiv:2307.09288 2023.
[77] A Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
[78] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964-22984. PMLR, 2022.
[79] Yihan Wang, Andrew Bai, Nanyun Peng, and Cho-Jui Hsieh. On the loss of context-awareness in general instruction fine-tuning. arXiv preprint arXiv:241 1.00688, 2024.
[80] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
[81] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models, arXiv preprint XXXv:2206.006822. 2022.
[82] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of tf--uhttgttttttttth prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
[83] Tongyu Wen, Chenglong Wang, Xiyuan Yang, Haoyu Tang, Yueqi Xie, Lingjuan Lyu, Zhicheng Dou, and Fangzhao Wu. Defending against indirect prompt injection by instruction detection, 2025.
[84] Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang. Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive Ilama with block expansion. arXiv preprint arXiv:2401.02415, 2024.
[85] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. arXiv preprint 2rXiv:2404.03552. 2024.
15

---

[86] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang, Parameter-efficcee fine-tuning methods for pretrained language models: critical review and assessment. arXiv preprint arXiv:2312. 12 148, 2023.
[87] Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01700, 2020.
[88] Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. Openp5: An open-source platform for developing, training, and evaluating Ilm-based recommender systems. arXiv preprint arXiv:2306. 11144, 2023.
[89] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. mies-merging: Resolving interference when merging models. Advances Neural Information Processing Systems, 36, 2024.
[90] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint urXiv:2412.15113 2024.
[91] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
[92] Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang. Demystifying reinforcement learning in agentic reasoning, 2025.
[93] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: arXiv: 1et..9955000000 2019.
[94] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
[95] Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference machine learning, pages 12697-12706 PMLR, 2021.
[96] Jiawei Zheng, Hanghai Hong, Feiyan Liu, Xiaoli Wang, Jingsong Su, Yonggui Liang, and Shikai Wu. finnituning large language models for dopaii-specific machine translation. arXiv preprint 2rXiv:2402.1500 15061, 2024.
[97] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36:55006-55000 2023.
[98] Jiaru Zou, Soumya Roy, Vinay Kumar Verma, Ziyi Wang, David Wipf, Pan Lu, Sumit Negi, James Zou, and Jingrui He. Tattoo: Tool groundddddddd thinking prm for test-time scaling in tabular reasoning. arXiv preprint arXiv:2510.06217 2025.
[99] Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux-prm: Trajectory-awar prms for long chain-of-thought reasoning in llm. arXiv preprint arXiv:2506. 18896, 2025.
[100] Jiaru Zou, Mengyu Zhou, Tao Li, Shi Han, and Dongmei Zhang. Promptinterm: Saving inference costs by internalizing recurrent prompt during large language model fine-tuning. arXiv preprint arXiv:2407.02211 2024.
16

---

Appendix
Table of Contents
A Additional Details Transformer Copilot
18
Architectural Advantages of T-Copilot.
18
A.2 Decoder-only Copilot Details
18
A.3 Buffer of the Mistake Log
18
A.4 Conceptual Rationale on T-Copilot's Learning Objective
18
Proof of Theorem 4.1
19
Additional Empirical Analysis
21
D Datasets
21
D.I Commonsense Reasoning
21
D.2 Arithmetic Reasoning
22
D.3 Downstream tasks: Recommendation
22
D.4 Fine-tuning Dataset Template
23
Experiment Setups
E.1
Hyperparameters and Training/Inference Details
24
E.2
T-Copilot Configurations and Implementations.
27
E.3
Baseline Details
28
F
Additional Experiments
29
F.1
Full Table Report Baseline Comparison
29
F2
Downstream Recommendation Evaluation
29
F3
Efficiency Analysis on Transformer Copilot
30
F.4 Ablations and Analyses Transformer Copilot
31
Additional Related Works
33
Broader impact and Limitation
34
17

---

A Additional Details on Transformer Copilot
A.1 Architectural Advantages of T-Copilot
In Section 3.1, we introduce the Copilot model inherited from the standard decoder module in Transformer [77]. However, our model design exhibits several key advantages compared to the standard decoder module: Specifically, our Copilot model (i) eliminates the need for positional embeddings to preprocess input sequences, (ii) does not require softmax layer to normalize high- dimensional logits distributions, and (iii) avoids waiting for the computation key-value (KV) pairs from the previous layer. These architectural design choices distinguish adaptation methods [26, 84] that modify internal Transformer layers, which inherently introduce additional computational overhead. As result, our method minimizes the gap in efficiency between our framework and vanilla models.
method from layer
A.2 Decoder-only Copilot Details
The decoder-only copilot model f inherits its structure from the pilot model and processes three inputs from the Mistake Log: the token-level discrepancy sequence &,, the embedded input sequence Xx, and the pilot model's hidden states h4- Note that, different from the encoder-decoder Pilot model, X₂ here derived from the input sequence X, after the positional embedding layer.
In the Decoder-only Copilot model, as stated Section 3.1, the alternating attention mechanisms effectively mirror the encoder-decoder structure, enabling the decoder-only Copilot to leverage information inside the Mistake Log corrected from the Pilot model. The loss function (RMSE) and target values (pt..: Pt.s) for the Decoder-only Copilot model remain identical those used for the encoder-decoder Copilot version. The fine-tuning and inference paradigm are also the same as the encoder-decoder Copilot model, as stated in Algorithm I and 2.
A.3 Buffer of the Mistake Log
As described in Section 3, we maintain a Mistake Log to record the Pilot model's internal learning signals, which serve training data for the Copilot model. To store this information efficiently with minimal GPU and CPU memory overhead, we detach all relevant outputs from the Pilot's forward pass and store them in CPU-resident buffer. By default, the most recent 128 training rounds. The buffer is updated at each training step, and all Copilot training samples are drawn exclusively from it. This design keeps the additional memory footprint lightweight, typically under 500MB of the CPU memory and less than 200MB of the GPU memory.
fixed-size buffer that retains
A.4 Conceptual Rationale on T-Copilot's Learning Objective
We elaborate here the motivation for adopting the RMSE loss in training T-Copilot. Recall that the objective of the Copilot to predict the distribution discrepancy, defined as
E (p Pt.s) = Pt,i -e.i-
in our formulation. Note that (p Pr,c) is not valid probability distribution, since E, (e Pr.)) * 1. Therefore, is natural to formulate this task as a regression problem, for which the RMSE loss commonly adopted.
In contrast, common language modeling objectives such as Cross-Entropp (CE) and KL Divergence are designed for distribution fitting and not directly applicable in this setting, unless manipulate (pt.s. with a softmax function to resemble a valid distribution. In that case, however, the loss would no longer optimize the original discrepancy (i.e., Pt. Pr,,), leading to potential information loss. In Appendix F.4, we further emprically validate the effectiveness of the RMSE loss for T-Copilot training.
18

---

B Proof of Theorem 4.1
Given the model parameters o and 0c, we denote the Pilot model as (=:0%) and the Copilot model as (-; Let X₁ Dx represent the imput sequence at inference step t. and X, be the input representation the X,; Y be the corresponding ground-truth answer for the input sequence X₁. For the t-th token prediction during inference, recall that:
Pt.i = (vc, X1, De,)))) Pr.i = softmax (X,, (e:600")))
E2 (x(x,, M
AP,A AA denote the distributions over the function classes of PP, 80, induced by the randomness in the fine-tuning process. Let [k] denote the k-th dimension of a vector in RIV Then, we define the expected error and variance of the Pilot and Copilot model at the k-th output dimension as:
Let AP,
6 = EX
(Pr..lA] Egp- [K Br,<,1) < cc,
o = (xx.Y))
Varro~ar Dt <i]] < 00,
è = ~M
[pr..[k] - Pr.ii1. [k] Egc [[ sl,2il) Mncil oo,
(X,,Yi)-D
a E 0" (X.Yi))D
(Vargo (0
1 dc <0
where we assume f and 1 have the bounded error and variance at k-th dimension. Here, Pr, denotes the ground-truth probability assigned to the token at dimension & E [[V[] of the vocabulary, for the i-th token prediction step within input sequence Xe. Then, we have the following theorem, which corresponds Theorem 4.1 in the main body.
Theorem B.1 (Restate). Given AP. A the Pilot model (=:A:P) the Copilot model (=( and data distribution D. For any [[V]]. suppose the Pilot model is imperfect, i.e., Cp > 0, and the Copilot model's error satisfies EC < V + op. Then there exists a constant Ao > 0 such that for any 0 X An the rectified prediction Pt,; Pr, + >ff., yields a strictly closer approximation the ground-truth distribution Pt,i at dimension k. Specifically, at the i-th token prediction step for X₁ Dx, we have:
[(P..[K] -b, (Eccnniee) < < OP [[p.i(k]] (b)( de
E
oF PP
G (N.
(Xe.Ye)DD
Proof. For brevity, omit the explicit expectation condition over the Pilot model's previously generated tokens Ct, the Copilot model's preceding outputs f.-3. and the dimension index [k] in the following proof.
Firstly, by the law of total expectation w.r.t. Y,) and the bias-variance decomposition w.r.t Pr,s.
EgEZaP [pr.. - 1 (XI,
E(X,,N)) EOP (pr, - 22.
=E(x,,Y)> (pt, - ESPNAR 12,,,)² + Var (prull
E(X.,Yi) (pr - EAPWAR (r..)) + (ax..x))ooVaaroaAa. [r.
=6 o
19

---

Secondly, by the law of total expectation w.r.t. (X2, Yi) and pt,s and the bias-variance decomposition w.r.t fE,0
E B pr-go (X.Yi)D
(pts - -D,, - -1,509
= EyPrAP [Baonsc((H, - Pr,s - -(,)) (X,N1))D
BOP~A' (Xr,i)~D
(PLs, Di, Egg A0 (r + Vargo Uiill
= BBPwap I(pe.- pc, - Ego (t 21 + EOP~AP [Vargo [f (X,,
Ap (X,,Y))
= =6 +oc.
Thirdly, by the law of total expectation w.r.t. (X,, Yi) and pr,s and the Cauchy-Schwaann Schwarz inequality,
Exe-AP 0a-A0
[(pe. (p,, - pr, f(,
(X,.Ki)
[Bgo Ao [pr, - (o,) - Dc,; -(,0)]
= EAP~AR (X..
[(pt,- Pi.I) Pe,t EgoNAc [f..]]
BBd Ap [Pt.s- - -p [ Epp-gp (Xiili/D
[(pe.- - pr,s - Ego (f
(X.Y1)D
V (5f + oz) E = o o
Together, as long as S X 1 1,
ExP~aR 10-A0 (X,Y)))
[(pr,- - it, -
AAP (Xe)
[
- -2,
Eyewat ~0
[((1 - ((11-)(pp) -,,i) + Mprr.
- - -1,))) - -Bi(pr. - -
(XD,
= (1 (1-A)²Eg.
2) NAP
[(P.. + EBP ~AP
[pt,s pr,. - $2.
gN-Ac
80 00-
(X
(X.V))DD
+ +111 A)AB
[(pri - -prs) pr.s
- "EgraA" [(pe: )1
0 AAP
gA~Ac
(X.D
= (1 -
(Prt -Po,))
8" AA
(X.,De)-DD
+ 2(1 A)AE
1(pE, -pw)) Pe,s sGiil - EOPNAP [(pe, )2.
NAB
0
(X1.
(X.Y)))D
=(1 - 2)² 11)
[(re, pc.1) + AF gC
[(Pr. -r.j p1
~A"
(Xo,
(X.D
+ +2(1 A)AE ~AP
1(pr. (p.) - Pr,s ff.
0
(XD
$ ((1 - 1)2 - 1) + o7) + 22 + +0²) + +2(1 - ))d /
= =X((lV -c) + +00)
2/ 2 + 0z2 (V 27p + o cc)))
which is strictly smaller than 0 as long as
0 << X min {
{1, 11, 2 2/26/ + + +ož -c)
=: Ao
(Ve + toz - =cc)² +0²
20

---

C
Additional Empirical Analysis
Question: Zoch
logride. The Forris Forrs wheel, roller log ticket Zach chould buy?
many
Pilof Model Onlyy Answer
Model I-Copilat Answeer
how manry ticksta meeds buy, wa need wod the cost of each rite and subtract the number tickets Zoch atready hos. The Forr a wheel costs 2 lickets. The roller coustar costs kikeess. The lng ride poatt 1 rickee, Addiing these up, we get 2 7 10. Zach rwads to buy 10-18 1 (10 tickets. Thereforw, Zach should buy 10 tickers.
each tickets Zech
Zuch
Forrss wheel
The roller costs buy 9.okkkets.
logg Addiing
should buy tickets.
PlotModd
SCoppat
Pilce Paoked Copeplet
Topr Lagits
likers
Model Top Precictirn Layer Loogt
T. Tos tayt Weee
Lever
Figure Example Copilot's Token-level Rectification on MAWPS.
Setups. In our empirical analysis, we choose LLaMA-3.2-3B as the Pilot model and T-Copilot- BB the Copilot model. The Copilot model's implementation details the same stated in Appendix E.2. We evaluate on two reasoning tasks, including SIQA [70] and MAWPS [42]. The dataset details provided later in Appendix D.
Example of Copilot's Token-level Rectification. Figure demonstrates another representative example of Copilot's 's token-level rectification the factual error made by the Pilot model. The token "10" is originally predicted wrong during the Pilot model mid-generation and is later corrected (token "9") through the Copilot model's logits rectification. To visualize the process, showing the top-5 tokens' output logits and probabilities in the current token prediction. Note that the Copilot not only increases the logits value the groundtruth token but also decreases the logits value on the original Pilot model's falsely predicted token. We further apply the Logit Lens [8], a standard interpretability tool, to project hidden state embeddings from each intermediate layer onto the vocabulary space to show how the Copilot adjusts the Pilot model's 's predictions on each state.
present three plots
D Datasets
D.1
Commonsense Reasoning
For the commonsense reasoning tasks, choose six open-ended multiple-choice QA tasks. The detailed description for each dataset is listed below:
PIQA [10]: A dataset for physical commonsense reasoning, requiring models to choose the more plausible solution for everyday tasks.
WinoGrande (WinoG.) [68]: A large-scale dataset for commonsense pronoun resolution, extend- ing the Winograd Schema Challenge [43] with diverse and harder examples.
HellaSwag (HellaS.) 193]: A benchmark testing commonsense reasoning in story completion by selecting the most plausible next sentence among adversarial choices.
BoolQ [18]: A question-answering dataset where models answer yes/no questions based on a given passage, requiring deep reading comprehension.
SIQA [70]: A dataset for reasoning about social and emotional situations by selecting the most appropriate response to everyday scenarios.
Openbook QA (OBQA) [55]: A dataset that tests knowledge-basse question answering by requiring models to combine common knowledge with reasoning over multiple facts.
21

---

In our commonsense reasoning experiments, we follow the experimental setup from [33] and fine-tune both our models and baseline models on the combined training dataset, Commonsense 170K, which is constructed by sampling and integrating the training sets of the aforementioned commonsense reasoning datasets. Each dataset's individual test set used for evaluation. Both fine-tuning and testing data instances utilize zero-shot input prompts.
D.2 Arithmetic Reasoning
For arithmetic reasoning tasks, we evaluate our method on four open-enden math mroblem-solvingg datasets spanning multiple mathematical domains. The detailed description of each dataset is provided below:
AQuA [48]: A dataset of algebraic and arithmetic word problems presented in multiple-choice format, requiring logical reasoning and numerical computation.
GSM8K [19]: A dataset of grade-school-leve math word problems designed to evaluate step-by- step reasoning and arithmetic skills.
MAWPS [42]: dataset aggregating math word problems from various sources, focusing on problem diversity and automatic equation generation.
SWAMP [60]: A dataset that introduces systematic variations of simple arithmetic word problems assess model robustness against linguistic perturbations
In our arithmetic reasoning experiments, we follow the experimental setup from [33] and fine-tune both our models and baseline model on the combined training dataset, Math10K. We also adopt the data preprocessing setup in [85] to avoid any potential training data leakage. Each aforementioned dataset's individual test set used for evaluation. Note that, unlike commonsense reasoning, fine- tuning for arithmetic reasoning involves labels with zero-shot Chain-of- Chain-of-Thought Thought (CoT) [41] prompts. Consequently, the training cutoff length is longer due to the increased token count and additional information contained in the prompts.
D.3 Downstream tasks: Recommendation
For downstream application experiments, we utilize two sequential recommendation datasets, as LL.M-based recommendation is widely adopted task to evaluate language models' generation and decision-making capabilities. The detailed description for each dataset is listed below:
Beauty [30]: The Beauty dataset comprises user-item interaction data from the Amazon beauty product category. It includes 22,363 users and 12,101 items, with a total of 198,502 interactions. The dataset has sparsity level of 99.93%.
LastFM [67]: The LastFM dataset contains 1,090 users and 3,646 items, with 52,551 interactions in total. The sparsity of the dataset is 98.68%.
In our experiments, we use the training and testing datasets from [88]. To ensure a fair comparison, we assign random numeric IDs items and evaluate our method and baselines on sequential recommendation tasks.
Metrics. For evaluation, employ two commonly used metries Hit K and NDCG@ K metrics with K E {5, 10, 20, 1003. We define each metric in detail below:
Hit Rate measures the proportion of users for whom at least one relevant item appears within the top K recommendations.
1
H@K
T(Rel(u) n RKK + 0)
UU
WEU
where U is the set of users, R the top-K recommended items for user u. Rel(u) is the set of relevant items for user u, and T(-) is the indicator function which equals if the condition is true, and otherwise.
NDCG evaluates both the relevance and position of items in the ranked list, assigning higher importance to relevant items that appear earlier, thereby reflecting the overall quality of the ranking
22

---

system.
relu. (++
NDCG@K
(10)
UUI
relu.a logy((+11
REU
where relu,; is the relevance score of the item at position in the ranked list for user u, and R is the ideal ranking of relevant items for user it.
D.4
Fine-tuning Dataset Template
Table 4: Examples of dataset templates used in Transformer Copilot.
Task Type
Template
Please choose the correct answer to fill the blakk to complete the given sentence: "After being diagnosed with Parkinson'ss ss he now was taking supplements and prescriptions, but the doctor wamed the would be ineffective." Optionl: supplements Option2: preseriptions Answer format: apiion//ppioon
Input:
Answer: The correct
option I.
Please choose the correct eoding complete the given senience: High jump: The buy lands his buck on to a red mat. The boy gets up from the mat. The boy: Ending I: starts doing spins. Ending2: colebrates by clagping and flexing both armss Linding3: is duncing on the mat. Ending:: does jump jacks on his stick. Answer ending
Input:
Commonsense Reasoning
format:
Answer: ending?
Input:
Please answer the following question with trae false, question: is clder scrells online the same as skyrim? ? Answer format: truc/false.
Answer: false
Please answer the following maih question step by step: Damel sprinted 0.875 lap and took break by jogging 0.75 of lap. How much farther did Dameel sprint than jog?
Iuput:
Answer: Step Find the total distance Darael covered. Total distance = distance sprinted distance
jugged Total distance = 0.875 + 0.75 = 1.625 laps Step 2: Find the distance Damel sprinted. Distance sprinted laps Step Find difference between the distance Damel sprinted and jogged. Difference = Distance sprinted - Distance jogged Difference 0.875 0.75 0,125 laps Therefore, Darnel sprinted O. 125 laps farther than jorged. The answer Anabic numerals 0.125.
0,875 laps Step Find the distance Damel jogged. Distance jogged
Input: the probability that Stock increase value during next month 0.56, and the probubility that Stock B will increase in value during the next month is 0.74. What is the greatest value for the prohubility that neither these two events occur?
Answer:
The probability that stock A does not increase 0.44, and the probability that stock does not increase is 0.26, Now, how can the prebubility that both do not increase be more than individual probability of not increasing for each? So the probubility that hoth do not increuse can not be more than 0,26. Busically the increase is between and
Arithmetic Reasoning
probubility that both
has rounds. There 8 games in first sound, in second round. 2 in the third round and I during the finals. If each game requires new tennis balls, and on average each game 5 cans tennis balls, how many tennis balls in total are used at the ead of the tournament each has tennis balls?
Iuput: typical toumnmmnn
Step 1: Calculate the number tennis balls used in each game. Each can has balls, so cans have x 5 15 tennis balls. Therefore, each game uses 15 tennis balls. Step 2: Calcuulate ate the total number of games in the louramentt. The first round has games, the second round has 4 games, the third round has 2 games, and the finals have game. Therefore, the total number of games in the tournament is 8 + 4 2 I 15.Step 3: Calculate the total number of tennis halls used in the tournament. Fach game uses 15 balls. Therefore, the total number of tennis balls in
Answer:
tennis so 15 games use tournament is 225.0.
Input: Considcring (dlotuset), user__ [wse[[[wr__dd iid) bas interacted with [daizser] items [kistory). What is the next recommenndtiion for the user?
Downstream Recommendation
Answer: [dataser] [songet] E.g. Beauty item. _123
In Table 4, we provide examples of data instances for each task mentioned above during model fine-tuning. All experiments are conducted in the zero-shot setting to better facilitate wisel-wise evaluation using p accuracy (i.e., based on a single generation attempt).
23

---

E Experiment Setups
E.1 Hyperparameters and Training/Inference Details
Tables 5-10 present our hyperparameter settings of each task for reproducibility. We perform hyperparameter tuning for both T-Copilot and baseline methods. Unless otherwise specified, both our method and baseline implementations use beam search decoding [23] during inference. All experiments have been run three times with random seeds, reporting average accuracy. For FLAN-TS, ILaMA-3, and Qwen2.5 models, checkpoints are saved every 1,000 steps to track parameters and monitor training ensure robustness and avoid overfitting.
Table Hyperparameter configuration of Transformer Copilot for LLaMA-3 and Qwen-2.5 series models on the Commonsense Reasoning Tasks.
Hyperpurameters
Pilot Model
Copitot Model
12-MA-32-18
1aMA-32-388 L1aMA-3.1-88
T-Copilot
X
[0.1. 0.S. L.]
Fine-tuning Configurations
Epochs
3
3
3
Batch Size
16
16
16
Micro Batch Size
4
4
4
Cut Length
256
256
256
Maximum Learning Rate
3e-1
3c-+
Se
Sem-
Learning Rate Scheduler
Cosine
Cosine
Cosine
Cosine
Optimizer
Adam
Adam
Adam
Warmup Steps
200
200
200
200
Weight Decay
LoRA Confgurations
LoRA Alpha
64
64
64
64
LoRA Dropout
0.05
0.05
0.08
Inference Configurations
Temperature
Top p
0.95
Top
Maximom New Tokens
61
Table 6: Hyperparameter configuration of Transformer Copilot for LLaMA-3 and Qwen-2.5 series models on the Arithemtic Reasoning Tasks.
Pilot Model
Copilot Model
Hyperpuramettrs
22--A-2-18
1JMA-322-88 LLLMA-3.1-88
T-Copilot
X
10.1. 0.3. O.S, 0.8. L.)
Fine-tuning Coufgurratius
Epochs
3
3
3
Batch Size
16
16
16
Micro Batch Size
4
4
4
Cul Length
256
256
256
Maximum Learning Rate
2c
2e
le-4
Se
Learning Rate Scheduler
Cosine
Cosine
Cosine
Cosine
Optimizer
AdamW
Adam|W
Adam 100
Adam
Warmup Steps
100
100
Weight Decay
0.00
0.00
0.00
LoRA Configrratons
LoRA Alpha
64
64
64
64
Dropout
0.05
Inference Configurations
Temperature
Top
0.95
Top
Reams
Maximum New Tokens
24

---

Table Hyperparameter configuration of Transformer Copilot for LLaMA-332-1BB LLaMA-3.2-3B. and -LaMA-3.1-88 on the Downstream Recommendation Tasks.
Päot Model
Copdlo Model
Hyperparameters
LLLMA-32-18 LLMMA-32--3B L1aMA-3.1-8B
T-Copilox (IB)
0.8. 1.)
Fine-tuning Configurations
Tpochs
3
Batch Size
Micro Batch Size
Cut Off Leagth
256
256
256
256
Maximum Learning
Leamning Rate Scheduler
Cosine
Cosine Adam
Cosine AdamW
Cosine
Optiiizer
AdamW
AdamW
Warmup Steps
Weight Docay
0.00
0.00
0.00
0.00
LoRA Configurations
Rank
LoRA Alpha
16
16
16
LoRA Dropout
0.05
0.05
0.05
0.08
Inference Configurations
Temperature
Top
0.95
Top
40
Num Beams
4
Muximam New Tokens
64
Table 8: Hyperparameter configuration of Transformer Copilot for FLAN-T5-small/ass//argg on the Commonsense Reasoning Tasks.
Hyperparumeters
Pilot Model
Capilot Model
FLAN-TS-amll FLAN-TS-bue FLAN-TS-Srge
T-Copilox (smuilobose)
[0.1D33,0,.0 10]
Fine-tuning Configuraaiins
Epuchs
12
12
12
Micro Batch Size
Length
Maximam Learning Rate
lc
Icm3
Ic-0
3
Learning Scheduler
Optimizer
Adam
Adam
AdamW
AdamW
Warmup
Weight Decay
0.01
0.01
0.01
0.01
Inference Configurations
Temperature
Top
0.95
Top
Num Beams
Maximam New Tokens
64
25

---

Table Hyperparameter configuration of Transformer Copilot for FLANTT5-small/bss//ag on the Arithmetic Reasoning Tasks.
Hyperparumeters
Pilot Model
Copálot Model
Fl AN-TT---ualll BLAN-TS-hose FLANN-TS-arge
T-Capplox (smallihose)
A
D,, O8, 1.)
Flue-tuning Confliurrations
Epochs
12
12
12
12
Batch
Micro Batch Size
Cut Off Length
256
256
256
256
Maximam Learning Rate
Learning Rate Schecluler
Cosine
Cosine AdamW 0,05
Cosine Adum 0.05
Cosine
Optimiie
Adoo
AdamI
Warmup Radio
0.05
0:05
Wright Docay
0.01
0.01
0.01
Drop Out
0.1
0.1
0.1
Inffevucce Configurations
Temperarure
0.1
Top
Top
40
Maximan New Tokens
236
Table 10: Hyperparameter configuration of Transformer Copilot for T5-small/base on the Down-
stream Recommendation Tasks.
Pilot Model
Copilot Model
Hyperparameters
T5-small
TS-base
TTCCopilot (small/base)
A
[0.1, 0.3, 0.5, 0.8, 1.0]
Fine-tuning Configurations
Epochs
20
20
20
Batch Size
16
16
16
Micro Batch Size
Cut Off Length
256
256
256
Maximum Learning Rate
le-3
le-3
1e-3
Learning Rate Scheduler
Cosine
Cosine
Cosine
Optimizer
AdamW
AdamW
AdamW
Warmup Ratio
0.05
0,05
0.05
Weight Decay
0.01
0.01
0.01
Drop Out
Inference Configurations
Temperature
Top
0.95
Top k
40
Num Beams
Maximum New Tokens
64
26

---

Table 11: Total and Trainable Parameter Statisties. We report the total trainable parameter count for encoder-decodee models. For other model types, we present the proportion of trainable parameters under LoRA fine-tuning relative to the total model size.
Size (Total)
Params (Trainable)
Type
Model
TS-small
61M
61M
T-Copilot-small
92M
92M
T3- T5-smalli2
122M
122M
15-base
223M
223M
I-Copilot-base
349M
349M
T5/FLAN-T5
TS-base24
446M
446M
FLAN-TS-small
77M
77M
+ T-Copilot-small
118M
118M
FLAN-TS-base
248M
248M
T-Copilot-base
385M
385M
FLAN-TS-large
783M
783M
T-Copilot-small
824M
824M
+ T-Copilot-base
920M
920M
Llama-Pro-8B
8.9B
0.832%
LLaMA
Mistral-Pro-SB
8.3B
0.858%
Mistral-7B
7.3B
0.721%
MoE
Ministral-8D
8.0B
0.821%
MergeKit
MergeKit-9F
8.9B
0.710%
Gemma
Gemma-2-9B
9.2B
0.813%
LLaMA-3.2-1B
1.3B
1.215%
T-Copilot-IB
2.4B
1.246%
LLaMA-3.2-3B
3.2B
1.018%
LLaMA
T-Copilot-IB
4.3B
1.018%
LLaMA-3.1-88
8.0B
0.700%
+ T-Copilot-IB
9.1B
0.705%
Qwen2.5-3B
3.IB
1.244%
T-Copilot-0.5B
3.6B
1.650%
T-Copilot-3B
6.1B
1.263%
Qwen
Wwen255-BB
7.6B
0.814%
T-Copilot-0.5B
8.0B
0.819%
T-Copilot-3B
10.88
0.815%
Qwen2.5-14B
14.8B
0.211%
E.2 T-Copilot Configurations and Implementations
In our implementation, we integrate the Transformer Copilot learning framework into both encoder- decoder and decoder-only LLMs mentioned above. Specifically, we introduce Copilot model as an auxiliary component to the original Transformer architecture. Below, we provide details our models' implementation and notations.
T5/FLAN-T5:
T-Copilot-small: This refers to our Copilot model being initialized from the decoder module pre-trained T5-small or FLAN-T5-small model. Specifically, T-Copilot-smail consists of decoder layers with hidden state dimension of 512, 8-headed attention, and a logit distribution dimensionality of 32,100. To adopt the model for our method, we exclude the conventional positional embedding mechanism and omit the softmax layer typically used for normalizing logits
27

---

denotes the integration of FLAN- IB indicates the incorporation of into probability distributions. Additionally, we add linear layer to map the Copilot inputs from the logits distribution dimension the decoder hidden state dimension. If the Copilot's hidden state dimension differs from the Pilot model, an additional linear layer is added for dimension alignment.
T-Copilot-base: This refers to our Copilot model being initialized from the decoder module of a pre-trained T5-base or FLAN--55-bass model. The overall model implementation is similar to T-Copilot-small: T-Copilot-baa consists of 12 decoder blocks with a hidden state dimension of 768, 12-headed attention, and a logits distribution dimensionality of 32,100.
LLaMA-3:
T-Copilot- 1B:
This refers to our Copilot model being initialized from the decoder module of pre-trained LLaMA-3.2-ID model. T-Copilot-IB consists 16 decoder blocks with hidden state dimension of 2048, 32-headed attention, and a logits distribution dimensionality of 128,256. To adapt the model for our method, we exclude the conventional positional embedding mechanism and omit the softmax layer typically used for normalizing logits into probability distributions. To accelerate training, we incorporate the flash-attention mechanism. To enhance inference efficiency, we upply mean pooling to the concatenated input hidden states h,, 01-1) without compromising performance accuracy. We add linear layer to map the Copilot inputs from the logits distribution dimension to the decoder hidden state dimension. If the Copilot's hidden state dimension differs from the Pilot model, an additional linear layer is added for dimension alignment.
This refers to our Copilot model being initialized from the decoder module of a pre-trained LLaMA-3.2-3B T-Copilot-1B 1 consists of 28 decoder blocks with hidden state dimension of 3072, 24-headed attention, and a logits distribution dimensionality of 128,256.
T-Copilot-3B:
Qwen2.5: The model configurations for Qwen2.5 similar to LLaMA-3 models they share similar model implementation details. We provide the additional model configurations below:
T-Copilot-0.5BB
This refers to our Copilot model being initialized from the decoder module of pre-trained Qwen2.5-0.5B. T-Copilot-O.5B consists of 24 decoder blocks with hidden state dimension of 896, 14-headed attention, and logits distribution dimensionality of 151,936.
T-Copilot-3B: This refers to our Copilot model being initialized from the decoder module of a pre-trained Qwen2.5-3B. T-Copilot-3B consists of 36 decoder blocks with a hidden state dimension of 2048, 16-headed attention, and a logits distribution dimensionality of 151,936.
Notation. In our experiments, we represent our methods using the original model name "+" the Copilot model. For example, T5-small with T-Copilot-small, and LLaMA-3.1-8D with T-Coppilo-IB.
E.3
Baseline Details
Frontier Models. Below, we detail the specific model versions of the backbone and baseline models in our experiments.
(i) Encoder-Decode Models: We use T5 and FLAN-T5 [65] with different sizes as our back- bone and baseline models for the encoder-decoder Transformer architecture: T5-small, T5-base, T5-large and FLAN-T5-smal1, FLAN-T5-base, FLAN-T5-large.
(ūi) Decoder-Onl Models: For the decoder-only models, we utilize the LLaMA-3 family [21] as our backbone and baseline models. Our experiments include LLaMA-3.2-1B 2-1B, LLaMA-3. -.233- LLaMA-3 1-BB, and LLaMA-2-13B.
(iii) MoE Models: For the Mixture-of-Exper based models, we use Mistral-78 with version Mistral-7B-va -B-v00. and Ministral-88 with version Winistral-8B-Instaeee Instruct-2410
Layer/Adapter Expansion Models. In experiments, also compare against baseline methods that utilize layer and adapter expansion approaches. Below, we provide the model configurations and implementation details for these baselines.
(i) LM Pro [84]: LLaMA-Pro-88 incorporates a content-addressablee working memory module to store and retrieve task-relevant information. In our implementation, initialized with the
28

---

LLaMA-3,1-8B base model and expanded the number of blocks from 32 to 40 using an interleaved approach. Mistra-Pro-85 is an enhanced version of the original Mistral model [37], augmented with additional Transformer blocks. The model excels in integrating general language understanding with domain-specific knowledge and follows the same methodology as LLaMA-Pro-8ll for block expansion. Following [84], we use the version of Mistral-Pro-8B-vO.11
(ii) MergeKit [26]: MergeKit is an open-source toolkit designed for efficiently merging LLM checkpoints to combine their strengths without additional training. In our experiments, we train and apply one MergeKit model named MergeKit- t- MergeKit-93 is initialized from LLaMa-3.1-88 and replicates additional layers with post-merge healing. The model is merged using the Passthrough method. In our experiments, we first compare the model with the original LLaMA-3.1-88 to ensure that the merged model does not lead to performance degradation.
(iii) TIES [89]: T5-small12 and T5- T5-base24 are T5 type models merged using the TIES method. 15-small12 merges two T5-small models and extends the original T5-small to 12 encoder and decoder layers. And TS-base24 merges two T5-base models and extends the original T5-base to 24 encoder and decoder layers by duplicating existing layers.
Model Parameters. In table 11, we provided the detailed model sizes and trainable parameters for both Transformer Copilot and baseline models.
F
Additional Experiments
F11
Full Table Report on Baseline Comparison
Table 12: Full performance comparison (%) with fronticr baselines under matched- saaacmed-parrmeter scales, Results are averaged over 3 independent runs.
Commonsensse Reasoning (Ace. f)
Arithmetic Reasoning (Acc. f)
Model
PIQA WinoG. HellaS. BoolQ SIQA OBQA Avg. AQUA GSMSK MAWPS SVAMP Avg.
LAR-level Frontier
Mistral-7B
7B
83.0
75.3
81.3
73.1
75.4
28.9
50.2
85.3
57.4
55.5
LLaMA-Pto-SB
88.4
86.9
80.8
92.5
63,5
62.9
LL.MA-3.1-8B
8B
85.4
84.3
90.9
69.6
79.9
82.6
82.1
37.3
63.5
89.1
73.6
65.9
Ministral-8B
8B
85.7
84.1
91.3
70.3
71.5
81.3
81.7
37.4
62.9
90.2
73.2
65.9
Qwen2.5 + T-Copilot-0.5B
3.5B
85.4
79.1
91.3
66.8
78.1
86.0
81.1
57.3
74.2
91.8
82.8
76.5
LLaMA-3 T-Copilot-5B
6B
85.6
83.7
91.3
72.8
79.2
81.3
82.3
40.1
63.1
91.2
71.4
66.5
Qwen2.5-3B T-Copilot.B 3B
6B
87.8
81.7
94.0
68.7
79.9
89.4
83.6
59.4
76.8
92.6
83.5
78.1
Qwen2.5-78 T-Copilot-0.5B
7.5B
89.3
85.3
93.5
73.6
80.0
92.1
85.6
61.4
78.2
93.0
86.5
79.8
3-88-level Fronteer LLMs
Gemma-2-9B
81.4
93.5
82.3
82.7
75.0
65.5
MergeKit-9B Qwen2.5-14B
86.1
71.1
79.3
82.1
37.0
65.2
90.3
75.2
66.9
14R
91.8
85.6
94.3
75.2
84.5
93.1
87.4
63.5
79.5
92.4
87.9
80.8
LLaMA-3, 1.-B--..--BBB--. T-Copilot-188
9B
86.2
86.8
93.5
71.8
82.7
83.2
84.0
38.9
66.
90.8
75.4
67.8
Qwen2.5-7B T.Copilos.33 3B
10B
92.5
87.2
95.3
74.8
84.3
94.9
88.2
64.2
79.7
94.8
88.1
81.7
Table 12 shows the full comparison results of T-Copilot against baseline models and methods with matched and larger parameter scales. Notably, under the same model architectures and with outperforms LLaMA-3.1 1-8B with 2B 70t.5---7----oploo------ outperforms Qwen2.5- 5-B with 4B fewer parameters, and Owen22.5-BB-TT-oppiioo also outperforms other layer/adapter expansion baselines. These results underscore the parameter efficiency and architectural strength of our learning framework.
less pre-trained knowledge, ILLMA-3.22 11-33.-3B=TT-ooplott-. fewer parameters,
outperforms Qwen2.5-7B with 1B fewer parameters. Our method
F.2 Downstream Recommendation Evaluation
In Table 13 and Table 14, we report the results of T-CCopilot on two downstream recommendation datasets: Beauty and LastFM. We choose T5 and LLaMA-3 series models as the backbone Pilot models. Overall, T-Copilot improves the Pilot models by an average of 16.6% across all evaluation metrics on the two datasets. Furthermore, compared to other basclines, incorporating T-Copilot
29

---

Table 13:
Performance comparison on Beauty. All methods evaluated using both Hit Rates (H@K) and Normalized Discounted Cumulative Gain (N@K). The performance gains are also reported relative respective backbone methods.
Beauty
Models
II@5
I1@
10
II@20
I1@ 100
N@5
N@ 10
N@20
N@ 100
T3-smally2
1.9
3.2
5.3
15.4
1.3
1.8
3.9
6.2
T5-small
1.7
2.9
5.4
14.6
1.0
1.4
3.5
5.7
+ T-Coopilot-small
2.4 (+0.7)
3,4 (+0.5)
6.2 (+0.8)
17.8 (+3.2)
1.6 (+0.6)
2.1 (+0.7)
4.5 (+1.0) (+0.7)
TS-base24
2.66
4.6
7.5
18.6
2.3
2.9
4.7
6.8
T5-base
2.3
3.3
6.2
17.4
2.1
2.6
4.5
6.2
+ T-Copillt-base
3.2 (+0.9)
4.4 (+1.1)
(+2.0)
19.8 (+2.4)
(+0.6)
3.3 (+0.7)
5.2 (+0.)) 7.2 (+1.0)
LLaMA-3.2-1B
5.2
7.4
10.0
18.8
3.8
4.4
5.1
6.7
+ -T-Copilot--BB
6.1 (+0.9)
8.1 (+0.7)
12.5 (+2.5)
24.6 (+5.8)
4.3 (40.5)
5.1 (+0.7)
5.8 (+0.7) (40.7)
I.LMA-3.2--B + T-Coppilot-IB
5.1
7.6
10.8
22.1
3.6
4.5
5.3
7.2
6.7 (+1.6) 8.6(+1.0)
13.2 (+2.4)
25.6 (+3.5)
4.3 (+0.7)
5.6 (+1.1) 5.9 (+0.6) 7.8 (+0.6)
I.MMA-3.1-8B
5.8
8.3
11.1
21.5
4.1
4.9
5.6
7.5
+ TT-Copilot-1BB
7.1 (+13) (20.9) 13.5 (+2.4) 26.4 (+4.9) 4.7 (+0.6) 6.2 (+1.)) 6.4 (+0.8) 8.1 (+0.6)
Table 14: Performance comparison on LastFM All methods are evaluated using both Hit Rates (H@K) K) and Normalized Discounted Cumulative Gain (N@K) The performance gains are also reported relative to respective backbone methods.
LastFM
Models
H@5
H@
H@20
H@ 100
N@5
N@ 10
N@20
N@100 100
TS-smallj2
2.5
3.8
4.9
12.4
2.2
2.8
3.9
T5-small
2.1
3.7
4.2
11.0
1.6
2.0
2,5
3,2
T-Copillt-smaall
3.2 (+1.1)
4.4 (+0.7)
5.7 (+1.5)
15.3 (+4.3)
1.9 (+0.3)
3.2 (+1.2)
3.8 (+1.3)
4.0 (+0.8)
T5-hase24
3.8
4.6
7.1
17.5
2.0
3.8
3.3
4.7
T5- 1S-base
2.7
4.2
5.3
14.9
1.9
2.4
2.9
3.4
+ T-Copilot-base
4.2 (+1.5)
5.1 (+0.9)
8.1 (+2.8)
19.4 (+1.5)
2.3 (+0.4)
3.5 (+1.1)
4.2 (+1.3)
5.2 (+1.8)
LLaMA-3.2-IB
5.0
5.7
9.1
21.9
2.4
3.0
3.9
6.2
+ TT-Copilot-1B
6.4 (+1.4)
6.8 (+1.1)
11.2 (+2.1)
24.7 (+2.8)
2.9 (+0.5)
3.5 (+0.5)
4.3 (+0.4)
6.7 (+0.5)
LLaMA-3.2-3B + '-CCopilot-IB
6.1
6.4
9.2
23.9
2.6
3.5
4.2
6.8
6.8 (+0.7)
7.4 (+1.0)
12.1 (+2.9)
25.1 (+1.2)
3.1 (+0.5)
4.2 (+0.7)
5.3 (+1.1) 7.5 (+0.7)
I.L.A-3.1--B
4.7
7.3
10.3
25.6
3.1
3.7
4.7
+ +T-Copplot--IB
(+2.2)
8.6
12.7 (+2.4)
28.0 (+2.4 3,9 (+0.8) 4.8 4.8 (+1.1) 5.4 (+0.7) 7.9 (+0.9)
achieves 16.7% and 8.6% higher performance than T5-smallj2 and T5-basez4, respectively, Beauty and LastFM, while using 30M and 126M fewer parameters. These results demonstrate that the error-correction capabilities of T-Copilot are confined reasoning tasks but also generalize effectively to other application domains, such as recommendation, where precise LLM decision- making is critical for downstream utility.
F33 Efficiency Analysis on Transformer Copilot
Table 15 presents the inference latency evaluation across six reasoning datasets. Our learning frame- work achieves lower latency than baseline models with comparable parameter scales. Specifically,
30

---

Table 15: Efficiency Comparison on Inference Latency. We report the total response time (s) per instance across six commonsense reasoning datasets, along with the average result.
Inference Latency ()) PIQA
WinoG.
HellaS.
BoolQ
SIQA
OBQA
Avg.
LLaMA-3.2-IB
0.33
0.36
0.27
0.23
0.27
0.24
0.28
+ T-Coppiot--B
0.36
0.39
0.28
0.26
0.29
0.25
0.31
LLaMA-3.2-3B
0.46
0,45
0.47
0.46
0.46
0.55
0.48
+ T-Copplot-1B
0.48
0.47
0.49
0.46
0.48
0.56
0.49
LLaMA-3.1-8B
0.52
0.52
0.51
0.49
0.49
0.62
0.53
+ T-Coopilot-1B
0.52
0.53
0.53
0.49
0.50
0.63
0.53
LLaMA-Pro-88
0.83
0.75
0.82
0.76
0.75
0.73
0.77
MergeKit-9B
0.64
0.64
0.57
0.54
0.63
0.72
0.62
LLaMA-3. 1------------oooo-------- consistently achieves 22.9% lower inference latency, 3% higher train- ing throughput, and 57% higher tokens-pert per-second (TPS) on average compared to methods such as LLaMA-Pre -AAM--Pro-BB-- and MergeKit-9B. Furthermore, we observe that incorporating T-Copilot increases the inference latency by less than 2% relative to the original Pilot models, while yielding significant performance gains.
F4 Ablations and Analyses on Transformer Copilot
In this section, we perform multiple ablation studies to evaluate the influence of key hyperparameters and alternative method design on the T-Copilot's overall performance.
Model Design of T-Copilot. Table 16 compares T-Copilot-11 1B with variant that excludes learning from the Pilot model's intermediate fine-tuning stages. The superior performance of T-Copilot highlights the advantage of our joint training paradigm, where the Mistake Log is continuously updated throughout the Pilot's training trajectory and enables the Copilot to effectively leverage intermediate-stage information.
Table 16: Ablation study on model design. We denote Latest as the variant where the IB Copilot is trained using only the latest Pilot checkpoint.
Table 17: Ablation study on X. We use the T-Copilot- 1B on LLaMA-3 series models.
HellaSwag
GSMSK
A
Pile
Copilo
AQUA GSMMK
MAWPS
SVAMP
Avg.
IB
3B
8B
IB
3B
8B
30.1
79.4
ULMMA-32-18 T-Copilot
28.3
32.2
81.5
51.6
48.4
0.3
62.0
90.6
90.9
29.8
56.8
64.4
LLAMA-3.2-38
Latest
34.6
57.1
87.5
65.2
61.1
0.5
62.8
90.9
91.5
30.4
57.6
65.9
T-Copilot
0.8
63.1
91.2
92.4
31.8
58.1
65.7
LLaMA-3.1-8B T-Copilot
Latest
37.6
64.6
90.0
73.9
66.5
1.0
63.3
91.1
93.5
32.2
58.2
66.1
38.9
66.1
90.8
75.4
67.8
Design Variants of the Decoder-Only Copilot. To validate the efficacy of our proposed decoder- only Copilot design, explore several alternative architectural variants and empirically compare their impact on the model's final performance. Specifically, we examine different insertion patterns for the Copilot's new attention mechanism. i.e., the input and hidden states representations from the Pilot model recorded in the Mistake Log. We experiment with various design patterns and modify the Decoder-only Copilot model accordingly. The design options are listed below:
Pattern 1 (Ours): Collect the hidden states across all Pilot model layers LP and insert them as key-value (KV) inputs for the even-numbered layers of the Copilot model.
Pattern Collect the hidden states across all Pilot model layers L and insert them as KV inputs for each layer of the Copilot model. This setup examines whether integrating hidden states into all layers of the Copilot model improves performance by leveraging more entry points for processing the pilot model's 's hidden states information.
31

---

Pattern 3: Collect only the first half (L* layers of the Pilot model's hidden states and insert them as key-value (KV) inputs for the Copilot model. Combined with Pattern 4, this setup investigates where the Pilot model makes more mistakes during the learning trajectory.
Pattern 4: In contrast to Pattern 3, Pattern 4 collects only the second half (In layers) of the Pilot model's hidden states and inserts them as KV inputs for the Copilot model.
Table 18: Empirical comparison of different design patterns of the Decoder-only Copilot model. We evaluate the LLaMA-3.22 -...--BB 1B Pilot model and T-Copilot-113 IB. We report the average accuracy on three independent runs. The highest accuracy for each dataset is highlighted in bold.
Input Patterns
PIQA
HellaSwag BoolQ
AQuA
GSM8K
SWAMP
Pattern
80.2
63.3
65.5
28.3
32.2
51.6
Pattern 2
78.4
61.2
62.8
27.1
27.9
49.3
Pattern 3
75.7
60.7
63.6
28.1
30.4
49.8
Pattern 4
79.3
63.1
63.6
27.2
31.8
50.4
We follow the same experiment setups as stated in Section 5. Table compares all 4 patterns on three commonsense reasoning and three arithmetic reasoning tasks. The results of Pattern 2 indicate that without the self-attention mechanism to capture dependencies in the Copilot model's generated outputs, the Copilot model struggles to effectively leverage the additional hidden state information during fine-tuning and inference. Additionally, the results comparing Pattern 3 and Pattern do not reveal a clear performance trend. This suggests that the Pilot model makes mistakes at different layers depending on the assigned task. Therefore, the Mistake Log My should capture all hidden states from the Pilot model to ensure that relevant error-related information is omitted during the Copilot model's learning process. Based on this empirical analysis, we demonstrate the effectiveness of Pattern for our Copilot model design.
Choice of A. theorem 4.1, we theoretically provide bound on the range of with Here, we empirically study the effect of different À configurations. The results in Table 17 show that performance generally improves with larger À values in the range [0, 1]. The optimal value is observed around À = 1.0. The results demonstrate that higher À amplifies the effect of T-Copilot, which aligns with our Copilot model design.
< Ag-
Table 19: Comparison of pooling methods for LLaMA-3.1-8B + T-Copilot- IB.
Table 20: Comparison of loss functions for Qwen2.5-7B + T-Copilot-3B.
Pooling Methods GSMSK AQuA
Loss Type
PIQA
AQuA
Max Pooling
63.2
37.5
CE
88.1
60.7
Sum Pooling
65.8
38.3
KL Divergence
90.4
62.8
Mean Pooling 66.1
38.9
RMSE
92.5
64.2
Pooling Methods of Copilot's Attention Input. In Section refine the Copilot model's attention mechanism by incorporating the mean-pooled hidden representations from the Pilot model as inputs (Eq. 6). To assess the effect of this design choice, also experiment with alternative pooling strategies, including max pooling and sum pooling. The comparative results are presented in Table 19, From the results, the mean pooling empirically provides the best balance of stability and efficiency.
Loss Types of T-Copilot. We adopt the RMSE loss as the training objective for T-Copilot. To validate this choice, we conduct ablations with alternative loss formulations, including Pross-Entropy and KL Divergence. The results, summarized Table 20, show that RMSE consistently yields superior performance across benchmarks.
32

---

G
Additional Related Works
Transformers for Language Modeling The Transformer is a sequence-to-sequenee model archi- tecture that employs attention-based mechanisms, making highly effective for autoregressive language modeling [2, 13, 64]. The vanilla Transformer [77] follows an encoder-decoder structure, comprising a stack of identical layers in both the encoder and decoder components. Each layer consists of multi-head self-attention mechanism [15]. layer normalization [4], a position-wise feedforward network, and residual connection [29]. The encoder-decoder structure serves as the foundation for many carly-stage influential LLMs, such as T5 [65], and BART [44]. These models have demonstrated strong capabilities on certain generation tasks [38, 66). On the other hand, (causal) decoder-only Transformer models [62, 63), trained with the autoregressive language modeling objec- tive [69, 78], have demonstrated exceptional performance in open-ended generation and reasoning tasks [13, 82, 81]. The superior generalization capabilities have established decoder-only Trans- formers as the backbone of recent sfate-of-the-art LLMs such as PaLM [16], Falcon [3], LLaMA [76, 21J, and ChatGPT [1, 50, 35]. In this work, we develop the Transformer Copilot framework to support both encoder-decoder and decoder-only Transformer architectures. Our intuition is to provide flexibility across broad range of model configurations and downstream task scenarios.
LLMs Adaptation with Fine-tuning, Large language models perform well across many NLP tasks [98, 92], yet adapting them to specialized tasks remains challenging [80]. The standard solution, full-parameter fine-tuning [51], retrains all model parameters on task-specific data. Applying full fine-tuning has proven effective at improving performance, but can sometimes be computationally costly [73]. Recent work on parameter-efficeent fine-tuning approaches [39, 32, 86, 100, 28], such as prefix-tuning [46] and LoRA [32], aims to reduce the computational overhead by tuning only small subset of model parameters while still leveraging the expressive power of pre-trained models. Our learning framework builds upon the aforementioned methods' fine-tuning paradigm and aims to refine the fine-tuning and inference by utilizing mistake information during the models' learning trajectory. Additionally, since the Copilot model retains the decoder module structure, our framework can seamlessly integrate with various adaptation techniques such as DoRA [49] and ReFT [85].
Differences from Boosting and neural exploration. The core idea of Boosting [9, 24] is to train a series of models, where each subsequent model focuses on correcting the errors made by the previous ones. However, the proposed Copilot framework is distinct from boosting in several key ways. First, boosting, the subsequent model is trained to correct the errors of a fixed, post-trained weak model, whereas the Copilot learns from the mistakes (errors) made by a strong, pre-trained pilot model during its fine-tuning trajectory. Second, the labels (errors) that the subsequent model in boosting attempts to predict are derived from the fixed parameters of the preceding weak model, whereas the labels that the Copilot learns are based on the fine-tuning dynamics of the Pilot's parameters. Third, while all models in boosting only take data features inputs, the Copilot also takes the internal state of the Pilot model as part of its input. Fourth, boosting does not require modifications to the base models, whereas the Copilot framework involves modifying the model structure, specifically the Transformer architecture. Another related work is neural exploration methods [36, 87, 61, 5, 7, 83). For example, one recent work called EE-Net [6] introduces trade-offs between exploitation and exploration in the contextual bandits setting. In contrast, the Copilot focuses learning from the mistakes of the Pilot model in offine, supervised learning regime, specifically tailored for Transformer-based soquence-to-sqqenne generation tasks.
exploration neural network manage
33

---

H Broader impact and Limitation
Broader Impact. This paper introduces Transformer Copilot, a novel framework that enhances LLM fine-tuning by introducing Mistake Log and an auxiliary Copilot model that learns to rectify errors during inference. Our approach improves model reliability and efficiency with minimal overhead and promotes more transparent and interpretable behavior by grounding predictions in prior training dynamics. While our method has broad applicability across domains, we do not foresee any specific societal risks or negative impacts that require special consideration.
Limitation. While Transformer Copilot demonstrates robust improvements in inference quality by leveraging model-internal training signals, one potential consideration for future work lies in the coverage and diversity of the Mistake Log itself. Since the Mistake Log is constructed from the forward pass during supervised fine-tuning, its quality is inherently dependent on the richness and representativeness of the fine-tuning data distribution. In scenarios with limited domain coverage or skewed data sources, the Mistake Log may capture a narrower set of error patterns, potentially limiting the Copilot's generalizability. On the other hand, in our primary experimental setup, we fine-tune on task-diverse datasets with ample coverage, which ensures that the Mistake Log remains informative and representative. Our transferability experiments the Copilot model further validate the Mistake Log's s utility across unseen Pilot models, suggesting robustness to architectural and distributional shifts. Still, exploring data augmentation strategies adaptive logging policies to enrich Mistake Logs for low-resource or domain-shifie settings remains an interesting future direction.
Overall. Transformer Copilot offers promising paradigm shift toward internal signal utilization during LLM fine-tuning. We are optimistic that future research will build upon these contributions to develop even more precise and generalizable models through the continued adoption of reflective learning mechanisms.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims are summarized properly in our Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitation discussion is included in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Justification:
We provide detailed assumptions and complete proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the method implementation, training details, and experimental setup in the Experiment Section and Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification:
Our code implementation is submitted along with the manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppcccpppppaeeeccccecccccceeee /
swiisssssiiiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Experiment details are provided detail in the Appendix. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Each experiment's results reported as the average three independent experimental runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
37

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //terrip///uuip..../ tttpss//aarrssscpppbbb/cc//
ines?
Answer: [Yes]
Justification:
We comply with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include the broader impact discussion in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not observe such risks of misuse in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing resources are properly cited in this paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
39

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any real- dataset. Other code implementations are detailed as supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: No human objects are involved in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
40

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The LLM usage is described in detail in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http://eeeripp..pe cc/ cceeee/eeeeeeeece22255555555 for what should or should not be described.
41


Reviews:
[{"review_id": "jDAQzyj6mI", "reviewer": "Reviewer_R6Fu", "summary": "The paper introduces the Transformer Copilot (T-Copilot), a novel framework designed to enhance the performance of large language models during SFT by incorporating a reflective learning mechanism inspired by human learning strategies. The core innovation is the Mistake Log, which systematically records a model's training errors, input questions and internal hidden states. This log is leveraged by a Copilot model, which works alongside the original transformer-based Pilot model to refine its performance through error-aware corrections. The framework is evaluated on 12 benchmarks, achieving performance improvements of up to 34.5% over baseline Pilot models.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The introduction of the Mistake Log is a novel contribution, which systematically captures input representations, rationales, and token-level errors, providing a structured way to track and learn from a model's mistakes during fine-tuning. This approach aligns with the growing interest in error-aware learning in LLMs.\n\n2. The framework demonstrates strong scalability across different model types (encoder-decoder and decoder-only) and sizes, as well as transferability to new Pilot models without additional training costs.\n\n4. The empirical results are robust, supported by experiments on 12 benchmarks and comparisons with strong baselines.\n\nWeaknesses\n\n1. The Mistake Log captures the model's behavior and errors specific to the fine-tuning dataset, which may not generalize well to OOD data. The Copilot's error-correction capabilities might be less effective when the Pilot model encounters significantly different data distributions during inference.\n\n2. The Copilot model samples from the entire Mistake Log during training, but as the Pilot model evolves through fine-tuning iterations, there may be a gap between errors recorded in early stages (when the model is less optimized) and later stages (when the model is more refined). This temporal discrepancy could lead to the Copilot learning from outdated or less relevant error patterns, potentially reducing its effectiveness in assisting the Pilot model during later training or inference.", "was_revised": true, "final_justification": "The authors’ response has addressed most of my concerns, and I will maintain my positive score.", "rebuttal": "Thank you for your constructive and thoughtful suggestions and feedback! We will address each of your concerns below.\n\n---\n## **[W.1] OOD Generalization of Mistake Log.**\nThank you for the insightful question on the effectiveness of T-Copilot on the OOD data. In response:\n\n**1. T-Copilot's Design on Generality.**\n- **Mistake Log Encodes Data-Agnostic Error Patterns:** \nThe Mistake Log systematically records **model-internal signals** (e.g., hidden states representations) **besides dataset-specific idiosyncrasies**, enabling the Copilot to learn the Pilot’s inherent error patterns.\n- **Initialization from pre-trained backbones enhances generality:**\\\nAs noted in `Section 3.1`, the Copilot is initialized from the same pre-trained backbone as the Pilot, leveraging broad language knowledge and helping avoid overfitting to any single fine-tuning dataset.\n\n\n**2. New Cross-Benchmark (OOD setting) Evaluation.**\\\nFor a better demonstration, we fine-tune T-Copilot on arithmetic reasoning data (Math10k) and evaluate it on commonsense reasoning tasks (PIQA, HellaSwag, BoolQ, and OBQA) to **assess T-Copilot's performance in OOD settings**.\n|Model|PIQA|HellaS.|BoolQ|OBQA|\n|-|-|-|-|-|\n|LLaMA-3.1-8B|77.2|81.4|54.3|68.2|\n|**LLaMA-3.1-8B+T-Copilot-3B**|**79.5**|**83.1**|**56.2**|**70.6**|\n\n- From the results, although the Pilot model (LLaMA-3.1-8B) achieves less accuracy in the OOD setting compared to in-domain fine-tuning (Table 2 in paper), integrating T-Copilot-3B still yields consistent performance gains to the Pilot model.\n- This suggests that **T-Copilot remains effective in OOD settings due to its strong error-correction capability**.\n\n\n**3. Limitations and Future Work.**\\\nWe also acknowledge, as noted in **Appendix A (Lines 620–621)**, that T-Copilot may face limitations under large domain shifts. We have explicitly identified improving its generalizability to such settings as an important direction for future work.\n\n---\n## **[W.2] Temporal Discrepancy in Mistake Log Sampling.**\nThank you for mentioning this thoughtful point regarding the potential temporal drift in error relevance as the Pilot model evolves during fine-tuning. \n\n\nThe **main objective** of the Mistake Log, which collects and maintains the Pilot’s evolving hidden states, is to:\n- enrich the diversity of error patterns;\n- enable the Copilot to learn from and handle a broad range of error signals during inference.\n\n\nDuring the early design of the Mistake Log, we identified two key challenges:\n- **First**, as noted by the reviewer, randomly sampling from the full training history may introduce temporal discrepancies, leading to potential instability in Copilot training.\n- **Second**, in practice, it might be infeasible to store the entire Mistake Log on GPU due to memory constraints.\n\n**To address these issues**, we correspondingly introduce the Mistake Log Buffer (Lines 892–899), which retains only the most recent set of training rounds (e.g., 128). This design ensures that:\n- The Copilot is trained on relatively recent Pilot states, striking a **balance** between reducing temporal discrepancies and avoiding overfitting to the latest state;\n- The buffer can be efficiently stored and accessed on GPU, improving the method’s **practicality**.\n\nWe hope our explanations of the Mistake Log design address the reviewer’s question. We will also incorporate these rationales into our paper to improve clarity."}, {"review_id": "vTKyPzXlOA", "reviewer": "Reviewer_2chS", "summary": "This paper introduces a fine-tuning framework for large language models (LLMs) called Transformer Copilot (T-Copilot). The key innovation is the Mistake Log, a mechanism that records contextual inputs, hidden states, and token-level prediction errors during supervised fine-tuning. This log is used to train an auxiliary Copilot model that learns to rectify the predictions of the base Pilot model by adjusting its output logits at inference time.\nThe paper proposes a comprehensive design: (1) a Copilot model architecture with modified cross-attention to attend to the Mistake Log; (2) a joint training paradigm where the Pilot and Copilot models evolve together; and (3) a fused inference method where both models contribute to token generation. The authors provide theoretical guarantees and extensive empirical validation across a suite of commonsense, arithmetic, and recommendation tasks. The Copilot improves performance by up to 34.5% and can enable smaller models to outperform much larger baselines with reduced computational cost.", "questions": "How do you manage the memory and storage overhead associated with logging intermediate states (e.g., all hidden layers)? Could this limit scalability?\n\nThe paper briefly mentions alternatives like Self-Refine and Reinforced Self-Training. Could you add brief empirical comparisons to these in a future version?\n\nMost experiments focus on reasoning and recommendation. Would the Copilot framework generalize to tasks like summarization or code generation where the error modes differ?\n\nCan you provide intuition or empirical evidence supporting RMSE over cross-entropy or KL-divergence for error prediction?\n\nDid you observe any training instability when jointly optimizing two models with different objectives? How sensitive is the joint training to hyperparameters like learning rates?", "limitations": "The authors mention limitations in the appendix. A key concern is the additional storage and compute demands of collecting and training from the Mistake Log, particularly for large-scale finetuning. Additionally, the current method still requires full model retraining to pair Copilot and Pilot, and more flexible plug-and-play designs could broaden adoption.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThe paper is well-structured, methodically developed, and includes theoretical justification with clear assumptions. The proposed method is tested across a wide range of tasks and model sizes, demonstrating robust and consistent gains. The implementation is non-invasive and practical, requiring no architectural change to the Pilot and scales well.\n\nThe writing is clear and pedagogical, with helpful diagrams and motivating examples. The authors do a commendable job of connecting the technical components to the core conceptual intuition (i.e., learning from one's own mistakes).\n\nThis work proposes a new and impactful paradigm of learning from internal model dynamics instead of just data or external feedback. The approach shows promise for broad application in fine-tuning pipelines across domains.\n\nThe Mistake Log idea is new in its formalization and integration into the model training process. The dual-model inference with token-level rectification adds a unique twist not seen in typical self-correction or adapter-based methods.\n\n**Weakness**\n\nWhile the experiments are extensive, some design decisions, like the alternating self/cross-attention in the Copilot or the choice of RMSE loss, could use deeper ablation. Also, the Mistake Log requires storing large volumes of intermediate state data, which may become impractical for larger models or datasets, and this is not deeply discussed.\n\nSome components (e.g., joint training, fusion-based inference) borrow from existing ensemble or multitask architectures, though their combination here is original.", "was_revised": true, "final_justification": "Thanks for the author for their detailed responses. I appreciate the ablations on Alternating Self/Cross-attention in the Copilot, RMSE loss, and clarification on the weakness I raised. I increased my score from 3 to 4.", "rebuttal": "We sincerely thank the reviewer for the detailed and insightful feedback. We address each of your remaining concerns below.\n\n---\n## **[Weakness] Ablations on Alternating Self/Cross-attention in the Copilot.**\nThank you for acknowledging our experimental efforts. We provide additional ablations on T-Copilot's design choices. For the design choices of Copilot's modified attention, we conduct two detailed ablations.\n\n**1. Pooling Methods of Copilot's Attention Input.**\n\nWe explored other pooling methods, including max pooling and sum pooling. The results are reported below:\n|Pooling Methods (LLaMA-3.1-8B + T-Copilot-1B) |GSM8K|AQuA|\n|-|-|-|\n|Max Pooling|63.2|37.5|\n|Sum Pooling|65.8|38.3|\n|**Mean Pooling**|**66.1**|**38.9**|\n\nFrom the results, the mean pooling empirically provided the **best balance of stability and efficiency**.\n\n**2. Insertion Pattern of Copilot's Attention.**\n\nWe also want to highlight that, in `Appendix G.6`, we investigated **four different insertion patterns for Copilot's modified attention mechanism**. The full results are reported in Table 18 of our paper.  \n\n**References:**\\\n[1] Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective.\\\n[2] A Brief Overview of Universal Sentence Representation Methods: A Linguistic View.\n\n---\n## **[Weakness & Question] Ablations on RMSE Loss.** \n> \"Can you provide intuition ... for error prediction?\"\n\nWe thank the reviewer for inquiring about T-Copilot's loss types. Below, we provide conceptual rationale and empirical ablations on **RMSE loss for T-Copilot.**\n\n**1. Conceptual Rationale** \n\nRecall that the objective of the Copilot is to predict the distribution discrepancy, defined as $\\ell\\_t(p\\_{t,i}, \\hat{p}_{t,i}) = p\\_{t,i} - \\hat{p}\\_{t,i} $ in our paper. Note that $\\ell\\_t(p\\_{t,i}, \\hat{p}\\_{t,i})$ is not a valid distribution, since Sum$(\\ell\\_t(p\\_{t,i}, \\hat{p}\\_{t,i})) \\neq 1$. Therefore, it is natural to formulate this task as a **regression problem**, for which the RMSE loss is commonly adopted.\n\nIn contrast, CE and KL-divergence are for **distribution fitting** and are **not directly applicable here** unless we manipulate $\\ell\\_t(p\\_{t,i}, \\hat{p}\\_{t,i})$ with softmax to resemble a valid distribution. In this case, the loss is not directly optimized on the original discrepancy (i.e., $p\\_{t,i} - \\hat{p}\\_{t,i}$), resulting information loss.\n\n**2. Empirical Ablations** \n\nIn practice, we conducted experiments with all three loss types, and RMSE consistently achieved the best performance.\n|Loss Type (Qwen2.5-7B+T-Copilot-3B)|PIQA|AQuA|\n|-|-|-|\n|CE|88.1|60.7|\n|KL Divergence|90.4|62.8|\n|**RMSE**|**92.5**|**64.2**|\n\n---\n## **[Weakness & Question] Memory Overhead of Mistake Log.** \n>  \"How do you manage... Could this limit scalability?\"\n\nThank you for asking the memory storage of Mistake Log. \n\n**First**, we would like to elaborate on how our current method and implementation are designed to handle the storage of the Mistake Log:\n\n**1. Maintenance of the Mistake Log Buffer.**\\\nWe adopt a buffer mechanism for Mistake Log in our implementation to reduce memory usage in practice:\n- All logged tensors are detached from the GPU and moved to a fixed‑size CPU buffer (Lines 894-895); GPU memory is therefore almost unaffected.\n- We utilize a fixed‑size buffer that only retains the 128 training rounds in Mistake Log (Line 896).\n- Experiments show that adding the buffer reduces the overall storage by roughly 92% while maintaining a similar performance. \n    \n**2. Mean Pooling on Pilot's Forward Pass.**\\\nIn Eq. (6), we perform mean pooling across all layers' hidden state representations before feeding into the Copilot, meaning:\n- During the Mistake Log correction process, **we only need to store the mean pooled vectors rather than the full hidden states across all layers**, which significantly reduces vector storage.\n- During Pilot forward pass, the hidden states have already computed within transformer's residual stream. **We can directly access these hidden representations without allocating extra space for storage or backpropagation gradients**.\n\n**Second,** to further improve the scalability of our method and adapt to much larger models and data, we can slightly modify our design:\n\n**1. Selective Storage:** We can design a Mistake Log Gate to filter and collect hidden states information only when typical error patterns (mistakes) occur. \n\n**2. Lightweight Mistake Log:** Instead of curating Mistake Logs over the entire training process, we directly use the Pilot’s information from the current round's forward pass as input to the Copilot. \n\n---\n## **[Weakness] Difference with Existing Ensemble or Multitask Architectures.**\nThank you for recognizing T‑Copilot’s architectural novelty and its relation to related methods. We highlight two key distinctions: \n- **Error‑Aware Supervision vs. Output Averaging.**\\\n    Traditional ensembles combine each model’s logits predictions post‑hoc, without specialized training and inference paradigms for error-correction. Our Copilot can explicitly identify and correct the Pilot’s errors.\n- **Discrepancy Modeling vs. Direct Generation.**\\\n    Instead of generating next-token predictions, the Copilot learns to predict the discrepancy between the Pilot’s output and the ground truth and rectifies the Pilot’s logits.\n\n---\n## **[Question] Comparison with Self-Refine and Refinanced Self-Training.**\nThank you for your suggestions. Below, we provide a comparison between T-Copilot and Self-refine [1] for better demonstration.\n\n**Setups:** We compare Qwen2.5-7B+T-Copilot-3B with Self-Refine on GSM8K and MBPP. For Self-Refine, we follow its original protocol with 3 feedback-refine iterations.\n|Methods|GSM8K|MBPP|\n|-|-|-|\n|Self-refine|73.2|63.3|\n|T-Copilot|**79.7**|**68.9**|\n\n**Additional Analyses:**\nSelf-refine [1] and Reinforced Self-Training [2] rely on external feedback and offline RL, whereas T-Copilot leverages the model’s internal learning signals during SFT. \n\nWe sincerely thank the reviewer for the suggestion and will include the experiments above and additional comparisons in our paper revision.\n\n**References:**\\\n[1] Self-refine: Iterative refinement with self-feedback.\\\n[2] Reinforced self-training (rest) for language modeling. \n\n---\n## **[Question] Generalize to Summarization and Code Generation.**\nThank you for the question. Below, we conduct new experiments on **code generation and text summarization tasks**.\n\n- **Code Generation** \n    - **Training:** We randomly select 5k examples from CodeContests [1] as the training set.\n    - **Evaluation:** We evaluate the standard test set of MBPP [2] and LiveCodeBench [3] version-2 (511 problems). We report the pass@1 accuracy.\n- **Text Summarization** \n    - **Training:** We randomly sample 10k training examples from the CNN-DailyMail [4] and provide a training split.\n    - **Evaluation:** We evaluate on the standard test set of CNN-DailyMail and report the ROUGE-1 F1-score.\n\n|Model|MBPP|LiveCodeBench|CNN-DailyMail|\n|-|-|-|-|\n|Qwen2.5-3B|62.7|23.8|33.9|\n|**Qwen2.5-3B+T-Copilot-3B**|**65.3**|**26.4** |**35.3**|\n|Qwen2.5-7B|65.8|25.7| 35.1|\n|**Qwen2.5-7B+T-Copilot-3B**|**68.9**|**28.3**|**37.6**|\n\nAcross all three tasks, T‑Copilot improves performance over the base Pilot, demonstrating the generalizability to diverse generation tasks.\n\n**References:**\\\n[1] Competition-level code generation with alphacode.\\\n[2] Program synthesis with large language models.\\\n[3] Livecodebench: Holistic and contamination-free evaluation of large language models for code.\\\n[4] Get to the point: Summarization with pointer-generator networks.\n\n---\n## **[Question] Training Stability and Hyperparameter Sensitivity.**\n> Did you observe ... different objectives? How sensitive ... rates?\n\nThank you for the insightful questions regarding training stability. In our experiments, **we did not observe notable instability** when jointly training the Pilot and Copilot models:\n- **Jointly Trained but Separately Optimized:** In our training paradigm, Copilot receives inputs via the Mistake Log (i.e., **detached information** from Pilot’s forward pass). This careful design ensures that Copilot learns from Pilot’s Mistake Log **without perturbing Pilot’s weights**, thereby preventing interference between the two models and avoiding potential instability or bias.\n- **Adoption of Standard Optimizers and Schedules:** We use AdamW with a cosine schedule - widely adopted in LLM fine-tuning for stable convergence. We save model checkpoints every 1k steps and monitor loss curves to ensure both Pilot and Copilot remain well-behaved.\n- **Consistent Performance under Different Learning Rates (LRs)** In our hyperparameter tuning, we have tested different learning rates and observed consistent performance below:\n|LRs of Pilot (Qwen2.5-7B)|LRs of Copilot (T-Copilot-3B)|PIQA|HellaS.|SIQA|\n|-|-|-|-|-|\n|$3e^{-4}$|$1e^{-4}$|92.2|95.6|83.9|\n|$3e^{-4}$|$3e^{-4}$|92.2|95.8|84.2| \n|$3e^{-4}$|$5e^{-4}$|92.5|95.3|84.3|\n\n---\n## **[Limitation] Plug-and-Play Flexibility.**\n\nWe thank the reviewer for providing the detailed suggestions.  Regarding the **plug-and-play designs**, we would like to clarify that:\n- **Transferability of Copilot:** In `Appendix G.3`, our transferability experiments show that a Copilot trained with one Pilot can pair with other Pilots of the same type, achieving equal error-correction gains. This demonstrates that **Copilot is not tied to a single Pilot** and can be **used plug-and-play with similar models without retraining**.\n- **Lightweight Integration with PEFT:** In experiments, we show that T-Copilot is effective with both full fine-tuning and PEFT (e.g., LoRA) on LLaMA-3 and Qwen2.5 Pilots. PEFT updates only a small subset of parameters in Pilot and Copilot, substantially reducing training compute.\n- **Additional Adoption:** We will explore more efficient, plug‑and‑play designs of T‑Copilot (e.g., eliminating the need for any Pilot model retraining) to broaden adoption in our future work."}, {"review_id": "02YT0gAyRs", "reviewer": "Reviewer_LvNr", "summary": "This paper presents a new LLM adaptation method where a co-pilot model is learned to correct the predictions of the main fine-tuned model (called pilot). The pilot model is trained to predict the differences between the gold labels and the prob outputs of the pilot model given the input data and the internal states of the pilot model. During inference, the tokens are sampled from the combination of the outputs from both the pilot and copilot models. Theoretical analysis shows that the corrected outputs can better approximate the gold labels given some conditions. Experiments demonstrate the proposed method performs better on several benchmarks.", "questions": "See the weakness section above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The proposed pilot-copilot is a new method for LLM adaptation, where copilot is trained to exploit the internal signals from the pilot model to correct its output.\n2. Theoretical analysis is provided to show the corrected outputs can be a better approximation to the gold labels than the original prediction by the main model.\n3. Experimental results on several benchmarks suggest the effectiveness of the proposed method.\n4. The copilot can be transferred to another pilot model under some circumstances.\n\nWeakness:\n1. The assumption given in the main theorem seems strong. Is there any further theoretical or empirical justification that the bias of the corrected output should be lower than sqrt((bias of the pilot)^2 + (variance of the pilot)^2)?\n2. For the experiment settings, one natural baseline is to train a simple ensemble of two different models. It would be better to include some variants of ensemble learning in the experiments.", "was_revised": true, "final_justification": "The authors' response clears some of my concerns regarding experimental settings. I feel my initial rating is appropriate and I would like to see the paper accepted.", "rebuttal": "Thank you for your positive comments and insightful feedback on our paper. We will address each of your remaining concerns below.\n\n---\n## **[W.1] Assumption in Main Theorem 4.1**\nWe thank the reviewer for the insightful question regarding the assumption of Theorem 4.1. We would like to clarify and justify this assumption from **both theoretical and empirical perspectives.**\n\n**1. In theory:**\\\nWhile the assumption $\\epsilon_C^2<\\epsilon_P^2+\\sigma_P^2$ might look strong at first glance, upon closer inspection, **it is actually a mild requirement regarding the Copilot’s learning capability**.\n- As elaborated in **Remark 4.2**, the assumption implies that the Copilot can tolerate a larger bias (i.e., weaker learning capability) than the Pilot, i.e., $\\epsilon_P^2<\\epsilon_C^2<\\epsilon_P^2+\\sigma_P^2$.\n\n**2. In practice:**\\\nWe implement the Copilot under the same decoder backbone architecture as the Pilot, and it is of relatively smaller size.\n- We believe it is reasonable to expect the two models to have similar learning capabilities or biases, since the upper bound ensures the Copilot is actually learned during the SFT as the Pilot does.\n- We empirically measure on three held-out tasks (GSM8K, SVAMP, and PIQA). In all cases, the Copilot’s combined MSE is **15–20% lower** than the Pilot’s (i.e., $\\epsilon_C^2 <\\epsilon_P^2+\\sigma_P^2$), thereby empirically satisfying the assumption condition.\n\n---\n## **[W.2] Ensemble Learning Baseline.**\nThank you for the insightful suggestion to compare with ensemble learning variants. In response:\n\n1. We first **highlight the key differences** between T-Copilot and ensemble learning methods. \n2. We then **provide an empirical comparison** with the ensemble learning variant to demonstrate T-Copilot's error correction capabilities.\n\n**1. Difference with Ensemble Learning.**\\\nUnlike ensemble learning methods that aggregate independent model prediction outputs, T-Copilot leverages the Copilot trained to **explicitly correct the Pilot's errors** through the Mistake Log. Instead of generating next-token predictions directly, the Copilot learns to predict the discrepancy between the Pilot’s output and the ground truth, and rectifies the Pilot’s logits accordingly.\n\n**2. Empirical Comparison.**\\\nTo better demonstrate the **benefit of T-Copilot's error-correction capabilities**, beyond the simple ensemble of two models, we follow the reviewer's suggestion and conduct a new comparison experiment below.\n\n**Experiment Setups:** We utilize the same training and implementation setups from `Section 5` of our paper, and compare with the following variant:\n\n- **Pilot-Copilot-Ensemble**: Following works [1,2], we train Pilot and Copilot models independently and ensemble at inference by averaging their per‑token logits distribution to select each next token.\n\n**Evaluation Results:**\n| Pilot Model     | Variant     | AQuA | GSM8K | MAWPS | SVAMP | Avg. |\n|-----------------|-------------------|------|-------|-------|-------|------|\n|LLaMA-3.2-1B| Pilot-Copilot-Ensemble – 1B | 25.1 | 27.6  | 76.3  | 48.1  | 44.3 |\n|| **T-Copilot-1B**| **28.3** | **32.2**  | **81.5**  | **51.6**  | **48.4** |\n|LLaMA-3.2-3B| Pilot-Copilot-Ensemble – 1B| 33.3 | 54.2  | 85.9  | 65.4  | 59.7 |\n|| **T-Copilot-1B**| **36.6** | **58.2**  | **89.1**  | **68.7**  | **63.2** |\n|LLaMA-3.1-8B| Pilot-Copilot-Ensemble – 1B| 36.5 | 61.8  | 88.7  | 71.9  | 64.7 |\n|| **T-Copilot-1B**| **38.9** | **66.1**  | **90.8**  | **75.4**  | **67.8** |\n\n- From these results, T-Copilot consistently outperforms the simple ensemble learning variant across different tasks. \n- This confirms that T‑Copilot’s improvements are not merely due to “the integration of two models,” but stem from **its error‑aware rectification mechanism.**\n\nWe appreciate the reviewer’s suggestion and will include a thorough discussion and comparison with the ensemble learning variant in our paper, as shown above, to better highlight the advantages of T-Copilot.\n\n\n**References:**\\\n[1] Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration.\\\n[2] Harnessing Multiple Large Language Models: A Survey on LLM Ensemble."}, {"review_id": "Snvu7C7rR3", "reviewer": "Reviewer_sdqu", "summary": "This paper introduces a new fine-tuning paradigm, the Transformer Copilot that incorporates an additional Copilot model that learns from the base model's (termed Pilot model) mistakes saved in a Mistake Log. The Copilot model's objective is to correct the Pilot model's output logits to improve performance on down-stream tasks. There are extensive empirical analysis and evaluations provided to support the Transformer Copilot's success. Further, a simple theoretical analysis is provided to understand the efficacy of their method.", "questions": "- Is there any reason the Copilot model should be of the same backbone as the Pilot model? Could it be possible that training a different backbone from scratch, not fine-tuning, may lead to better results?\n- Have other pooling methods (besides mean pooling) been tested?\n- There seems to be a trend for increasing $\\lambda$ values to improve performance. What would be the effect of increasing $\\lambda > 1$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Well-written and structured paper, allowing for smooth reading and clear understanding of each section.\n- A novel approach to learn how to correct a model's ouput logits during inference. The method is intuitive and simple to understand. \n- Extensive experiments are conducted to validate the proposed methods successfully, particually appreciated Table 2 that compared performances with baselines of similar parameter count as this was a natural question that arose during the reading of the paper. \n- Emprical evidence is also provided to validate that the Copilot model indeed does adjust the Pilot model's logits significantly, leading to a change in final prediction. \n\nWeaknesses:\n- Theoretical justifications only focus on specific dimensions to prove a range of $\\lambda$ values that ensure improved performance. Is it possible to extend these to consider the vector as a whole? Since, each dimension $k$ may have a different range of $\\lambda$ that are disjoint, this could lead to scenarious where an optimal $\\lambda$ cannot be found. Though, the experimental results emprically validate the method sufficiently, this is more for theoretical curiousity. \n- In Figure 4, instead of showcasing a simple example of how the final token prediction has changed, it might be stronger to include some metrics that show how often Copilot changes the final token prediction correctly and wrongly.", "was_revised": true, "final_justification": "My initial assessment of the paper was already quite positive and the authors have carefully considered and responded to my questions. Overall, I believe this paper is a novel approach to fine-tuning that will contribute well to the community.", "rebuttal": "We sincerely thank the reviewer for your extensive and thoughtful feedback! We will address each of your remaining concerns below.\n\n---\n## **[W.1] Theoretical Justification for $\\lambda$**\nThank you for this insightful question! Below, we justify both theoretically and experimentally. \n\n> \"Is it possible to extend these to consider the vector as a whole?\"\n\n**Theoretically**: Yes, Theorem 1 can be extended to establish that there exists a constant $\\lambda$ such that the theoretical guarantee holds for all $K$ dimensions (where $K = |V|$). Specifically, recall that for each dimension $k \\in [K]$, there exists a threshold $\\lambda_0^k$ such that the guarantee holds for that dimension when $0 < \\lambda < \\lambda_0^k$, where\n$\\lambda_0^k = \\frac{2\\sqrt{\\epsilon_P^2+\\sigma_P^2}\\big(\\sqrt{\\epsilon_P^2+\\sigma_P^2}-\\epsilon_C\\big)}{\\big(\\sqrt{\\epsilon_P^2+\\sigma_P^2}-\\epsilon_C\\big)^2+\\sigma_C^2}$. Let $\\widehat{\\lambda}_0 = \\min\\{\\lambda_0^k: k \\in [K]\\}$. Then, the guarantee holds uniformly across all $K$ dimensions whenever $0 < \\lambda < \\widehat{\\lambda}_0$.\n\n**Empirically**: It is not always necessary for $\\lambda$ to fall within this range for all dimensions in practice. As the reviewer pointed out, it is often sufficient to choose a $\\lambda$ that satisfies $0 < \\lambda < \\lambda_0^k$ for the majority of dimensions $k$, which typically leads to good empirical performance (as demonstrated in Tables 1 and 2 of our paper).\n\n---\n## **[W.2] More Quantitative Analysis on Copilot Corrections**\nThank you for the helpful suggestion. In response, we include additional metrics in our error-correction analysis to quantify how often the Copilot correctly or incorrectly modifies the final token predictions.\n\nWe include the following new metrics:\n- **Correction Rate:** The percentage of test examples where the Copilot changes the final token prediction of the Pilot. \n- **Correction Precision:** Of those examples where the Copilot changes the token prediction, the percentage where the token correction leads to the correct final answers.\n- **Correction Error Rate:** Of the changed examples, the percentage where the Copilot’s correction leads to incorrect final results.\n\nWe randomly sampled **1,000 examples** from the test sets of **PIQA, BoolQ, AQuA, GSM8K, and LastFM** to ensure broad coverage across different tasks conducted in our paper.\n\n**Results:**\n|Metric|Avg. Value |\n|-|-|\n|Correction Rate|17.3%|\n|Correction Error Rate|7.4%|\n|Correction Precision|**92.6%**|\n\nThe results show that T-Copilot achieves much higher accuracy in correcting the Pilot’s logits predictions, highlighting its **robustness and effectiveness in error-correction**.\n\n\n---\n## **[Q.1] Backbone Choice for Copilot Model**\n>\"Is there any reason the Copilot model should be of the same backbone as the Pilot model? Could it be possible that training a different backbone from scratch, not fine-tuning, may lead to better results?\"\n\nThank you for this thoughtful question. In short, **T-Copilot does not restrict the Copilot to using the same backbone architecture as the Pilot; however, using the same backbone is more practical and simplifies implementation.**\n\n\nFor a more detailed response below: \n1. We **first** provide the key conceptual and practical reasons for adopting the same backbone as the Pilot;\n2. We **then** discuss potential extensions of the Copilot to alternative backbones, as suggested by the reviewer.\n\n**1. Justification for using the same backbones.**\n- **Unified Vocabulary Space:** Using the same backbone ensures that both the Pilot and Copilot share the same tokenization scheme and vocabulary space, enabling the Copilot's consistent token-level logits correction. Using different backbones might require additional vocabulary projection or alignment heads, which complicates the overall integration.  \n- **Compatibible Architecture and Residual Alignment:** The Copilot receives the Pilot’s hidden states and residual errors as inputs. Sharing the same backbone ensures representation compatibility, simplifying attention alignment, logits fusion, and decoding dynamics. A radically different backbone may potentially introduce misaligned inductive biases.\n- **Parameter Reuse and Training Efficiency:** Initializing the Copilot from the same backbone allows efficient reuse of pretrained knowledge and ensures stable training. However, training a distinct model from scratch may require substantially more data and computational resources to achieve comparable performance.\n\n**2. Extension to different backbones.**\nWe **completely agree** with the reviewer that T-Copilot can be smoothly extended to train a Copilot with different backbones from scratch (e.g., pretrain a Copilot model on residual errors across various tasks), potentially offering stronger error-correction capabilities. \n\nAchieving such (pre-)training scale would typically **require substantially more data and computational resources** than the current scope of our experiments allows. We appreciate the reviewer’s suggestion and plan to explore this direction in our future work.\n\n\n---\n## **[Q.2] Have other pooling methods (besides mean pooling) been tested?**\n\nThank you for the question. **Yes**, we have explored other pooling methods during our early stage method design, including max pooling and sum pooling. We report the results below:\n|Pooling Methods (LLaMA-3.1-8B + T-Copilot-1B) |GSM8K|AQuA|\n|-|-|-|\n|Max Pooling|63.2|37.5|\n|Sum Pooling|65.8|38.3|\n|**Mean Pooling**|**66.1**|**38.9**|\n\nWe adopted mean pooling over alternatives because:\n- In our preliminary experiments shown above, the mean pooling empirically provided the **best balance of stability and efficiency**.\n- Prior works [1,2,3] suggest that mean pooling is the most common and widely adopted approach for processing transformer hidden-state representations.\n\n**References:**\\\n[1] Sentence Embeddings using Siamese BERT-Networks.\\\n[2] Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective.\\\n[3] A Brief Overview of Universal Sentence Representation Methods: A Linguistic View.\n\n---\n## **[Q.3] Impact of $\\lambda > 1$**\n>\"What would be the effect of increasing $\\lambda > 1?$\"\n\nWe thank the reviewer for the detailed question on $\\lambda$. \n- **Intuitively**, as shown in Eq.(8), the Copilot primarily serves as an **\"assistant\"** to help the Pilot rectify its logits during generation. Therefore, the Copilot’s output should not be weighted more heavily than the Pilot’s (i.e., $\\lambda \\leq 1$).\n\n- **Empirically**, we follow our ablation study of $\\lambda$ in `Appendix G.6` and evaluate the downstream performance when $\\lambda > 1$ on LLaMA-3-8B + T-Copilot-1B:\n    |$\\lambda$| HellaS.|GSM8K|SVAMP|\n    |-|-|-|-|\n    |1.0|93.5|**66.1**|**75.4**|\n    |1.2|**94.2**|65.5|74.7|\n    |1.5|93.8|65.8|74.2|\n    \n    The results show that increasing $\\lambda > 1$ does not consistently yield additional gains on tasks such as GSM8K and SVAMP. Therefore, we consider $\\lambda = 1$ to be the **optimal setting** for T-Copilot during generation."}]

Author rebuttal:

Meta-review: This paper introduces the Transformer Copilot, a  framework designed to improve an LLM by learning from its errors during fine-tuning. The core contribution is the "Mistake Log," which systematically tracks the Pilot's errors and internal hidden states. This log is then used to train a secondary "Copilot" model that learns to rectify the Pilot's output logits at inference time. The paper shows that this approach leads to performance gains across a wide range of tasks.

On the positive side, the reviewers found that the paper has several strengths. First, the introduction of the Mistake Log is a nice contribution. Furthermore, reviewers noted that the empirical evaluation shows improvements across 12 benchmarks, which is quite thorough (Reviewers R6Fu, 2chS, sdqu). The authors show transferability across model sizes and architectures, justifying the generalizability of the approach (Reviewer R6Fu). Finally the theoretical results are non-trivial and add a foundational understanding to the approach of using a mistake log.

The reviewers identified areas of improvement, such as the initial need for a clearer justification of certain design choices, a more thorough analysis of the method's out-of-distribution (OOD) generalization, and a comparison against a standard ensemble learning baseline. Other points included the potential memory overhead associated with the Mistake Log and further ablation studies on the Copilot's architecture and loss function. During the discussion period, the authors provided additional experiments and clarifications that addressed these concerns, including new OOD results, a comparison to an ensemble baseline, and details on memory management. By incorporating this feedback, the authors will improve the paper, resulting in a more polished and high-quality final version.

Based on these reviews, I recommend accepting this paper. The reviewers were excited about the novel learning paradigm presented, particularly the formalization of the "Mistake Log" as a method for a model to learn from its own internal dynamics. The consensus was that the paper is well-written, thoroughly evaluated, and has strong empirical results. The authors were also responsive and effectively addressed the reviewers' concerns during the rebuttal period. I expect that the authors will incorporate the feedback and additional experiments from the review process into the final version of the paper.
Decision: Accept (spotlight)