MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
Authors: Yupeng Chen
Senmiao Wang
Yushun Zhang
Zhihang Lin
Haozhe Zhang
Weijian Sun
Tian Ding
Ruoyu Sun
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large corpora and subsequently fine-tuned on task-specific datasets. However, during fine-tuning, LLMs may forget some knowledge acquired in the pre-training stage, leading to a decline in general capabilities. Existing approaches to mitigate forgetting often rely on access to pre-training data, which may be unavailable in many real-world scenarios—such as fine-tuning checkpoint-only open-source LLMs. To address this challenge, we propose a new fine-tuning algorithm termed Momentum-Filtered Optimizer (MoFO). 
MoFO is an extension of greedy block coordinate descent (BCD) methods: in each iteration, MoFO only updates the model parameters with the largest momentum magnitudes, while keeping all other parameters fixed. MoFO achieves similar fine-tuning performance to the default fine-tuning algorithm while effectively mitigating knowledge forgetting. We validate MoFO through rigorous convergence analysis and extensive experiments, demonstrating its effectiveness in mitigating forgetting without pre-training data.

Published in Transactions on Machine Learning Research (10/2025)
MoFO: Mornntum--Fltered Optimizer for Mitigating Forget- ting in LLM Fine- Tening
Yupeng Chen" The Chinese University of Hong Kong, Shenzhen, China
gupengchen @link. calnkk.cchk.comm edu.co
Senmiao Wang"
senmiaauaagg Wlink, cuhk.
The Chinese University of Hong Kong, Shenzhen, China
Yushun Zhang
The Chinese University of Hong Kong, Shenzhen, China Shenzben Research Institute of Big Data
Zhihang Lin
Shenchen Research Institute of Big Dala
Haozhe Zhang
Huawei Technologies Ltd, Chine
Weijian Sun
Huawei Technologies Co., Ltd, China
Tian Ding
dingtian Eeigtian@sridd.cm
Shenzhen International Center for Industrial and Applied Mathematics Shenzben Research Institute Big Data
Ruoyu Sun'
wwwruoyuBuuk..eda
The Chinese University of Hong Kong, Shenzhen, China Shenzhen International Center for Industrial and Applied Mothematics Shenzhen Rlsearch Institute Big Data
Reviewed OpenReview: https: // opw.eeviee........ net/ per/forum? id= TIgXIDn9my
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large сorрoга and subsequently fine-tuned on task-specific datasets. However, during fine-tuning, LLMs may forget some knowledge acquired in the pre-training stage, leading to decline in general capabilities. Existing approaches mitigate forgetting often rely on accoss to pre-training data, which may be unavailable in many real-world scenarios such fine-tuning checkpoint-only open- source LLMs. To address this challenge, propose new fine-tuning algorithm termed Comentuu-FFlterrt Optimizer (MoFO). MoFO is an extension of greedy block coordinate descent (BCD) methods: in each iteration, MoFO only updates the model parameters with the largest, momentum magnitudes, while keeping all other parameters fixed. MoFO achieves similar fine- tun-ttning performance to the default fine-tuning algorithm while effectively mitigating knowlodge forgetting. We validate MoFO through rigorous convergence analysis and extensive experiments, demonstrating its effectiveness in mitigating forgetting without pre-training data
"Equal contribution.
1 (Corrsppondigg author.
'Our code available httpps//gStthb. com/YChen- z2z/NoFO

---

Published in Transactions on Machine Learning Research (10/2025)
1 Introduction
The success of large language models (LLMs) lies in their strong capabilities in language understanding and generation. Typically, LLMs are first pre-trained on extensive corpora to acquire general capabilities, then fine-tuned on smaller, task-specific datasets to adapt to particular tasks or domains (Dai & Le 2015 Kenton
& Toulanova 2019;
2018). However, it has been observed that during the fine-tuning process, LLMs may forget the knowledge acquired in pre-training, leading to decline in general capabilities (Lin
Radford al.]
et al. 2023 Chen et al.) 2020 Dong et al. 2021 Korbak et al. 2022 Luo et al. 2023ah. Therefore, addressing
the forgetting issue in LLM fine-tuning has become an important rescarch direction.
In the field of continual learning, mitigating forgetting has already been a central focus. Continual learning [Wann et al.) (024a) involves training models sequentially on different tasks, which is analogous to the process of ppe-traiiing followed by fine-tuning in LLMs: Both involve different stages of training and face the challenge of forgetting previously acquired knowledge when learning new information. To address this forgetting issue, replay-based methods (Rolnick et al., 2019 Wang et al. 2020 Ouyang et al. 2022) use replay buffer to store and revisit past data, in order to reinforce prior knowledge while learning new information. In LLM training, similar replay-based methods are also used to mitigate forgetting (Shi et al. 2024 Roziere et al, 2023 Huang et al. ) However, replay-based methods face some practical limitations in LLMs. First, the access to the original pre-training data is often restricted or infeasible. Many open-source LLMs, such as the Llama series (Touvron et al.] (023), do not fully disclose their pre-training datasets. Second, even when pre-training data is available, incorporating it into the fine-tuning process can substantially increase computational and memory costs, as the model must process much larger and more diverse dataset.
In continual learning, another class of methods focuses on modifying the optimization process of models to mitigate forgetting (Wang et al. (024a).. These methods do not require access to the pre-training data. However, many of them need to store and exploit the pre-training weights and/or intermediate gradients throughout most of the fine-tuning process. For example, some regularization--ass methods such as Ln or Ti2 regularization (Panigrahi et al. 2023 Li ct al., 2018) require storing the pre-training weights for regularization computation during the whole fine-tuning process. However, in the context of LLMs, storing additional model weights requires substantial memory due to the large model size, introducing considerable overhead.
Given the limitations of prevalent algorithm that neither relies on past data nor introduces additional memory overhead. In general, our algorithm design goals are twofold: (G1) preservation of pretrained knowledge and (G2) strong performance on the fine-tuning task. To pursue (G1), we adopt a key insight from continual learning Kirkpatrick et al. 2:17]: the closer a fino-tuned model stays to the pretrained paramcters, the less forgetting tends to occur. At the same time, enforcing proximity too aggressively can hinder adaptation to the new task, thereby harming (G2). This raises natural question: What optimization method might move small distance from the inctial point, but can still find a minimam 2 of the loss function?
approaches, we aim to develop an optimization
We notice that the classical block coordinate descent (BCD) method Tseng 2001 is a good candidate, since it updates only a subset of parameters at each iteration, thus implicitly biased towards closer solutions. Nevertheless, incorporating BCD into LLM fine-tuning presents some challenges. One challenge is that Adam, the predominant optimizer for LLM training (Radford et al., 2018 Zhang et al. 2024c). differs substantially from the optimizer studied in traditional BCD methods (e.g. GD SGD) (Nutini et al. 2015 Zhao et al.) 2014 It complicates both optimizer design and convergence analysis. Consequently, combining BCD with Adam is not a straightforward task.
In this work, we propose Momentum-Filterdd Optimizer (MoFO), a new optimization algorithm that integrates Adam with BCD. At each iteration, MoFO only updates the most effective parameters for reducing the fine-tuning loss those with large momentum magnitudes, while keeping other parameters fixed. MoFO only modifies the optimizer without the need for pre-training data or introducing additional memory costs, which helps achieve its efficiency and effectiveness during fine-tuning. Our contributions are summarized as follows:
IIn this paper, we the term 'minimum' (or 'minima' in the plural) to refer a parameter configuration whose fino-tuning loss is near its lowest value small neighborhood, while acknowledging that this terminology may not strictly represent local minimum of the fine-tuning loss function.
2

---

Published in Transactions on Machine Learning Research (10/2025)
2.0
Pre-trained model
Pre-trained model
4.0
6
Lion
1.4
Lion
3.7
0.7
3.3
Adam
Adam
4
0.0
4
3.0
2.7
2
-0.6
2.3
dAdam
dlian
-1.3
d
2.0
0
*
0
-2.0
1.7
1.3
-2
-2.7
-2
1.0
-2.5
0.0
2.5
5.0
7.5
-2.5
0.0
2.5
5.0
7.5
(a) Fine-tuning loss landscape
(b) Pre-training loss landscape
The loss landscapes of Pythia-160M after fine-tuning on a subset of the FLAN dataset using Adam and Lion. We plot the loss landscapes on (a) the fino-tuning dataset and (b) the pro-training datasot (Pile datasel (Gao et. al., 2020)). We visualize 2D weight-space plane spanned by the vector from the pre-trained model to the Lion-tuned model (x-axis) and to the Adam-tuned model (y-axis). Axes are normalized that one unit equals the length of the pre-trained-AAdam vector. The color bar indicates the loss value-(a) fine-tuning loss and (b) pre-training loss. A logarithmic scale is applied to the loss values for better visualization. Two training methods converge to different minima with similar fine-tuning loss. Lion converges to a farther minimum from the pre- traiera model and performs more forgetting than Adam.
Figure 1:
We propose MoFO, training algorithm designed to mitigate the forgetting of pre-training knowledge during fine-tuning.
We present rigorous theoretical convergence result of the MoFO algorithm, providing a solid theoretical foundation that supports its good performance in fine-tuning tasks.
We conduct experiments on various lasks, demonstrating that MoFO outperfomss existing methods both in fine-tuning performance and mitigating forgetting.
2 Momentum Filtered Optimizer (MoFO)
2.1 Motivation
Correlation between Distance and Forgetting.
In LLM fine-tuning, different training methods typically converge to different minima of the loss function. These minima may yield similarly low fine-tuning loss, all achieving reasonably good fine-tuning performance. However, their distances from the pre-trained model can vary significantly. A key observation in traditional continual learning (CL) is that the exlent of forgetting increases as the model deviates further from its original state. This insight has influenced the design of many forgetting-mitaan mitigation methods (Krkpatrick et al. 2017 Li al. 2018).
In this work, conduct exploratory experiments to investigate this correlation among minima produced by different LLM fine-tuning methods. We conduct two sets of experiments. First, fine-tune Pythia- 160a on subset of the FLAN dataser using Adam (Kingına & Ba 2014) and Lion (Chen et al. 2024). As illustrated in Figure []), Adam and Lion converge to distinct minima. Notably, while both optimizers achieve similar fine-tuning losses, the minimum reached by Adam much closer to the pre-trained model compared to Lion, being only about 20% of Lion's distance. (Specifically we measure the model distance by the average L2 norm difference over parameter blocks; see Appendix C.4 for the formal definition). As shown in Figure [b),
Thhe subset. used is 'definite pronoun resolution 10templates, available https://muggingffacccc//aaasss Muennighoff/flan The learning rate 26-5 and the batch set. as 64.
3

---

Published in Transactions on Machine Learning Research (10/2025)
2.75
MoFO
Pretrained Model
Adam
Loss on RedPajama
2.50
Lion
0.40
correlation -0.79
2.25
Epoch (154 iter)
Epoch 1.0 (309 iter)
MMUU
0.35
MoFO
2.00
Epoch 1.5 (463 iter)
Adam
Epoch 2.0 (618 iter)
Lian
1.75
0.30
Epoch 0.5 (154 iter)
1.50
correlation = 0.97
Epoch 1.0 (309 iter)
Epoch 1.5 (463 iter)
1.25
Pretrained Model
0.25
Epoch 2.0 (618 iter)
10-2
10
10-2
10-1
Distance
Distance
(a) Losses RedPajama
(b) Acc. on MMLU
Figure 2: (a) Loss changes on the RedPajama dataset and (b) average accuracy changes on MMLU benchmark (measuring the preservation of factual knowledge) of Llama-2-7B after fine-tuning on MetaMathQA using Adam, Lion, and MoFO for 0.5, 1, 1.5, epochs. We note that Padaaaamaa project was explicitly designed open-source reproduction of the LLaMA training dataset (Weber et al. 2024h. Thus, it serves reasonable proxy for the original MAa training dalaset since the latter has not been publicly released. See Appendix [B.1]oo for the rationale. The results show a strong positive correlation between the distance from the pre-trained model and the extent forgetting after one epoch. Further discussion of carly-training behavior and a comparison of different optimizers are provided in Appendix B.3
Adam exhibits a smaller increase in pro-training loss than Lion. Further, Adam leads to better preservation in general capability than Lion (see Appendix B.2
Second, we conduct a larger-scale exploratory experiment to test the generality of this observation. We fine-tune larger model, LLaMA2-7B, on the MetaMathQA dataset (Yu et al. 2024b) using three optimizers: Adam, Lion, and our proposed MoFO (to be introduced in Section 2.2 To achieve varying distances from the pro-trained model, fine-tune each model for 0.5, 1.5, and epochs (309 steps per epoch). Details are provided in Appendix C.3 Figure 2 a) demonstrates a strong positive correlation between distance and the increase in pre-training loss. Figure bb indicates a negative correlation with MMLU accuracy (Hendrycks) et al. 2021), which measures the preservation of factual knowledge, once the model has been trained for at least one epoch. Taken together, these results suggest that, during LLM fine-tuning, remaining closer to the pre-trained state appears be associated with reduced forgetting. Further discussion of early-training behavior and a comparison of different optimizers are provided in Appendix B.3
Selective Rule Based Momentum Magnitudes
Motivated by the correlation between forgetting and the distance from the pre-trained model, we seek to design an optimizer that encourages the fine-tuned model to keep closer to the pre-trained model. To achieve this, we draw inspiration from the classical block coordinate descent (BCD) method [Tengg 2001). which updates only a subset of parameters during each iteration. We anticipate that restricting updates a subset of parameters- similar to the BCD approach-wi result in smaller overall deviations from the pre-trained model compared to full-parameter updates across all iterations, thereby mitigating the forgetting of pre-training knowledge.
To further accelerate convergence under the BCD framework, we adopt Gauss-Southwell rule, i.c., the greedy rule (Nutini et al.] 2015). Gauss-Southwell rule selects the parameters with the largest gradients at each iteration, as those are expected to yield the greatest immediate reduction in the loss. It has also been shown in Nutini al. (2015) that BCD using the Gauss-Southwell rulc-also referred to as greody BCD can converge faster than the traditional random BCD. However, BCD algorithns, including greedy BCD, are mostly developed based the GD SGD framework, but in LLM training, Adam has replaced SGD
4

---

Published in Transactions on Machine Learning Research (10/2025)
Adam Iteration
MOFO
Iteration
Iteration 1+1
Momentum
Gradient
Filter
Gradient
Parameters
Figure 3: Illustration of MoFO.
the default optimizer (Zhang al. 2024ch. Our experiments in Section 4.4 show that directly following the Gauss- Southwell rule in Adam that is, always updating parameters with large gradients- does not lead to satisfactory performance on fine-tuning tasks.
Adam inherently incorporates momentum term in parameter updates. Therefore, we propose to modify the Adam optimizer to update only the parameters with the largest momenbum magnitudes. By focusing on partial yet significant updates, our method, named MoFO, aims effectively fine-tune models while maintaining closer to their pre-trained state. We first introduce MoFO in the next subsection. Further theoretical analysis and empirical exploration in the selection rule will be provided in Section 3.2 and respectively.
2.2 Formulation of MoFO
Algorithm Momentum Filtered Optimizer (MoFO)
1: Input: Filtering threshold a, number partitions with the k-th partition of size de, hyperparameters B2 of Adam optimizer, learning rate schedule {m Initialize mo: vo
tensors.
for iteration from 1,2.... until converge do
4: for partition from 1 to B do
Dt VinsLrenettnd (6,-1)
(k)
(k) 3im! (k) + (1 31) (k)
(k) = 32ut-1 +(1 - -32)9
(k) gt.
7:
(k) = my
/11-/(11)) (k) 1 B1)
8:
m
0f (6) (k) /11- (3))
10:
for entry index i from 1 to de do
[FLT! (m:)); 1 if (mm()),
(k)
(x))a within the top-a of Im ( values clse
11:
12:
end for
13: (()) 0
(m (R)
о FLT (k)
(m.))/ (())
# Momentum Filtering
14: end for
0. = Concat (0 (...,)(B))
15:
16: end for
We formally introduce the Momennuum-Fiteers Optimizer (MoFO) in Algorithm ! First, all model parameters are partitioned into B blocks. At each iteration, MoFO first computes the gradient and momentum terms for parameters in each block following the standard rule of Adam, as shown in Lines 5-9. Then, MoFO selects and updates the parameter entries with the largest a momentum magnitudes in each parameter block,

---

Published in Transactions on Machine Learning Research (10/2025)
shown in Lines 10-13, where the update fraction is pre-determined hyperparameter. This momentum filtering mechanism is illustrated in Figure
Mathematically, the filter can be represented as follows. Consider momentum vector m = (m (2) ..... m mBB),
where each m(() E Rd corresponds to the k-th block of parameters with dimensionality dg. The top-a filter, (FLT(") denoted as (m), is defined as FLT (m) (FLT& (m), (m.,...,,LLI) (B) (m)), where the i-th entry of FLT (m) is given by
(R
F [. = {1 if (mm [m,a) is within the top-a of |m( values, for i 1,2, ..2,,.... , k 1,2 .... ...
0 otherwise,
,. In our Momentum-Filterns Optimizer (MoFO), this filter FLT, is applied to the momentum my, selecting the entries with the largest magnitudes for updating.
For the parameter partitioning, note that the network architecture is naturally composed of different modules (e.g,, weight matrices, and bias terms). In the PyTorch implementation, the parameters of different modules (along with their gradients and momenta) are naturally stored in separate data tensors. Therefore, we adopt the default partitioning of model parameters as implemented in PyTorch. For Transformers, this means that parameters such as query (Q), key (K), value (V) weights in the attention layers, as well as feed-forward network (FFN) weights, are grouped inlo distinct partitions following PyTorch's 's default. scheme. This allows us to select and update the top-a parameters in each block without introducing much implementation overhcad. See Appendix C.4 for further explanation of the partitioning.
At each ileration, MoFO efficiently selects and updates the most. "influential" parameters, as dictated by the momentum's magnitude, while keeping other parameters fixed. We argue that filtering the momentum is more effective than filtering the gradient. In Section [.4] we will empirically demonstrate that MoFO's um-beut filtering rule outperforms other filtering rules fine-tuning tasks.
3 Theoretical Analysis
3.1 Convergence Result
In this section, present the convergence result of MoFO for non-convex loss functions. For the simplicity of analysis, we consider the full-batch version of MoFO, with hyperparameters satisfying the following assumption.
Assumption 1. Loss function L lower bounded by L The gradient VL is Lipschitz continuous with constont L.
Theorem 1 (Convergence of MoFO). Suppose that the first- and second-order momentum hyperporameters 31 and B2 satisfy 0 < B1 < V < 1. The learning rate schedule at step t is nι = 1// Vt for some nj > 0. Then, under Assumplion 1 MoFO satisfies
[VV)) @ FLT. (VE(0))) = ( VT
(log] T (l08T
min 0S1ST-1
as T - xo.
Moreover, this bound directly implies
VL = (0(o) logg T /T
min 0S1ST-1
as T + 00,
for any p E oo).
Although MoFO is designed to mitigate forgetting by updating only a small subset of parameters at each step, it is guaranteed to converge to critical point of the fine-tuning loss function under the Lipschitz smoothness assumption. This result provides theoretical evidence that MoFO can achieve competitive performance in fine-tuning tasks.
Proof Sketch of Theorem 7. Our proof is inspired by the convergence analysis for full-batch Adam Shi et al. (2021), but we introduce additional techniques tailored to MoFO's filtering mechanism. We will highlight these additions precisely at the points where they arise below.

---

Published in Transactions on Machine Learning Research (10/2025)
Let 9: V.((0-1). A central step is to establish, for suitable constants C1, C2 0, Vi |1aal S L(0x-1) [()e) + C2
G
(1)
Summing this inequality from = 1 to T and using Et=1 t- log + ((1) yields the convergence result for Adam in terms of diminishing norm of gradient, given by
a = log T ISST
min
(2)
VT
Choice of norm and two subgoals. In finite-dimensional spaces, all norms are equivalent, so convergence statements like (2) can be expressed in any fixed norm up to norm-equivalence constants. In practice, however, specific analyses instantiate (I) with a particular norm: Shi et al. (2021) work with the L1 norm ll for full-batch Adam, whereas full-batch MoFO analysis will use the Li,to-- norm 11 FLT, (9.)ll; which will be defined in Appendix A.1
To keep the logic precise, we separate our argument into two subgoals:
Step I (Key inequality for MoFO). Show that there exist constants C₁, C2 > (independent of t) such that
C₁ Vi |ls (LT s L(0,-1) -(0.-1)-((0)) [(0.) + C2
Here, we recall that (-) preserves the a-fraction of largest-magnitude coordinates in each partition and zeros out the rest.
Step II (Norm property of the left-hand side of (3) This component one of our new technical ingredients (absent from Shi et al. (2021)). We first verify that the mapping x + || O FLT.(x)]] defines norm on R., which is referred to Littop- norm. This is proved in Proposition (Appendix A.1 by checking nonnegativity and definiteness, positive homogeneity, and the triangle inequality. Moreover, for the Lp upper bound in Theorem we use the norm equivalence between the Littop- norm and the Lp norm, shown in Lemma
With Step II established, remains prove the key incquality in Step A direct adaptation of Shi et al. (2021) to MoFO is infeasible due structural differences. We proceed follows:
(i) Recap key elements from Shi
(2021)
(ii) Identify challenges in extending them to MoFO;
(iii) Resolve these challenges by carefully handling the momentum filter.
Part (i): Key elements of Shi et al. (2021). For full-batch Adam with bias-corrected moments myg Og and the learning rate schedule TI 1// Vt, the parameter update is
m
0--0,1= VE
Vo
By the L-smoothness of the loss and the descent lemma,
Dit
Dost
miat
<((0,-)) L(0.) 2 - 0.-1113.
L
(4)
Lemma 5 in Appendix
A.2 lower-bounds the per-coordinate contribution as
Mii
B
Algi,t
Pi.t
Di,t
Vi
7

---

Published in Transactions on Machine Learning Research (10/2025)
for some constants A, > 0. Substituting this into summing coordinates, and controlling the with the Ly-norm:
quadratic term in 44 yields the fundamental inequality
C1 Vt 19.ll1 - ((0--1) C(O,) + t
C2
Part (i): Challenges in extending to MoFO. For the full-batch version of MoFO, the parameter update becomes
"f me Vi V o FLTa (me),
i.e., the step is filtered by momentum magnitudes (NOT gradient magnitudes). Building upon the convergence analysis of Shi et al. (2021) in Part (i) leads to
Ci
ll @ FLTa(mllli S. ((0--1)-CC)) C2 Vi
which brings a notable difference to the target key incquality 8 the incquality (5) applics the momentum filter FLTa(m;), whereas the desired bound applics the gradient filter FLTx(9)). This introduces a non-trivial challenge because the |[9. @ FLT, (me)ll being small does not. naturally imply that llg. @ FLT, (gt)lla is small: large entries of It might be excluded if their momenta lags. Consequently, additional analysis required to bound the discrepancy between them.
Part (iii): How we overcome the challenge. Here we introduce another new technique to address the discrepancy between lg.t @ FLT. (m and l1 @ FLTa(ggt)) (gt)ll. We deal with the challenge via Lemma [ in Appendix A.1 which establishes that the Li-deviation in filtered outputs is Lipschilz-stabll under input perturbations. Specifically, for any 2, E Rd.
- x O - S -
tap-a filtered error
imput
This result points out that while the filter FLT,()) itself is unstable under perturbation, the filtered out- put remains controllable through. It effectively "smooths' the discontinuity that would plausibly prevent convergence analysis.
With the Lipschitz stability established, we now deal with the challenge. We first control the Lı distance between the bias-corrected momentum th 1- and the gradient St by showing that ||M -gell O(1vVt. Second, we apply Lemma with I = g: and y = my and yield
1g FLTa(g:)) llg. FLT. (m;) s 2 gell1 O(1/V))
Since the filtering function FLT.(.) (-) is invariant under positive scaling, can definitely replace my with the original momentum m,:
1gg O (L)lll:)ls llg. @ (m:)( s 2 gell1 = O1/Vi))
Combining it with and subsuming residual O(1/t) terms, we obtain our target key incquality 3
C2 +
ll.. c FLT. (9.) < ((0--1) L(0,) +
In conclusion, statement of Step I primarily comprises:
Step I.1 (Part (i) and (ii). Extending the Adam convergence framework (Shi et al. 2021 to
incorporate MoFO's momentum filtering mechanism, yielding (5
Step I.2 (Part (iii)). Resolving the momentum-gradeeeee gradient filter discrepancy via our Lipschitz stability analysis (Temma [] which is new technique introduced in this work.

---

Published in Transactions on Machine Learning Research (10/2025)
Combined with the norm property in Step II these results enable us to establish Theorem 1
another new ingredient not present in Shi et al. (0021
We remark that the choice of 31 and 82 in Assumption [ aligns with that used in analyzing full-batch Adam (Shi ct al. 2021). Furthermore, the use of a diminishing learning rate in Thcorem is crucial for ensuring the stability updates and avoiding divergence in the optimization process.
In summary, Theorem ! demonstrates that despite updating only a subset of parameters, MoFO maintains the same convergence rate Adam. This highlights the theoretical robustness of the momentum filter design in MoFO. We believe this result could provide valuable insights into adaptive optimization methods with filtering mechanisms.
3.2 Initial Analysis on Forgetting Mitigation
Does MoFO converge to model that is closer to the pre-trained LLM than Adam, thereby reducing forgetting? In this subsection, we attempt to address this question by providing an initial theoretical analysis on an illustrative example.
Example 1. Suppose the parameter space is Rd, and the updating ratio is a 1/d, i.e., only one coordinate is updated in each iteration. We assume the pre-training loss is Lpretrain (0) = ] 1000 and that the model has been truined to the global minimum Ppretrain = (0, O, .,.... during the pre-training phase. The fine-tuning loss is given by L(6)= = IT (a:0; b,)2, where a, ba > O for any 1 I iis d. In this example, the set of global minima of L(0) is a union of hyperplanes:
d S = U
S. where S; =- 10 E Rd: = b./a,)
1=1
Here, each S; represents a hyperplune in Rd.
Remark 1. We note that the loss landscapes of neural nelworks are generally non-conver Lóu et al. 2022) Here, we also adopt a non- conver fine-tuning loss [(0). Further, we note that the set of global-minima S consists of infinitely many minima spread across multiple hyperplanes, aligning with the observation on the degenerate structure of minima in neural networks [Lin et al. 20246).
Theorem 2. In Example [ if the learning rates are chosen appropriately, then MoFO converges to a minimuum 0% MOOO that is closer to the pre-training state than the minimuum O' oblained by Adum, i.e.,
[10aPo @pretrain ll2 < 110 0prttriill2..
Morcover, MoFO attains strictly lower pre-training loss, i.C.,
Spretrain (f < Spretrain (0 (ddam))
Next, we visualize the landscape of this example to provide additional intuition. Consider a simplified two-dimensional case. Specifically, let 0 E IR2 and L(() = (0, (0,
In Example | each hyperplane S,, which forms part of the global minima of L, can be viewed as an attractor for the optimization algorithms. These attractors (S,'s) influence the model's update direction during training. As illustrated in Figure 4 the attractors in this case are two straight. lines: 0, = 1 and 02 1. When using Adam, the model is simultaneously pulled by both attractors, causing it to move diagonally along the orange line and converge at 1), with resulting pre-training loss of 1.
In contrast, MoFO influenced by only one attractorr (0) 1), leading it to follow the green line and converge to (1,0), which achieves a lower pre-training loss of 0 We hypothesize that, for fall-parameter fine-tuning (Adam), interference among multiple attractors drives convergence toward a "balanced" solution, whercas
By symmetry, MoFO may also he influenced by another attractor 0g and converges vertically (0. 1)-

---

Published in Transactions on Machine Learning Research (10/2025)
1.50
Default FT (Adam)
1.2
1.25
MoFO
Pre-trained model
-0.0
1.00
-1.2
0.75
-2.4
0.50
3.6
-4.8
0.00
0.25
-15.0
-0550
-0.50-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
6.
Figure 4: The fine-tuning loss landscape and the training paths of different optimization methods. The color bar indicates the fine-tuning loss value.
A logarithmic scale is applied to the loss values for better visualization. MoFO converges to minimum closest to the pre-trained model.
MoFO mitigates such interference by selectively updating only parameters with large momentum magnitudes. Such an updating rule helps MoFO move toward a single altractor, thereby converging closer and lorgetting less. In addition Adam and MoFO, we provide convergence paths of some other baseline methods in Appendix B.4
We believe that the above analysis provides an initial insight into the effectiveness of MoFO. More in-depth analysis of MoFO in forgetting mitigation is left for future work.
4 Experiments
4.1
Experimental Settings
We verify the effectiveness of MoFO instruction fine-tuning and continual fine-tuning. We
Llama-2-7B TTovrron et al. 2023 Gemma-2B-IT (Team et al. 2024 and TinyLlama-1.1B (Zhag et al., 2024b) as our base models. The instruction fine-tuning datasets cover question-answer pairs from different. domains like mathematical reasoning and medical knowledge. Specifically, the datasets include: MetaMathQQ Mor-LLLLMM--nnstuttiotn (Wu et al. 2024), Magicoder-EBi Eesioodrrr-yo----struee (Wei et al., 2023). We randomly sample 39.5K and 51K instances from these datasets, respectively, for training the LLMs. Additionally, We investigate the performance of MoFO in the continual fine-tuning scenario by implementing our approach on the TRACE benchmark dataset (Wang al. 2023b).
Yuu et al., 2024b),
Evaluation metrics for instruction fine-tuning. We employ widely used benchmarks to assess the performance and potential forgetting effects on the general capabilities of LLMs after instruction fine-tuning.
These benchmarks include MMLU (Hendrycks et al. 2021) (0-shot) for factual knowledge; ARC-Challenge,
ARC-Easy (Clark et al. 2018). and HellaSwag (Zellers et al. 2019) (0-shot) for commonsense reasoning (CR):
GSMSK (Cobbe et al. 2021 (5-shot) for mathematical reasoning; HumanEvval (HEval) (Chen et al. 2021
pass ① 10) for code generation; PubMedQA (Jin et al.) 2019), ModMCQA (Pal al., 2022). and McdQA (Jin
et al. 2021 (0-shot)) for medical question answering (MedQ) 5 IFEval (0-shot) for instruction following.
Evaluation metrics for continual fine-tuning. To evaluate the LLM's performance in continual learning, we consider two key metrics in this secnario: Overall Performance (OP) CChaddhry et al. 2018 and BackWard Transfer (BWT) [Lopez-Paz & Ranzalo, 2017).
For more descriptions and implementation details of these metrics and datasets, see Appendix C
For CR and MedQ, report the average of the benchmarks they comprise.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: The performance of the fine-tuning task (math), measured by GSM8K, and the general capability
scores of Llama-2-7B after fine-tuning on the MetaMathQA dataset. The results show that MoFO achieves comparable performance in the fine-tuning task, while significantly mitigating forgetting of general capabilities. Bold values denote the best results among these methods.
General Capability
Method
GSM8K
CR
MMLU
HEval
Avg.
Llama-2-7B
13.7
65.6
42.0
24.2
43.9
Default FT
49.4
62.3
36.6
16.1
38.3
HFT
47.5
65.5
42.3
23.6
43.8
LoRA
43.3
65.1
37.7
26.4
43.1
MoFO
47.7
65.7
42.7
24.6
44.3
4.2 Instruction Fine-Tuning
In this section, we investigate the effectiveness of the MoFO algorithm in both preserving general capabilities and learning fine-tuning lasks. The implementation details are provided in Appendix The specific hyperparameter settings in each experiment are provided in Appendix C.3
LLM Fine-tuning strategy baselines. We compare MoFO with the default fine-tuning approach and other methods designed to mitigate forgetting. These baselines include: Default fine-tuning (Default FT) refers to the full-parameter fine-tuning approach using the Adam optimizer. Half Fine-tuning (HFT) 2024) randomly updates half of the parameter blocks within cach transformer layer at cach teeaaiin while the other half are frozen. HFT can be considered a specific case of the BCD algorithm. LoRA (Hu et al. 2022) is a widely-used, parameter-effiieet fine-tuning method. LoRA trains low-rank matrix adaptations on the base model's weights. Recent work (Biderman et al.) 2024) demonstrates that LoRA can
(Hui et al.
mitigate forgetting.
Results of fine-tuning MetaMathQA. We fine-tune Llama-2-7B MetaMathQA using various baseline methods and present the experimental results on mathematical reasoning (GSM8K) and general capabilities in Table[] We report the experimental results of LoRA under the best-performing hyperparameter configuration on the fine-tuning task. These results demonstrate the effectiveness of our proposed MoFO algorithm in both optimization and mitigating forgetting.
MoFO compatible to the performance of Default FT and HFT on the math task, yet significantly outperforms these methods preserving general capability. Specifically, Default FT shows decline of 5.4% in MMLU accuracy and HFT experiences drop of 0.6% in HumanEval. In contrast, our MoFO not only maintains but slightly improves these general capability scores by an average of 0.4%.
Comparison from a Pareto perspective. Generally, improving performance on the fine-tuning task and reducing forgetting are often a pair competing objectives. It is intriguing to study how different iine-tuning methods balance this tradeoff. By adjusting the hyperparameters of different methods, we can observe a set of fine-tuned models, cach representing a different tradcoff between fino-tuning performance and forgetting. The Pareto frontier formed by these models helps visualize the tradeoffs, and we can identify which method offers the best balance between fine-tuning and forgetting.
In this comparison, we also include traditional regularization methods such as Le-regularization [Li et al. 2018) (denoted as L2 reg) and Li-regularization (Panigrahi et al.) 2023) (denoted as L1 reg). which are not specifically designed for large models. These methods modify the original fine-tuning loss Liinetune (0) by adding regularization term. For I--rogularization, the modified loss fiinetune (0) + >2 0 0oll and for Lr-renularizatin, it is L Lfinetune (0) + X₂ 1 Boll1, where X2 and X are the respective regularization hyperparameters.
11

---

Published in Transactions on Machine Learning Research (10/2025)
We fine-tune the Llama-2-7B model on the MetaMathQA dataset using L1 and L2 regularization, well LoRA, and compare their performance with MoFO. We present the results in Figure and plot Pareto optimal fronte for these methods. Details of the hyperparameter configurations for this experiment are provided in Appendix C.3. These results show the effectiveness of the MoFO algorithm in both optimization and mitigating lorgetting.
0.50
0.50
0.45
0.45
0.40
0.40
Default FT
Default
GSM8K
0.35
HFT
GSMBK
0.35
HFT
0.30
L1 reg
0.30
L1 reg
reg
L2 reg
0.25
0.25
MOFO
MOFO
0,20
LoRA
0.20
LoRA
0.15
Llama-2-77
0.15
Llama-2-7B
0.63
0.64
0.65
0.66
0.30
0.32
0.34
0.36 0.38 0.40 0.42 0.44
CR
MMLU
Figure 5: The performance on the math task (GSM8K) and the scores in general capabilities of Llama-2-77 after fine-tuning on the MetaMathQA dataset. Only points on the Pareto front are shown solid points, while the remaining points presented semi- tramsaaansp.rent. The results show that compared with L1, L2 regularization, and LoRA across various hyperparameter configurations, the MoFO algorithm achieves a better Pareto front.
The result reveals that MoFO consistently achieves a better Pareto front in comparison to baseline methods. When compared to regularization methods and LoRA, MoFO exhibits less forgetting and can even maintain general capabilitics with comparable GSM8K accuracies. Additionally, MoFO outperforms regularization methods in math lasks when the magnitudes of forgetting are similar. We also note that Lı and L2 regularization (Panigrahi et al., 2023 Li et al. 2018 require storing the pre-training weights throughout the entire fine-tuning process for regularization computation, which incurs additional memory overhcad. For preliminary analysis on why MoFO might compare favorably to L regularization, see Appendix F.5
Appendix reports additional instruction tuning results. Specifically, we evaluate
LLM variants: Gomma-2B-IT and Alama-2-7B--n 2-7B--ha;;
Domain-specific datasels: medical dataset (PMC-LLaMA-Instruce (Wu al.
2024 1), coding dataset.
(Magicoder-Esve Instruntt (Wei al. 2023 ):
Different baselines: HMA [Lin et al. 2024a CoFiTune Zhang et al.
2024a Soft-masking
Ke
et al. 2023a b).
MoFO Converges Closer to the Pre-trained Model.
In this part, empirically investigate whether MoFO converges closer to the pre-trained model. Building the fine-tuned models in Table [ compare their distances to the pre-trained model. In addition, for the L1 and L2 regularization baselines in our Pareto analysis above, we select the models that achieve the best performance on the GSMSK benchmark (corresponding to the fine-tuning task).
Figure 0 shows that models fine-tuned with MoFO are closer to the pre-trained model compared to other baseline methods.
Sinnce it is impractical exhaust hyperparameter configurations real experiments, present linear interpolation approximations of the Pareto fronts Figure g
12

---

Published in Transactions on Machine Learning Research (10/2025)
0.014
0.012
Distance
0.008
0.006
0.004
0.000
Reg
Reg
HFT
Figure The distances for the fine-tuned Llama2-7B on MetaMathQA. The results show MoFO achieves minima closer to the pre-trained model.
4.3 Continual Fine- Tuning
In this scction, we explore the performance of our proposed MoFO in continual fine-tuning on the TRACE benchmark (Wang et al., 2023b). We sequentially train Tiny Slnmaaa---111 on the TRACE dataset, which includes the eight tasks from different domains. The implementation details are provided in Appendix C
Continual lcarning basclincs. We consider several traditional methods from the field of continual lcarning to compare with MoFO. These methods can also be orthogonal combined with MoFO to further enhance performance. Replay involves optimizing the model using current data along with memory buffer containing samples from previous tasks to mitigate forgetting, and we follow the implementation in Wang et al. (202b)). Gradient of Episodic Memory (GEM) (Lopez-Paz & Ranzato (017) mitigates forgetting by using gradients from old tasks to adjust the parameter updates during the training of new tasks. Elastic weight consolidation (EWC) (Kirkpatrick et al. 2017) uses a diagonal approximation of Fisher information matrix, which can be calculated by gradients from previous tasks, regularize parameter updates.
Table 2: The OP and BWT scores of TinyLLama-L.. after fine-tuning on TRACE benchmark. The results show that MoFO outperforms Default FT, HFT, GEM, and EWC in contimual learning and can combine well with continual learning methods. Bold values denote the best results among these methods in each group.
OP
BWT
Default FT
38.4
-10.3
HFT
39.9
-10.1
MoFO
41.3
-5.4
GEM
40.8
-8.5
GEM + MoFO
41.7
-6.7
EWC
41.1
-8.3
FWC + MoFO
43.2
-4.4
Replay
45.5
4.7
Replay MoPO
47.0
4.8
Results continual finc-tuning. We present the experimental results of soquentially fine-tuning LinLLlama-L1BB the TRACE benchmark with various methods in Table 2 The results indicate that in continual fine-tuning, MoFO not only outperforms other fine-tuning baselines but also surpasses GEM and EWC. Moreover, MoFO combines well with the Replay method, offering a 1.5% performance gain on the OP metric compared to using Replay alone. Moreover, MoFO works well in combination with EWC, yielding at least. a 2.1% improvement in the OP metric over using EWC alone. Additionally, when combined with the GEM method, MoFO provides a 0.9% improvement on the OP metric compared to using GEM alone.
In summary, these results underscore the superior performance of MoFO continual fine tuning and its effectiveness in alleviating forgetting.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: The performance on the math reasoning task (GSM8K) and general capability scores of Llama-2-7B after fine-tuning on MetaMathQA using different updating strategics in MoFO. Bold values denote the best results among the BCD methods.
General
Capability
Method
GSMSK
CR
MMLU
HEval
Avg.
Llama-2-7B
13.7
65.6
42.0
24.2
43.9
Default FT
49.4
62.3
36.6
16.1
38.3
Random BCD
35.0
65.8
41.1
25.1
44.0
Grad BCD
40.2
66.0
41.6
28.0
45.2
MV BCD
42.2
66.0
40.0
27.6
44.5
MoFO
45.4
65.7
43.5
27.4
45.5
4.4 Impact of Update Strategy in MoFO
In addition to MoFO, we consider three other BCD methods with different filtering strategics: random BCD, gradient-fitered BCD, and MV-filtered BCD. Random BCD updales a random subset of parameters each iteration. Gradient-filtered BCD replaces MoFO's filter FLTa(m)) with (gt), while MV-filtered BCD uses FLTa /ve).
We fine-tune Llama-2-11 and present the results in Table 3 Experimental results show that all four BCD methods exhibit significantly less forgetting compared to Default FT, demonstrating the effectiveness of BCD algorithms in mitigating forgetting.
MetaMathQQ using these four methods with 10% parameter update fraction
In terms GSM8K performance, our proposed MoFO method significantly surpasses Random BCD, Gradient- filtered BCD, and MV-filtered BCD, indicating that updating parameters with the largest momentum leads to strong optimization power. Additional comparative experiments BCD filtering strategies presented in Appendix More insights towards this result are provided in Appendix F.4
4.5 Furthur Analysis
Guidelines for setting a. Experiments show that setting the updating fraction a = 15% works the best for most of our experiments; and 5% - 15% all work quite well. We provide a more detailed guideline for determining a Appendix D The guideline involves randomly sampling small proxy subset and performing grid search over possible values.
Efficiency Analysis. We provide efficiency analysis on MoFO in Appendix F F.2 The results show that MoFO requires only around 4% - 5% additional training time compared with Default FT throughout the entire training process.
5
Related Works
5.1
Forgetting in Continual Learning
Calastrophic forgetting, a significant issue where models forget previously learned information upon learning
new data, has received considerable attention in machine learning (McCloskey & Cohen 1989 Goodfellow
ct al. 2013 Kemker al. 20188 Ramasesh et al., 2021 Verwimp et al. 2023 Liu al. 2024). Traditional
continual learning primarily focuses on addressing catastrophic forgetting in sequential-tass learning scenarios.
In addition to investigating the forgetting of pre-training knowledge during fine-tuning, Section conducts
14

---

Published in Transactions on Machine Learning Research (10/2025)
experimental studies on catastrophic forgetting in sequential-tase fine-tuning processes, which aligns more closely with conventional continual learning paradigms.
Replay-based methods. In sequential-task learning, these methods leverage past experiences to facilitate the learning of new tasks. The most classical scheme is experience replay, which involves replaying data
of past tasks during incremental training (Rolnick et al. 2019)
(Aljundi et al.] 2019a] Hayes et al. 2019
Cha et al., 2021 Chaudhuy al. 2019b Riemer et al. 2019bb.
Other variants utilize gradient information
from old tasks (Lopez-Paz & Ranzato 2017 Riemer et al. 2019a Chaudhry et al.]
2019a; Farajtabar et al.,
2020 Aljundi et al. 2019b Chaudhry et al., 2021 Tiwari et al.
2022). In LLMs, Yin et al. (2023): Wang
ct al.
2024b): : Ouyang et al. (2022) propose replay- baalay-basedaoo methods to mitigate forgetting. While MoFO is a
replay- free method, MoFO can be combined with replay strategies.
Regulacizaatio--aased methods. These methods introduce constraints to the training process to preserve past knowlodge, such as adding regularization to the loss functions (Kirkpatrick et al. 2017 Aljundi et al.,
2018 Zenke et al. 2017 Li et al., 2018| Ritter et al. 2018 Kumar et al. 2023) or the embedding/ oul
changes (Li & Hoiem 2017 Rannen et al. 2017 Buzzega et al. 2020 Huang et al. 2021 Cha et al. 2020
Some regularization--asse
approaches still rely on partial information from previous models (Kirkpatrick
ct al.
2017). In contrast, MoFO does not require past information and does not alter the original loss function, making it inherently orthogonal to regularization-basel methods. To improve generalization and robustness several studies introduce explicit regularization (Li & Zhang 2021 Zhang
to noise after dustruction-tuning
let al. 2023). In particular, Zhang et al. (2023) proposes a Hessian-based penalty that encourages convergence to flatter minima. It is an interesting direction for future research to evaluate whether MoFO's momentum filtering mechanism implicitly favors more stable minima, thereby further enhancing generalization and noise robustness.
Optimization-basee methods. These methods focus on modifying the training algorithm to mitigate forgetting. In traditional continual learning, optimization- based methods commonly include, but are not limited to, gradient projection techniques (Wang et al. 2023a Lopez-Pazz & Ranzato 2017], meta-learning approaches (Beaulieu et al., 2020 Javed & White, 2019), and stralegies leveraging the structure of the loss landscapes (Mirzadeh et al. 2020a|b) b). When comes to forgetting-nin miitiiitiimitiiiiinnnn in LLM training, recent studies have explored optimization strategics that update only subset of parameters at cach iteration. For instance, Hui el al. (2024) randomly freezes half of the model's parameter modules and updatets the rest at each iteration. Ke et al. (2023b a introduce a soft- -asking mechanism that selects parameters for update based on their importance values. Further, (Zhang et al. (2024a) combines selective module updating with soft-masking, MoFO, which also falls into this category, updates parameters with largest momentum magnitudes at each iteration. Compared to these works, our study provides a theoretical convergence guarantee of our proposed method, thereby establishing its effectiveness in LLM fine-tuning.
Model merging methods. These methods balance learning new knowledge and retaining old knowledge by merging the new and past models. One line of research focuses on model averaging, which interpolates
between the weights of different LLMs (Wortsman et al.
2022a b Eeckt et al. 2022 Yadav ct al. 2024 Lin
ett al., 2023 2024ah. Another line of research relies on the observation that l Sask-iei knowledge lege largely
resides in a subspace of the weight space (Ilharco et al.) 2023
Panigrahi et al.
2023
Gueta et al.] 2023
Zhu
ct al. 2024 He et al.] (024), and leverage task vectors or task localization to preserve pro-training knowlodge
in the line-tuned models (Panigrahi et al. 2023: Yaday et al. 2024 Yu et al.
2024a).
Architecture-based methods. These methods modify the model's architecture in training. LoRA Hu ct al. 2022). as the most popular parameter-cffiieet finc-tuning (PEFT) method, freczes the pre-training weights and introduces low-rank trainable matrices. Variants of LoRA are applied in continual learning for LLMs (Ren et al. 2024 Wang et al. 2023a). However, LoRA is observed to forget less but also learn less than default fine-tuning (Biderman et al. 2024). Apart from LoRA, Adapters (Houlsby et al.) 2019) and BitFit
Zaken al.] 2021) are also well-known PEFT methods. In Appendix E.4 we include them as baselines for comparison and find that, while they are slightly less effective than MoFO in mitigating forgetting, their performance on the finc-tuning task is substantially worse than that of MoFO.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Other approaches adaptively expand model capacity or isolate partial weights to mitigate interference between new and old tasks (Wang et al. 2023a: Razdaibiedinaa et al. 2023). In contrast, MoFO updates subset of parameters at each iteration, but does not alter the total trainable parameters.
5.2 Block Coordinate Descent
Block Coordinate Descent (BCD) involves iteratively optimizing over a block of coordinates while holding the others constant. The foundational work of Tseng (2001) provides a comprehensive analysis of the convergence properties of BCD under cerlain conditions. Subsequent research has explored various BCD variants (Hong ct al. 2017). including random BCD (Nesterov 2012, Richtárik & Takáč 2014 Lu Xiao 2015), cyclic BCD (Sun & Hong 2015; Razaviyayn et al., 2013), and greedy BCD (Nutini al. 2015). Among these, the greedy variant, also known Gauss- BCD method, has drawn attention due to its ability to prioritize coordinates that yield the most substantial improvement in each iteration, thereby potentially accelerating convergence.
In the realm of machine learning, BCD has also found applications NNutini et al.
2022 For example, Luo
et al. (2024) leverages BCD to perform memory-effciiett fine-tuning of LLM and Xu & Zhang (2024) uses
random masking to perform this. In foderated learning, Rothchild al. (2020) adopts top-k momentum value
unsketch rather than our lop-k momentum filtering to tackle communication bottleneck and convergence
issues. In LLMs, some concurrent works propose BCD-based algorithms leveraging task vectors to enhance ine-tuning performance (Li et al. 2024) and mitigate catastrophic forgetting in multi-task learning (Panda et al. 2024). Our approach can be regarded as a type of greedy BCD adapted to Adam, achieving good performance in fine-tuning tasks and alleviating forgetting.
6 Conclusion and Limitations
This paper presents the Momentuun-Filteree Optimizer (MoFO), a new approach designed to mitigate the crucial issue of pre-training knowledge forgetting in LLMs during fine-tuning, By selectively updating the parameters with the largest momentum magnitudes in each parameter block, MoFO converges to point closer to the pre-trained model compared to full-parameter fine-tuning and effectively preserves pre-trained knowledge. Our experimental results demonstrate that MoFO not only achieves comparable performance to default fine-tuning but also effectively alleviates lorgetting.
While this work provides a preliminary exploration of applying traditional block coordinate descent methods mitigate forgetting LLM training, several avenues remain open for further investigation. First, the current framework
uniform and consistent update fraction across all parameter blocks throughout training, whereas future work may explore adaptive update fractions and block-wise dynamic adjustments. Second, although our focus centers on forgetting mitigation during supervised fine-tuning, extending this methodology to downstream phases such as RLHF represents a promising direction for improving LLM development pipelines.
Acknowledgement
The authors would like express our sincere gratitude to the reviewers for their insightful feedback during the discussion phase. The authors also thank Congliang Chen and Ziniu Li for their helpful suggestions. This paper is supported by NSFC (No. 12326608 and No. 12401409); Hetao Shenzhen- -oong Kong Science and Technology Innovation Cooperation Zone Project (N..H7QQWS-KCCYB--22222 Fund UDF01001491, the Chinese University of Hong Kong, Shenzhen; Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001).
University Development
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytclaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the Europeon conference on computer vision (ECCV), pp. 139- 154, 2018.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019b.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff' Clune, and Nick Cheney. Learning to continually learn. In ECAI 2020, pp. 992- 192-1001. IOS Press, 2020.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. LoRA learns less and forgets less. arXiv preprint arXi:22405.099 09678, 2024.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on compuler vision, pp. 9516 9525, 2021.
Sungmin Cha, Hsiang Hsu, Taeback Hwang, Flavio P Calmon, and Taesup Moon. Cpr: classifier- poooffter-rroecctionn regularization for continual learning. arXie preprint arXi::2006.0P 073266 2020.
Arslan Chaudhry, Puncet K Dokanía, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Undorstanding forgetting and intransigence. In European Conference Computer Vision, pp. 556 572, 2018.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhosciny, Thalaiynsingam Ajanthan, Puncet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning, ar preprint 1XXix:1902.14486 2019b.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6993-7001, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. XiXiv preprint ar arXio:2107.0339 0333774 2021.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
2020.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint 1803 05, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. Xrxiv preprint or Xin:21 10. 14168, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forby-firet International Conference on Machine Learning, 2024.
Andrew M Dai and Quoc V Lc. Semi-supervised sequence learning. Advances in newral information processing syslems, 28, 2015.
Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness? Advances in Neural Information Processing Systems, 34:4356 4369, 2021.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
Steven Vander Eeckt et al. Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition. arXiv preprint arXiv:2210.52282 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Aug Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773. PMLR, 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint vrXfc:2011.00027 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023. URL https://zenodo org/recoords/1025633
Ian J Goodfellow, Mchdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical invostigation of catastrophic forgetting in gradient-based neural networks. arXiv preprinl arXiv:1312.622 62111 2013.
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is a region in weight space for fine-tuned language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 1350 1370, 2023.
Tyler Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference Robotics and Automation (ICRA), pp. 9769 9776. IEFE, 2019.
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stitchh: Efficient model merging via sparse lask arithmetic. arXio preprint arx: 13656, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. In International Conference on Learning Represenlations, 2021.
Mingyi Hong, Xiangfeng Wang, Meisam Razaviyayn, and Zhi-Quan Luo. Iteration complexity analysis of block coordinate descent mothods. Mathematical Programming, 163:85-114, 2017.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficier transfer learning for nlp. In International conference on machine learning, pp. 2790-2790 PMLR, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.00 01244, 2024.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapler of the Association for Compulational Linquisties: Human Language Technologies, pp. 2736-2 2746, 2021.
Tingleng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, and Hua Wu. Hf:: Half fine-tuning for large language models. Xrxiv preprint arXiv:240,.18466, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt,, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR). International Conference on Learning Representations, 2023.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaplation with tulu 2. ar Xiv preprint arXiv:2311.1077 2023.
Khurram Javed and Martha White. Meta-learning representations for continual learning, Advances in newral informalion processing systems, 32, 2019.
Di Jin, Eileen Pan, Nassim Oufatvole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 7567,2577, 2019.
Zixuan Ke, Bing Liu, Wenhan Xiong, Asli Celikyilmaz, and Haoran Li. Sub-network discovery and soft- masking for continual learning of mixed tasks. In Findings of the Associalion for Computational Linguisties: EMNLP 2023, pp. 15090 15107, 2023a.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In International Conference on Learning Representations (ICLR). International Conference on Learning Representations, 2023b.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of AA pp. 4171-4186, 2019.
Diederik P Kingma and Jimmy Ba. 1xXx:1112.66800 2014.
Adam: A method for stochastic optimization. arXiv preprinl
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka, Grabska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017.
Tomasz Korbak, Hady Elsahar, German Kruszewski, and Mare Dymetman. Controlling conditional language models without catastrophic forgetting. In International Conference on Machine Learning, pp. 11499 11528. PMLR, 2022.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regular- ination. arXio preprint ar Xiv:2308. 1 11958, 2023.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Aduances in Newral Information Processing Systems, 34:27249 24:2224-277262, 2021.
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Yujiu Yang, Qi Chen, and Peng Cheng. Gradient-mask tuning elevates the upper limits of Ilm performance. arXiv preprint arXiv:2406.1550 15330, 2024.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pp. 2825 2834, PMLR, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2944 2017.
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprinl arXie:2309. 2ef::2399000250, 2023.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hamning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirioal Methods in Natural Language Processing, pp. 580 6006 2024a.
Zhanran Lin, Puheng Li, and Lei Wu. Exploring neural network landscapes: Star-shaped and geodesic connectivity. arXiv preprint orXiv:244.06390 2024b.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameleerieed non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85 59:85-116, 116, 2022.
Chengyuan Liu, Shihang Wang, Yangyang Kang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, and Fei Wu. More than catastrophic forgetting: Integrating general capabilities for domain-specific 1lms. aixiv preprint arXi::245.. 1830, 2024.
David Lopez-Paz and Marc' Aurclio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Zhaosong Lu and Lin Xiao. On the complexity analysis of randomized block-coordinate descent methods. Mathematical Programming, 152:615-642 642, 2015.
Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter training method for large language models. arXiv preprint arXiv:2404.028: 2024.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgelting in large language models during continual fine-tuning. arXiv preprinl arxi2:2308. 08747, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingleng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023b.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXie preprint ar arXiv:2010.04400 2020a.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33: 7308 7320, 2020b.
Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 2212).341-36,, 2012.
Julie Nutini, Mark Schmidt, Issam Laradji, Michael Friedlander, and Hoyt Koepke. Coordinate descent. converges faster with the gauss-southwell rule than random selection. In International Conference on Machine Learning, pp. 1632-1641. PMTRR 2015.
20

---

Published in Transactions on Machine Learning Research (10/2025)
Julie Nutini, Issam Laradji, and Mark Schmidt. Let's make block coordinate descent converge faster: faster greedy rules, gaasagg-passing active-set complexity, and superlinear convergence. Journal of Machine Learning Research, (3(11)): 74, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248 260. PMIRR, 2022.
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406.1677 16797, 2024.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp. 77011-27033. PMLR, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Nlya Sutskever. Improving language understanding with unsupervised learning. 2018.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proceedings of the IEEE international conference on computer vision, pp. 1320-1328, 2017.
Meisam Razaviyayn, Mingyi Hong, and Zhi-Quan Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization, 23(2):1126 1153, 2013.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2801122314 2023.
Weijieying Ren, Xinlong Li, Lei Wang, Tianxiang Zhao, and Wei Qin. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. arXiv preprint 222 2024.
Peter Richtárik and Martin Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing composite function. Mathematical Programméng, 144(1):1 38, 2014.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Trina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Represenlations, 2019a.
Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele Franceschini. Scalable recollections for continual lifelong learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 1352-1359, 2019b.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillierap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, lon Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Commiicattion-dffiientt federated learning with sketching. In International Conference on Machine Learning, pp. 8253 8265. PMLR, 2020.
Baptiste Roziere, Jonas Gchring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint ar Xiv:2830 12950, 2023.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, and Hao Wang. Continual learning of large language models: A comprehensive survey. arXiv preprint (11::24041667888 2024.
Naichen Shi, Dawei Lı, Mingyi Hong, and Ruoyu Sun. Rmssrrp converges with proper hyper-parameter. Tn 9th International Conference on Learning Representations, ICLR 2021, 2021.
Ruoyu Sun and Mingyi Hong. Improved itcration complexity bounds of cyclic block coordinate descent for convex problems. Advances in Neural Informalion Processing Systems, 28, 2015.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of- thought can solve them. Xiv preprint arXiw:2210.09200 2022.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0880 08299, 2024.
Tijmen Tieleman. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Newral networks for machine learning, 4(2):26, 2012.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2022.
Together. Redpajama, a project to create leading open-source models, starts by reproducing Ilama training dataset of over 1.2 trillion tokens. https://www. att.://cc..r.aabboooorrppppaaaaa 2023. Accessed: 2025-08- 14.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint urXiv:2307.00928, 2023.
Paul Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications, 109:475-4944 2001.
Eli Verwimp, Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, et al. Continual learning; Applications and the road forward. arXiv preprint arXiv:2311. 11908, 2023.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: theory, method and application. IEFF Transactions on Pollern Analysis and Machine Intelligence, 2024a.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In The 2029 Conference on Empirical Methods in Natural Language Processing, 2023a.
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning large language models. arXiv preprint arXiv:2310.006 06762, 2023b.
Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Inscl: A data-efficient continual learning paradigan for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 663-677, 2024b.
Zirui Wang, Sanket Vaibhav Mchta, Barnabás Poczós, and Jaime G Carbonell. Efficient meta lifelong-learning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 535 548, 2020.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NewrIPS Datasets and Benchmarks Track, 2024.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. arXiv preprint N.iu:2312.02112. 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-oppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference PMLR, 2022a.
machine learning, pp.
Mitchell Wortsman, Gabriel Ilbarco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roeloss, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVI conference on computer vision and pottern recognition, pp. 7959 7971, 2022b.
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pme-llama: toward building open-source language models for medicine. Journal of the American Medical Informalics Associalion, pp. ocae045, 2024.
Nachuan Xiao, Xiaoyin Hu, Xin Liu, and Kim- Chuan Toh. Adam-family methods for nonsmooth optimization with convergence guarantees. Journal of Machine Learning Research, 25(48):1-53, 2024.
Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine-tuning. Xrviv preprint Xiv: 02596, 2024.
Prateck Yadav, Derck Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Newral Informalion Processing Systems, 36, 2024.
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur: A dynamic growth paradigm for Instruction-tuiing data curation. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 4031-4047, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilitics from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024a.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024b.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-bant based masked language-modee models. arXiv preprint 3rXiv:2106.10111 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Avnual Meeling of the Association for Compulational Linguistics, pp. 4791-4800, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Tn International conference on machine learning, pp. 3987 3995. PMLR, 2017.
Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. Balancing speciality and versatility: a coarse to fine framework for supervised fino-tuning large language model. arXiv preprint arXiv:240,.10806 2024a.
Hongyang Zhang, Dongyue Li, and Haotian Ju. Noise stability optimization for finding flat minima: A based regularization approach. XrXio preprint arXi::230600857 2023.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllmaa: An open-source small language model. Xixiv preprinl arXii:2401.002 012055 2024b.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi- Qui-Quan Luo. Why transformers need adam: A hessian perspective. arXiv preprint arxio:2402. 16788, 2024c.
Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint vrXiv:2406.167933 2024d.
Tuo Zhao, Mo Yu, Yiming Wang, Raman Arora, and Han Liu. Accelerated mini-batch randomized block coordinate descent method. Advances in neural information processing systems, 27, 2014.
Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. arXiv preprint ar arXiv:240..1204 2024.
24

---

Published in Transactions on Machine Learning Research (10/2025)
A Theoretical Analysis
Appendix A is organized into three self-contained parts. A roadmap is provided to guide the reader in locating and understanding each theoretical result efficiently.
Roadmap to Appendix (Quick Reference).
Appendix Supplementary Analysis on the Top-a Filter
This section formalizes the top-a: filter (.) and the induced quantity I|= @ az)l that will serve our working norm in the analysis. Proposition I verifies that this is indeed norm. Lemma ! establishes a Lipschitz stability property for filtered outputs, which later lets us pass from the momentum-defined filtering to the gradient-define filtering. Lemma 2 relates the Litopp- norm to standard Lp norms, which is used at the very end to translate the convergence bound to any p E [1, oo].
A Appendix A.2 Proof of Theorem (Convergence of MoFO)
For high-level narrative, see the proof sketch accompanying Theorem 1 in the main text; here provide quick-reference map to the technical steps.
Lemma 3 bounds each step size and the fz-movement of parameters using the number of active coordinates. Lemma controls the drift of gradients across iterations via L-smoothness. Lemmaa 5 lower-bounds the per-coordinate alignment term /..mt////.. which is the driver the descent incquality. Lemma 6 shows that the bias-corrected momentum my tracks de in & at rate O(t-1/2) Combining those with the descent. lemma yields a key inequality in which the filter is (m)):
C2 t G Il @ (mu)ll S -(0--1)--11
Finally, Lemma 1 help convert to the desired incquality with the gradient filter (LTx summing over gives minosesT-i (V.) (VL(((()) T/vT). By Lemma this implies the same OlogT//VT) rate for any Tpp norm minggegT-1 [VL(()))
Appendix A.3: Proof of Theorem 2 (Illustrative example: forgetting mitigation of MoFO)
In illustrative example with updating ratio 1/d, MoFO converges to single-attractor minimum and is strictly closcr to the pre-training state than Adam, thus attaining lower pre-training loss. We prove this by mathematical induction.
Appendix A.4 Challenges and Potential Extensions of Theorem 1 to Nonsmooth Objectives
This subscction explains why extending the convergence analysis of Theorem nonsmooth objectives is nontrivial and outlines several possible future directions.
25

---

Published in Transactions on Machine Learning Research (10/2025)
A.1 Supplementary Analysis on the Top-a Filter
In this section, we provide supplementary analysis on our top-a filter, which serves as preliminary for proving Theorem ! in Appendix A.2
As introduced in Scction 2.2. the entire parameter space divided into B parts, with the k-th part having a dimension of dx. We assume the parameler space is IR,, which can be expressed as the product nd z pd x I2
Rdo, For any E Rd, we represent as:
= Concat(2(), x((1)
(B))
where 2k) E Rd for each S k $ B.
Definition For any z E RR, we define the top-a filter of z as
(Z) = Concat(e, ( S e ; : E Rd
(2)
(B))
where
Sy = [dx]
1 ranks within the top-a of all (x)) 's entries (12) 12122
(2) 1..
(8) 1}}
and es2 a dy di-dimmesiooal vector where the i-th entry is 1 if E Sk, and 0 otherwise.
(k)
Remark 2. To ensure that the lop-a filter (z),(z) is well-de well-defined, when multiple entries share identical absolute values and including all of them in the set Sk would result in exceeding the a threshold of set size, the construction of Sk prioritizes the entries with the smallest indices among those with the same absolute values. Definition 2. For any Rd,, define the L Lopoo- norm of
1121,,,00-- == I| 2 (2)l(
Proposition 1. Il-la,top-a indeed norm in Rdd.
Proof. By Definition we get
|| = ||= @. FLTx(
Σ
| o es
(k)
(6)
First, ||:|11,op-a = 0, then by 66, (k) 3 e ||| = for any 1 S B. Thus,
|| (k) 1000 arg max (1)
(0
(k) es 1|1 0.
13idde
So (k) is a zero vector for any 1 1 k and then z is a zero vector.
Second, for any given ee R4, f1xg" (1) hsiga, and floz (k) Isiid have the same order. So and ez share the same filter (z) and
||eI|l,top- = ||ez O FLTa d|l FLTa c||=||1,
Third, for any x, E Rd, we let
S {i E [dx] | (1) ranks within the top-a of all [xc(*) entries (lx((*) 1,, (...,
..... 11 (4)
(),
5% = {i E [d+] |n: + 1 ranks within the top-a of all
(k) + (s) entries (aa (k) + (k)
+ y/2 ,...,, lor Vdd 1)}-
()
(8)
Then we have
FLT, (x) = Concat(e ( S, :...e
(1) (2). B and FLT, + y) = Concat
go gy: egg
26

---

Published in Transactions on Machine Learning Research (10/2025)
By the construction of 1 for any 1 k I B, have
| 1kx(0)
(1)
S
1|x
o oes
(8)
||
So
FLT f)lb Σ
|
FLT, (x)]l
5
Similarly, it holds that
yy 2 (L + b)1l S ly o FLT.
Thus, we have
1 + 3/l1,top--
[1(x + +) @ (L y)1l
|| FLT. ( y FLT, (t +)
< | O FLT, 1lve (z)
s I o FLT ll O FLT (1)ll
= || + |lyl|i,
We propose a lemma which is useful for the proof of Theorem ! Lemma 1. For any y, Rd, it holds that
I - | FLTa(V))l s 2|1
ylla-
Proof. By Proposition
||-|| I,to-- norm in Rd, so we have
I O (x) | @ (L)l())
x FLTa()))
1ly (1)l + bl @ FLTa())ll - || (L))
|
1ly FLT, || (1)l(
< || 3/11,top-a + +1 o FLT. (v)ll1
< || y/12 + +1yy-111
2||x y/1 |-
The lemma below quantifies the relationship between Livop- and Lp norms for p E +00).
Lemma 2. Assume the parometer space Rd is decomposed into B blocks Rd ... x Rdd and z
Concat(z(1), ,..., (z())..., BB) with 2(h) E RRd. Then for any z E Rdd and any p [1, oo), it holds that
a||=1,, S ||=l1, (da + B)2- 1|21|pp
Proof. We first prove the p case. Fix any block k. Write the absolute values in nonincreasing order (k) ... 2 a 2 0, where a; (k) = |. Let my [2)( by the definition of the top-a filter (Definition and the tie- bie-breaking rule in Remark my [. Since the average of the top my mumbers is at least the overall average, we have
|2 (k)
11
2
dp
(ESe(=)
Summing over yields ||2||,,0op-< 2 a Σx (k)
|| a||2|11 The upper bound ||2||1,1--- $ |2 immediate
since (2), is (0, 1) filter.
27

---

Published in Transactions on Machine Learning Research (10/2025)
Next, let p E (1, (1,+ The lower bound follows from
1 2 a|||||| 2 a||=||p-
since ||=||= 2 |==1b= For the upper bound, fix & and apply Hölder's inequality on the subset Sx(2):
de
Σ
Σ
18)
1/9
1)
1/8
=m
m 1- 12/01//1))
=1
(=)
(=)
where q = p-1
_ and we used 1/4 1/p. Summing over and applying Hölder again to the finite sum
Σk m 1111/p11(00) gives
1
1/D
|| 1.tpa
Σ
m
11
S
m4
11 [b]
(=
x=1
R=1
We set m = Ex= my-
Since
Ex [1 = ||=1B (blocks are disjoint) and m = Ex my = Ex(ada s
ad B, we obtain
||2||1.oo-- S m 1 ||21|p v (ad + B)¹ ->
For p = +oo the same argument applies with q 1 and yields Eves((a) 12 S mx mx|l2(")|loo hence ||=||2,
m|| the lower bound ||=||1,top-a 2 a||=||>o follows from || 2 || This completes the proof.
28

---

Published in Transactions on Machine Learning Research (10/2025)
A.2 Proof of Theorem 1 (Convergence of MoFO)
Notation recap (Appendix A.2).
= V[(0 E Fd the full-batch gradient at step t.
Step size: 1/t = n/ Vt; hyperparameters satisty &1 vB2 <
First/second moments: my, U Rdd with updates
(maye v(.e) ((1 31)gt, + f1mit-1, (1 32)9 + :201,0-1);;
m bias-corrected first/sccond moments: m = m2m/111 (1 -1), 0₃ = 01/(1 (3))).
MoFO's filter F-),,-) E {0, 1) in each partition it E [B] of size dg (with Σn d2 d), we keep [dga] entries and zero out the others, We write ||2 FLT, (z)ll1 for the f--nrrm of the kept coordinates and 1|opa - ||x @ FLT
MoFO's update: 0,, - 01,--1 = - - ma,a/ V De, if (L)) 1, and otherwise.
Our proof of Theorem | follows the convergence analysis of the full-batch Adam optimizer in Shi et al. 2021
with novel adaplations to address the unique aspects of MoFO.
To maintain consistency with the notation used in MoFO (Algorithm in Section 2.2
we denote
2 = Concat (1) 24 (
where represents the model parameter 0, the gradient the first moment estimate m, or the second moment
estimate v. Notably, each of these variables belongs to Rd. Thus, for any 1 S S d, we can denote ži, as the i-th entry of 2, when a represents 0, 9: m, or t.
By the update rules of the first and second moment estimates, we have
mi, = 81)gr, + B1m4,t-1, m2,0 = O,
V1 = (1 - B2)9 + B20 24,0 = 0.
So by mathematical induction, for any 1 S i S d, we have
Ma,t (1 31)
81- Di.*
(7)
$=1
and
Ui.t (1 (2)
8/ 8/-+ gi.
(8)
We will frequently Equation
7 and 8 in the proofs of the subsequent lemmas and theorems.
Lemma 3. For the full-batch version of MoFO with hyperporameters satisfying 3r < D c= holds that
1
10 01.-1l5 v1 B2(1 - 31/(3,) Th (LTT(mm)) for any coortinate 15isd
Moreover, il holds lhat.
110 00-1112 S CT..
where C
B
(1-
29

---

Published in Transactions on Machine Learning Research (10/2025)
Proof. When the i-th entry is not in our filter at iteration t, ic. FLTa(mi); 0, we have 8i 02,1-1. Then
1
0 0i,--11 = 0= 32(1 31/VB3) 1) FLTa (m:);
When the i-th entry is in our filter, i.. FmTa(mm); =1, by the weight updating rule of MoFO, have 0s. 02,1-1 = - Voa.- We first analyze m441 and vi.t-
By Equation
and 8 the first/sen brrs//sccond moments mie, Uia satisfy
mm,el (1--31)
31 199,sh
v = (1- 32)
Vi
- Fi. 2 (1 (1-1111) 32) sh.
for any 1 1s s S t.
So we get
mi.t
/ B32
10,, 01,1-11
-ttt
T]
/m,,e// Voi,t
1117
1 Bi
31
(1
31) - gi.s
1 - B1
3
S
= 1)6
(B / (B)²
1 Bi
1 3
82
(1
B2) 1ginl
1-1
1/1
112///2)
<
VI 32
T/
V1 - 3a - B1
Here, the last inequality holds because of the assumption 31 < VB2 < 1.
The parameter vector is partitioned into B blocks with sizes {dx 1 and MoFO actually choose [dea] entries to update in each part of parameters. Then for any 2 E RRd, we have
# #1siis d: FLT, 1} Σ [daa] S
(dea 1) da + B.
k=1
k=1
Then for the Lorm of the parameter update, we have
10: 0-11/12
1 01,-112- 2. FLT, (m+);
1=1
: (()===1)) 11) 1
n
(VI 32 11 (1/VD2))
da + B
VI 32(1 - -31//32) Th
Ctp-
Lemma 4. Suppose that the gradient VL is Lipschitz continuous with constant L. Suppose that the full-batch version of MoFO has the hyperparameters satisfying 31 < VB2 1, e and the learning rate schedule "h = Vt. For any iteration steps 2 and any coordinate it holds that
2 2v2LCn(t - s)
1gtt g. S l1.. goll2 <
Vt
30

---

Published in Transactions on Machine Learning Research (10/2025)
where C
B V1 V1-BA(L-BL/VAA)
Fix the iterations steps and with t 2 8 2 1. Since VL has Lipschitz constant L, the gradient difference between the step and 8 satisfies
Proof.
19t. gi,s S li - 9ull2 [V((0-1) V - L1 - 02-1112.
(9)
By Lemma 3 for any >> 8 2 1, the parameter difference satisfies
110-1- - -0---00-11 12-1112 5
10 0u-1112 S CC) Ju
u=s
$ Cn Σ
S Cn Σ
22n E(vi Vu-1) 1)
Vu
Vu I+ Vu
2C'n(t s)
= 2Cn(vi 1 V8
Vt 1 vs
I 2Cn(t- - ()
2Cn(t - 8)
VI-I &
11/2
2v20m(t -8) 8)
Vt
When t == 8 > 1, it is obvious that
10:-1 - 0,-1/12= =05 2/2Cn(ttSh -))
v1
Combining it with (9), for any 6 8 2 1, we have
2v/LC1n(t s)
19t. Dr.al S llge 90l12 S
vi
Lemma 5. Under the assumptions in Lemma 4 for any iteration step t 2 and any coordinate i, it holds that
Ri.t m4,1
2v/231
LC
2 VI
B2 19t.l
0
(1- B1)
1 B2
V
Proof. By Lemma we get
2//2CTm( - -8)
Di. = Sh, giee Di.z) 2 2h.0 - 19;.l Di.
Di.zl
2 oh
ggrel-
Vt
Then for the product of gradient and momentum, we have
95 = =1 (1))) Σ
8
gi.ths,
==1
2 2V2LC1
2
2 Re (1- - 31)
3
l9h.d (1- 31) Σ
gt-8
(t (-s) s)
(10)
Vt
==1
8=1
1-1
2/2LCT
t-1
2 2. (1 - 31)
Σ
31
lgiell - (1 - (1))) Σ 3.
V1
==0
8=1
31

---

Published in Transactions on Machine Learning Research (10/2025)
Since we have
1- 39
d
d
81
Σ Σ
Σ
831 5
801
Σ
Bi
(11)
1 31
d3
d31
1 B1
(1
31)3
==0
it holds that
2v231LC1
Mi 2 RHS of (10)
2 (1 -1)0
(1 (1-Bi)vE l9..ll-
(12)
For the second moment Vi,i we have
Ui, = (1 (2)
32 F1.
S
Σ
3/ 3 (1ohrd + +196s - gi.l)2
2/21.11(1 - .1) 2
1-1
2/21.C1 2
S (1 - (2) Σ 32-
[gitl +
= =(1- - B2)
Σ
$2
|onel +
Vt
vt
1=1
==0
(13)
[-]
4V2LCM (1 - (32)
1-1
= =Iantl-- (1 - (2))
Σ
32
+ +19h,o
Σ
352
Vt
8==0
6=1
8
(1 -32
Σ
B2
1
1m/1
Since we have
1-1
1- 3
1
Σ
1 B2
1 - 32
1-1 Σ
8
d
d
18-1
<
852
d3
Σ 32 =
d3g
1 1- Bz
=
(1 - -3)221
==0
1=0
5=0
[-1
3² g----1
D2
<
8 P2
2g
= 32
Σ
8(8
1)35 2
8 Qx-1
8=0
==0
8==
8=0
:2
d
Σ
31
83
d
1
+
d33
32
d
1 8g
-1- k2)
+=0
2,3
+ (1- - B2)
(1-3,)3
+ B2
(1 83)3
it holds that
4V2B2LC
8(1 + C2
U4. < RHS of 13
1 + +1r.dl
(1 B2)
(1- B2)
8LCn
16
19 + los.el
(1-32) B2)
(1- -t)tt
(Iacel + (1 -1- 32)
4LC1
2
Thus, get
4LCn
Ver S 19r,e + (1-32)v]
32

---

Published in Transactions on Machine Learning Research (10/2025)
Recalling 12
we have
mi 2 (1- |ssisl + (1 - (1--)v)))
4LC1
(1ond) (1 -1- 81)(1 - -B1)vi (1 (1- B2)v1
2V23,LCm 4LCn
ALCn
2v221LCC
4LCn
+ (1 11-88 (8)) - -1-2)v
(1- --0 -Bive (1- (1-12)
2 (1 - (4) [grial + (1 (1-32)v (1=33)/1) (l9cal (1 -1- (1--3)((1-)) - -BIVE
4LC11)
2v22BLLCn
4LC1
(1 32)
2/2
4LCn
2 (1 - ViV (gial
(1-31)(1- (11 Bi)Vi (1-32)Vi
Therefore, for the bias-corrected first/second moments mixe and ús it holds that
mi,
v1 - 12
Mi,t
2 /1 33
|gr.el
2V23,LCM
4LC1
9i
Digt
1 31
9i,t
Ui
(1 (1) Bi)vi
(1
11-33)V1
2v231
LOn
2 v 32
Igr.el
(1 - B1)2 1 $2 vi
Lemma 6. Under the assumptions in Lemma 4 for any iteration step t 2 and any coordinate i, it holds
that
2V221VaLCy
| gell1 <
(1 (1-3))2v
Proof. Recalling the calculation of the momentum m, in
we get
my = (1 31)
9s:
1=1
and
my (1- (1 B1) Σ
8 "(1 -g)).
By Lemma 4
and Equation
11 in the proof of Lemma get
||m
gell2
m
9t
1 - 31
3 8 -sll2 S Σ 8 f1 gs
B
81
2 2/2LC
$t-(t 5) 2 21
1-1
3.9
Vi
Vt
2v23;LCM
< (1 P1
By Schhar-s inequality, we have
2v2331 VoBIvv
||m, relli S Vallmn. 9olla 5
(1-31)21
33

---

Published in Transactions on Machine Learning Research (10/2025)
Now we will complete the proof of Theorem
Proof of Theorem 7 By the descent lemma, since VL is Lipschitz with constant L, we have
L
((0,) [(0(-1) S VC(00-1) (T00--))((00 (0, - 02-1) 2 2110: - 0z-1 12
(14)
S g (0. - -0-1) + 10: 01-1122
By Lemma 3
and Lemma
have
mit FLTa(mi)i
LC
((0.) ((01-1) I RHS of 114
1pp Σ
9i,t
2
1
11
Σ
v 32 |.9.al
2//211
4
LOn
FLT (m4);
2t
Vi
(1- - 31)2
1
32
Vi
(15)
VI
32
2V221 VI 32
C
LC12
lgg. O FLT, (m+)ll +
Vi
(1 B1)²
v1
32
2
32 17
2v2 vI B2
C
LC B)
lg.. FLT.
Vt
(1 (1-81) 31)2
VI 32
1.
By Lemma ! and Lemma 6 we have
119. @ FLTa - l19. @ (m))ll
= 119. @ FLTa
9r
② FLT
m
1 31
= 119t O FLT (1)))l
2 meel2
4V232v VaLCn
(1 - B2)2
Thus,
((0,)) C(0,-1) S RHS of 15
< -I-Ba .2 "119.
T2V281 VI 1- B2
4
C
LCn B)
FLT. (g:
Vt
(1 11-31)²
V1
82
t
4v23, VdLCn?
(16)
+
(1 1t
C1 19 1,tppa
C2
< C
C2
+
min ||gh 10tll +
Vi
t
V
where
C = v1 -I-12 -32.11
{
12 [2228VT
4
C
(da + B) + 4/281 Vd
C2 = ICT?
+
+
(1- (1-31)2
VT- 32
2
(1 - (1-3))3
Taking the summation of 15 from to T, we get
L. C(0p) S [((T) - L((0)
Σ [(O.) L(0--1)
T
T + C2
< -
Σ
min
l1.. 3 FLT.
Σ
34

---

Published in Transactions on Machine Learning Research (10/2025)
Since
2
2 2(v++1 Vt) -
V
+t++
Σ
=1+
7
T-1
11+
T-1
du = 1+ S du 1 + log
1+1
"
1=1
t=1
1=1
we get
min 0SKST-1
[VC(0)]]
min 1S1ST
| 1gell1.top- = min 15157
1 2 FLT. (g:)
[(0o) - - + C2 C2 7
Ci C 4i
((0)) f- C2((1 + log 20
Thus, we have
log 7 VT
min OSTET
IStTT min [[((0) o (CT.
By the relationship between Litop-a norm and Lp norm in Lemma 2 for any p [1, +00,,
min 0515T-1
[[((0))
min 0S1ST-1
Therefore,
log 7'
min 05:ST-1
VL()))) = O
VT
35

---

b. 0 b. b. ai ai Published in Transactions on Machine Learning Research (10/2025)
A.3 Proof of Theorem 2
(lllustrative example: forgetting mitigation of MoFO)
Proof of Theorem
Let the set U :- (0 E R 0, < bi/ai, V1 iisiin dj. We note that:
1. The boundary of U is the subset of S U S which is the collection all global minima of the fine-tuning loss.
The pre-training state ppretrain (0, 0, o)) which is also the starting point of fine-tuning, lies in U.
We may as well assume that with proper learning rates, the parameter remains within U during training, unless it converges minimum on the boundary. goes across the boundary at certain iteration before converging, the learning rate can be adjusted to ensure that it remains within U. For any E U and coordinate i (1, d], we have
Oc 00,
2C(0)
2a, b,) [a - b1)² 0,
<0. 0. h
(17)
jfi
04
For clarity of definition, we let lr, represent the parameter at iteration t. We let Oies It,t, m4, denote the i-th coordinate of Oz, St. my at iteration t, respectively.
Analysis of MoFO. We will use mathematical induction to show the following results: If MoFO selects the coordinate in at the first iteration step, it will always select in at any iteration Moreover, for any coordinate A io, we have
Mio, S M S 0, and mio < mi,t SO O for any iteration step 2 2.
big Oip.t hi Oitt for any ileration step 1 1 if the algoritlem has not reached the minimeum.
Base case (1st iteration step). At the first iteration step, the momentum m. = (1 B1) So for any 1 i < d, get
dc
1, (1 -1)g1. (1)
0.
00,
ppeetratn
According to the momentum filtering mechanism of MoFO, have
ac
b
in E arg max |ma.el = arg max Igi.l = arg max Isisd
= arg max
arg min 1sidd
1sidd
1sisd
00,
Isidd
6 pretrain
a,
Obviously, get mi.11 I m4,1 S O for any coordinate io-
The parameter updates at the first iteration are:
n/ PaVI- B2m -
719.00
( [C (8 (0pretrai)) 00,
0..1 = 0%0,0
0.0.0 l9iol
000,0 siiig
> 0:0.0:
(1 BB.) Bi)/uro.t
8.1 02
Vi * in-
If the algorithm has not converged at the first iteration, then we have 0:g.1 < big/ Pto/ Moreover, for any Oi.rettrain
# io,
b a a
b. 0.1
Piopprrtrain
ai
Induction step. Suppose that the induction hypothesis holds up iteration t. Then, for any coordinate * io, have
Migt mi,t 0.
36

---

Published in Transactions on Machine Learning Research (10/2025)
big aio
Oig as
by O2
So for the gradient,
2C(())
2[((:)
gin
0ig.t b:
0 0ig a,
b:
90,1+1 < O,
and
Mi,,++1
Simooo (1 B1)gto,e+1
< B1m (1 B1)gi,(+1 M4..++1 0.
Thus, to is the only coordinate in arg maxisisd m4,1+1| and MoFO still chooses the coordinate in to update. In addition,
0:0,441 Oigst 7 0i,t:
Voret11
01,411 0s,t;
Vi # io-
If the algorithm has not converged at iteration step t + 1, then we have 0io.441 < /aio. Morcover, for any o in,
< b: a;
b,
b 0,t
b
0,,t-1
a,
Conclusion. MoFO consistently updates 0i and eventually converges to MMOO = ......,,, Osp
big
0, .,....
with pre-training loss
62
preetrain (MMO)O) =
2a
Analysis of Adam. Unlike MoFO, Adam updates all the parameters. By Inequality By the momentum update rule of Adam:
17
we have Is.e < 0.
Mi,411 Bime, + (1 - B1)gs,t11
get that mi for any 1 < SisA and any iteration t. Therefore, it holds for Adam that
McC(O.)
0 01, 0z. b.
Assuming that Adam converges to Adams have
Adam. for any 5 $ d,
There exists jo such that OG /jj-
Recall that at iteration 1, MoFO selects
bc
in arg min
1sidd
0s
Thus, the pre-training loss for Adam is
82
82
62
Spretrain (ddamm)
2a
0
Adam,
>
2a
20
2
2a
ig
Spretrain ( (M)OF))-
$4
In other words,
10 - pretraiill 2fpretrain (OMOFO) 2f (0) (0)am))
Opeetrain 112
In conclusion, MoFO converges to minimum closer to the pro-training state than Adam, preserving lower pre-training loss.
37

---

Published in Transactions on Machine Learning Research (10/2025)
A.4 Challenges and Potential Extensions of Theorem 1 to Nonsmooth Objectives
This subsection outlines challenges and possible future directions for exlending our convergence analysis to nonsmooth objectives; a complete extension is left for future work.
Why the extension is challenging.
Our Theorem 1 relies on the standard I-smoothness assumption iee., the gradient of I--Lipschizz
invoke the descent lemma and to derive an upper bound on minosisT-1 VZ(()))] When is nonsmooth,
(i) VZ(()) may not cxist at nondiffcrentiable points, which requires working with generalized gradients (c.g., subgradients) rather than classical gradients in the derivation.
(ii) The upper bound on minosisT-1 VVC)))) in the proof of Theorem | (Appendix A.2 scales with the smoothness constant L; when L is unbounded (or effectively very large), these inequalities become non-informative.
Possible extensions.
We outline below several plausible directions; full development is left for future work.
(i) Subgradient analysis. Replace classical gradients with subgradients and assess convergence via mubgradient-based stationarity criterion.
(ii) Smoothing. Introduce a family of smoothed surrogates Lus analyze the MoFO algorithm under these surrogates obtain y-dependent bounds, and then let μ $ 0 recover convergence results for the original loss C.
(iii) Algorithmic modifications. Incorporate techniques such as gradient clipping or stochastic subgradient steps (Xiao al. 2024), and then analyze the convergence of the modified algorithm.
38

---

Published in Transactions on Machine Learning Research (10/2025)
B Supplemental Figures and Explanations
B.1 Reason for using RedPajama approximate LLaMA-2's training data
We note that original LLaMA-2 training dataset has not been publicly released. Thus, we can only rely on public datasets approximate LLaMA-2's original training data.
open-source reproduction of the LLaMA training dataset (Wbeer et al.) 2024 Together 2023). It closely mirrors the data sources outlined in the original LLaMA paper and adopts similar strategics for data collection, mixture, and preprocessing. We believe it serves as a reasonable proxy for approximating LLaMA-2's training dataset.
RedPajama project was explicitly designed
B.2 Supplementary Figures for Figure
Pre-trained model
2.0
1.4
Pre-trained model
4.0
40
Adam
Pre-trained model
Lion
Lion
3.7
Lion
Adam
0.7
Adam
33.3
Accuracy
35
0.0
3.0
2.7
30
-0.6
-..3
dadam
-1.3
2.0
25
2.0
1.7
1.3
-2
-2.7
-2
20
1.0
HellaSwag
ARC-e
ARC-c
Average
-2.5 0.0
5.0
-2.5
0.0
7.5
(c) CR
(a) Fine- Fine-tuning loss landscape
(b) Pre-training loss landscape
Figure 7: The loss landscapes of 1yytha-160MM after fine-tuning on a subset of the FLAN dataset using Adam and Lion. We plot the loss landscapes (a) the fine-tuning dataset and (b) the pre-training dataset (Pile dataset [Gao et al. 2020)) and (c) the accuracies CR tasks, including HellaSwag, ARC-c, and ARC-e. We visualize a 2D weight-space plane spanned by the vector from the pre-trained model to the Lion-tuned model (x-axis) and to the Adam-tuned model (y-axis). Axes are normalized so that one unit equals the length of the ie-trrined-AAdai vector. The color bar indicates the loss value- (a) fine-tuning loss and (b) pre- see-training loss. A logarithmic scale is applied the loss values for better visualization. Two training methods converge to different minima with similar fine-tuning loss. Lion converges farther minimum from the pre-trained model and performs more forgetting than Adam.
B.3 Supplementary Experiments on the Correlation between Distance and Forgetting
this subsection, we augment Figure b(b) by probing the relationship between a model's parameter distance from its pre-trained state and evaluation accuracy under additional training budgets and optimizers. Concretely, under the same settings as Figure Bbb, add runs 0.1 and 0.2 epochs and extend training beyond 3 epochs, using Adam and MoFO. We report results MMLU (as in the main text, mcasuring preservation of factual knowledge) and newly include HumanEval (measuring preservation of code-generation ability). Since the fine-tuning task is math, both can serve as forgetting mitigation metrics. The corresponding scatter plots are shown in Figure S (MMLU) and Figure ! (HumanEval), When examining the points for each optimizer separately, we make the following obscrvations:
Observation 1 (sufficient training). Once training exceeds approximately epoch, MMLU and HumanEval scores show a consistent strong negative correlation with the parameter distance to the pre-trained stale.
Obsorvation (carly training). For Adam or MoFO less than epoch, we may observe a mild correlation with the parameter distance the pre-trained state. The correlation may be unstable and can be positive negative.
39

---

Published in Transactions on Machine Learning Research (10/2025)
0.45
Pretrained Model
0,40
MMLU
0.35
0.30
MoFO
Adam
0.25
Lion
10-2
10
Distance
Figure 8: Average accuracy on the MMLU benchmark (measuring preservation of factual knowledge) for Llama-2-7B after fine-tuning on MetaMathQA with Adam, Lion, and MoFO. Building on Figure 2 b), we add points for runs where Llama-2-7B was trained for 0.1, 0.2, and >3 epochs using both Adam and MoFO. The marker size encodes the number of training epochs (larger means more epochs). Red arrows indicate the points obtained after exactly 1 epoch for each optimizer.
0.30
0.25
Pretrained Model
0.20
HumanEval
0.15
0.10
MoFO
0.05
Adam
Lion
0.00
10-2
10
Distance
Figure 9: Scores on HumanEval benchmark for Llama-2-7B after fine-tuning on MetaMathQA with Adam, Lion, and MoFO. Building on Figure b)... we add points for runs where Llama-2-7B was trained for 0.1, 0.2, and >3 epochs using both Adam and MoFO. The marker size encodes the number of training epochs (larger mcans more epochs). Red arrows indicate the points obtained after exactly 1 epoch for each optimizer.
We speculate that this short-lived positíve trend may be related to benchmark alignment. The MMLU benchmark (used to measure the preservation of factual knowledge) might share partial overlap with the patterns of our math fine-tuning task (measured by GSMSK benchmark); in the early training steps, the model might incidentally acquire features that also benefit MMLU, resulting in temporary gain. By contrast, as for HumanEval in Figure g. which measures code generation and differs from our math fine-tuning in both domain and output format, may exhibit an unstable correlation whose sign can be either negative or positive. Another possible factor could stochasticity, since less than 1 epoch the dataset has not yet been fully traversed.
Overall, the negative correlation becomes clear and consistent after sufficient training; while the early training presents a mild, benchmark--eependeuu relationship.
In addition, we emphasize an empirical point: preservation of pre-trained knowledge is evident across optimizers. In Figure bb) g and the parameter distances roughly follow the ordering Lion > Adam > MoFO, whereas the scores measuring preservation of pre-trained knowledge follow the inverse ordering MoFO > Adam > Lion. For broader comparison,
the negative relationship between parameter distance and the
40

---

Published in Transactions on Machine Learning Research (10/2025)
evaluate five optimizers Pedimizers-MoFO, NAdam (Dozat 2016 Adam, RMSProp Tieleman 2012 and Lion. After two epochs of training, we report (i) their parameter distance to the pre- state and (ii) their forgetiigg-mitiiationn performance on MMLU and HumanEval, shown in Figure [0fa) a) and Figure 10 b), respectively. The results show a consistent negative correlation between distance and performance. Notably, MoFO remains closer to the pre-trained stale and achieves higher scores compared with the other optimizers. Therefore, this cross-optimizer rank-order correlation is sufficient to motivate our algorithm design: favor optimizers that converge closer to the pre-trained state, so better preserve the pre-trained knowledge.
0.25
MoFO
0.40
NAdam
0.20
Adam
MoFO
RMSProp
NAdam
HumanEval
MMLU
0.35
Adam RMSProp
0.15
Lion
Pre-trained model
0.10
Lion
0.30
Pre- tre-trained model
0.05
0.25
0.00
-0-2
10-1
10-2
10
Distance
Distance
(a) MMLU Accuracics
(b) HumanEval Scores
Figure 10: (a) MMLU accuracies and (b) HumanEvval scores of Llama-2-71 2-7B after fine-tuning the Meta- MathQA dataset using different optimizers. All experiments are run for 2 epochs.
B.4
Supplemental Explanation of Example
1.50
Default FT (Adam)
12
Default FT (Adam)
1.2
1.25
LoRA
1.00
MoFD
Random BCD
Izyregularization
.0.0
1.00
MoFO
Ly-regularization
0.2
-1.2
Random BCD
HFT Pro-trained model
0.75
Pre-trained model
-2.4
0.50
2.0
0.25
3.1
0.00
4.2
-0.5
-0.25
-5.3
-0.50
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
-0.5
0.0
0.5
1.0
1.5
6.
0,
Fine-tuning loss landscape
(b) Pro-training loss landscape
Figure 11: The loss landscapes of the example. We plot the landscapes (a) the fine-tuning loss and (b) the pro-training loss. The color bar indicates the loss valuc-(a) fine-tuning loss and (b) pro-training loss. A logarithmic scale is applied to the loss values for better visualization. In Example T MoFO converges to a minimum closest to the pre-trained model, with a low pre-training loss.
In addition Default FT and MoFO, we also analyze four other optimization methods Example 1, namely Le regularization (Li et al. 2018). Half Fine-tuming (HFT) (Hui al.) 2024). Random BCD (Nesterov) 2012 and Low-Rank Adaptation (LoRA) (Hu al. 2022 These methods also introduced in Section
41

---

Published in Transactions on Machine Learning Research (10/2025)
For the L2 regularization method, we add regularization term 2|6 - 6 pretrain 11 to the fine-tuning loss to encourage the model to stay closer to the pre-trained state. As shown in Figure [b), the L-2 regularization approach remains closer to the pre-trained model, thereby achieving a smaller pre-training loss. However, since the fine-tuning objective modified, Figure I1aa shows that Ly regularization does not reach the minimum of the fine-tuning loss.
We note that HFT operates in a manner similar to Random BCD, which randomly selects a subset of coordinates (e.g., 8 82) to update. Figure 11 further illustrates that both HFT and LoRA do not converge to minima close to the pre-trained model as MoFO does, indicating that they may undergo higher levels of forgetting compared to MoFO.
For LoRA, we make the following modelling in the landscape visualization example. The core principle of LoRA (Low-Rank Adaptation) involves approximating the original training space by a low-rank subspace. Since we consider a two-dimensional training space for visualizing the landscape, we set the rank of LoRA space to 1. Specifically, the parameters 0, and 02 exhibit a linear relationship. Given that the pre-trained model is 00, the parameters under LoRA are set to satisfy 0₂ 380, where 3 is hyperparameter we set to 0.5. Figure shows that LoRA converges to closer local minimum than Default FT.
B.5 Synthetic Experiment for Example 1
In this subscction, conduct synthetic experiment to provide a concrete illustration of Example [ Specifically, set the parameter dimension to d == 10, with parameters 0 (f1....,010)) The pretraining loss is defined as the squared L2 norm of the parameters: Trrtrrain (0) |1012, with the pre-trained model given by Ppettrain (0, .,.,,,..,,, 0). Starting from the pre-trained model, we optimize the parameters with respect to the fine-tuning loss (())
- be)2, where a,, b, > 0 for any S i d. The coefficients a; and by are sampled from a standard normal distribution; to ensure positivity, we take their absolute values and add 0.3 and 0.1, respectively.
In this experiment, we compare two optimizers: Adam and MoFO. For each, we perform a grid search for the optimal learning rate over the set (10-2, 10-3 10-4). We consider the fine-tuning process to have converged to a minimum when the fine-tuning loss drops below 10-8 within 10000 iterations. At convergence, we record two metrics: the Euclidean distance from the fine-tuned model the original pre-trained model, and the value of the pretraining loss. The entire experiment is repeated across three different random sceds for robustness.
As presented in Table from the pre-trainod model compared to the one found by Adam. Concurrently, MoFO achieves a lower pre-training loss. These findings provide strong evidence that MoFO can converge to minima closer to the pre-trained model in Example T thereby supporting Theorem 2
4 the results indicate that the minimum found by MoFO is at roughly half the distance
Table 4: The Euclidean distance from the finc-tuned model to the original pre-trained model, and the pretraining loss of parameters after optimizing the fine-tuning loss using Adam and MoFO. The results show that MoFO finds fine-tuning minimum that is closer the pre-trained model compared to the one found by Adam.
Adam
MoFO
Distance to pro-trained model
0.549
0.275
Pre-training loss
0.151
0.038
42

---

Published in Transactions on Machine Learning Research (10/2025)
C Implementation Details
C.1 Datasets for Fine-Tuning
MetaMathQA Yu et al. 2024b). This dataset comprises 395K math question- answer pairs. Numerous studies indicate that LLMs significantly enhance performance metrics on mathematical benchmarks such as GSM8K after fine-tuning on this dataset. We randomly select 10% of this dataset for training LLMs, which includes 39.5K question-answer pairs.
AMC-LLaMA-Imstruttionn (Wu et al. 2024 This dataset comprises 514K instruction-responss pairs. Fine-tuning LLMs on this dataset has been shown to enhance performance on medical NLP tasks, such as PubMedQA (Jin et al. 2019), MedMCQA (Pal et al.) 2022). and MedQA (Jin et al. 2021). We randomly sampled 51K instances with prompt lengths less than 750 characters for training our models.
(Wei et al. 2023). This dataset comprises 110K instruction- pairs related to coding, and is decontaminated and redistributed from the Pvol-CodeAApacea dataset. (Tuo et al. 2023b) We randomly select 39.5K question-answer pairs from this datasct to fine-tune LLM and enhance their coding capability.
Eeggscoddr-Evol-nnsttee
TRACE benchmark dataset (Wang et al. 2023b). TRACE benchmark is designed with a comprehensive set of 8 distinct tasks across various domains, including domain-spocific knowledge, multilingual proficiency, code generation, and mathematical reasoning.
C.2 Evaluation Metrics for Instruction Fine- Tuning
We employ a comprehensive suite of widely used benchmarks to assess the performance and potential catas- trophic forgetting effects on the general capabilities of LLMs after instruction fine-tuning The benchmarks are as follows:
Factual knowledge (MMLU): We use the Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al. 2021 to evaluate factual knowledge across 57 diverse subjects, ranging from STEM fields and the humanities to social sciences. Evaluations are performed using 8-bit precision with the open-instruct implementation, and by following the setup of (Hui et al. 2024 we report the 0-shot accuracy.
To measure the commonsense reasoning capabilities of LLMs, we employ the widely recognized benchmarks ARC-Challenge (ARC-C), ARC-Fasy (ARC-E) Clark ct al. 2018), and HellaSwag [Zellers et al.
Common sense reasoning (CR):
2019). collectively referred to as the Commonsense benchmark. We use the average of their metrics the evaluation, conducting assessments using the LM Eval Harness framework (Gao et al.
2023 and reporting the 0-shot accuracy based on the
norm, none" metric.
Mathematical Reasoning (GSM8K): We assess mathematical reasoning capability using GSMSK Cobbe ct al. 2021). which consists of 8.5K high-quality grade school math problems. Evaluations conducted on the (est set using the LM Eval Harness framework prompting in 5-shot setting, reporting the "exact__match, flexible-extract! metric.
Code Generation (valanEval):: We adopt HumanEval (Chen el al. 2021 comprising 164 unique programming problems, to evaluate the coding capabilities of LLMs. For chat experiments, we report the pass@10 performance.
Medical Question Answering (MedQ): To assess medical knowledge, we utilize three bench- marks PubMedQA (Jin et al., 2019), ModMCQA (Pal al.) 2022), and MedQA (Jin et al., p. Evaluations are performed using the LM Eval Harness framework. For PubMedQA, we report the "ace, none" metric; for MedMCQA and MedQA, we report the "acc. norm, none" metric.
43

---

Published in Transactions on Machine Learning Research (10/2025)
We evaluate the instruction-folloinng ability of LLMs using the [Feaal benchmark. Evaluations are conducted with the LM Eval Harness implementation, and we report the 'inst level strict acc, none" metric.
Instruction Following (IFEval):
All benchmarks including CommonSense, GSM8K, PubMedQA, MedMCQA, MedQA, and IFeval are evaluated using the LM Eval Harness framework (Gao et al. 2023), following their default settings unless specified otherwise.
C.3 Hyperparameter Configurations
Instruction fine-tuning. In our instruction fine-tuning experiments, we follow the implementation of Ivison et al. (2023). For instruction fine-tuning, batch size to 128, and we train the model for epochs. For the Llama-2-7B model, 2e-5 and O warm-up ratio, with cosine decay learning rate scheduler. The learning rate is set to 26-5 for fine-tuning both the Llama-2-7B-Chat model on the MetaMathQA dataset and the Gemma-2B-- model, while a learning rale of le-5 is used for fine-tuning the Llama-2-7B-Char model on the PMC-LLaMA-Instrus dataset; all these settings employ a warm-up ratio of 0.03 and a cosine decay learning rate scheduler. For LoRA, we set the learning rate as lc-4. The other hyperparamcters the experiments are as follows.
the maximum sequence length to 1024, the global
learning rate of
Fine-tuning Llama-2-7B MetaMathQA.
Learning rate: 2e-5.
Update fraction of MoFO: 15%.
LoRA: r = 4, 16, 64, 256. We report the best-performing hyperparameter configuration for the fine-tuning task in Table 1 which, in this case, 256.
Fine-tuning Llama-2-7B-Cha on PMC-LLaMA, Insttruct.
Learning rate: 1e-5.
Update fraction of MoFO: = 15%.
LoRA: 16, 256. We report the best-performing hyperparameter configuration for the fine-tuning task in Table 5 which, in this case, is r = 256.
Fine-tuning Llama-2-7B-Chat MetaMathQA.
Learning rate: 2e-5.
Update fraction of MoPO: 15%.
LoRA: r 16, 256. We report the best-performing hyperparameter configuration for the fine-tuning task in Table 7 which, in this case, 256.
Fine-tuning Gemma-2B-IT on MetaMathQA.
Learning rate: 2e-5.
Update fraction of MoFO: a 5%.
LoRA: r = 16, 256, 512. We report the best- hyperparameter configuration for the fine- tuning task in Table g which, in this case, is 512.
Fine-tuning Llama-2-77 Maama-2-7B-CCr 7B-Chht
44

---

Published in Transactions on Machine Learning Research (10/2025)
Learning rate: 1e-5.
Update fraction of MoFO: a = 20%.
LoRA: = 16, 256. We report the best-performing hyperparameter configuration for the fine-tuning lask in Table 6 which, in this case, is r 256.
Hyperparameters in the Pareto comparison. To provide comprehensive comparison, explore various hyperparameter settings for Å1 X2, LoRA's rank, and the update fraction in MoFO in Figure Specifically, we set a as Ic-4, le-5, 1c-6, 1c-7, while x2 is set as 1c-2, 5e-3, 1c-3, 50-4, and 1c-4. The update fraction a in MoFO is set as 5%, 10%, 15,, 20%, 40%, 80%. The rank of LoRA is set as 4, 16, 64, 256.
Continual fine-tuning. In our continual fine-tuning experiments, we follow the default settings of the TRACE benchmark. We sequentially train TinyLlama-1.1B on the TRACE benchmark datasets: C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-em, NumGLUE-ds, and 20Minuten for 3, 5, 5, 5, and 7 epochs, respectively. We use learning rate of 1e-5 with cosine decay schedule and a batch size of 64. The parameter update fraction for MoFO is set to 5%.
All experiments are conducted on four A800 (80GB) GPUs.
C.4 More Explanation the partitioning and Calculation of distance
Partitioning. We use the default partitioning scheme in PyTorch's Transformer implementation. Different types of parameters within the Transformer, such as query (Q), key (K), value (V) weights for attention heads, and feed-forward network (FFN) weights, are divided into separate partitions. Notably, in the default PyTorch implementation, within a layer, the query (Q) weights of all attention heads are grouped into a single partition. The same applies the key (K) and value (V) weights. Our momentum-based filtering mechanism applied each partition individually, A different parameter partition scheme, along with its corresponding experiments, is presented in Appendix F.6
Calculation of distance. Following the notation in Section 2.2 we suppose that the parameter parameters are partitioned into
= (0(1) ((2)
((B)).
Denote the pre-trained model by 8g and the finc-tuned model by 6.
First, we calculate the relative change of parameters [10(*) _g00)
in each partition & E (1, BB. Second, we compute the distance from the pre-trained model 0g the fine-tuned model by averaging the relative changes across all partitions, defined as:
1106
[0(R) 064
D(O 0o)
|0%
45

---

Published in Transactions on Machine Learning Research (10/2025)
D
Guideline for Setting Cr
Given a pro-trained LLM and datasct for fine-tuning, we recommend the following procodure:
Random Sampling: Randomly sample a small subset of the dataset to serve as a proxy.
2. Grid Search: Perform grid search over candidate values of using this proxy subset.
Selection: Choose the a configuration that strikes good balance between fine-tuning performance on the target dataset and preserving the model's general capability.
To illustrate this procedure, from the current training set of MetaMathQA dataset (39.5k) for fine-tuning, and then perform grid search over a values of 5%, 10,, 15%, 20%, 40%, and 80%. As shown by the green line in Figure 12(a). a), the fine-tuning performance relatively stable across these values a. However, the green line Figure indicates that a 15% best preserves the model's general capability. Therefore, we set a = 15%.
Llama-2-74 as example. We randomly sample 10% of the instances
50
Average accuracy changes
Accuracy on GSMBK
40
MoFO
-2-
MoFO
MoFO with small Pre-trained model
MoFO with small data
20
-4
Pre-trained model
Default SFT
Default SFT
5%
10%
15%
20%
40%
80%
10% 15%
20%
40%
80%
Parameter update fraction
Parameter update fraction
(a) Fine-tuning performance
(b) Preservation pre-training knowlodge
Figure 12: (a) Fine-tuning performance: Accuracy on the GSM8K math reasoning task for LLMs of different sizes, fine-tuned via MoFO with varying update fractions (a)). (b) Preservation of pre-training knowledge: Average accuracy changes MMLU, HumanEval, and commonsense reasoning benchmarks relative to the original pre-trained LLMs, illustrating how much pre-training knowledge is retained. All results are obtained by fine-tuning Llama2-78 7B on MetaMathQA and its proxy subset. The performance trends under different update fractions on the proxy subset align with those observed on the full dataset.
Furthermore, by comparing the blue and green lines in Figure 12 we observe that the trend of model performance with respect to on the small proxy subset is consistent with the trend observed when fine- tuning on the full dataset. This implics that small, randomly sampled subset is sufficient to guide the selection of suitable
Empirically, the optimal often lies between 5% and 20%. A more fine-grained grid search within this range can be performed if needed. Designing more refined and efficient strategies for tuning a is left for future work.
46

---

Published in Transactions on Machine Learning Research (10/2025)
E
Additional Experiments on Instruction Fine-tuning
This section begins with a comparison of MoFO and bascline methods across additional datasets and models explore the combination of LoRA and MoFO to assess their performance. Finally, in F33 compare MoFO with several algorithms designed to mitigate forgetting.
in
F.1
In
R.2
E.1 More Experimental Results in Instruction Fine- Tuning
Table 5: The performance on the fine-tuning task (medical QA task), measured by MedQ, and general capability scores of Llama-2-7B- Chat after fine-tuning the PMC-LLaMA-Instreet datasct. The figure the right visualizes both MedQ accuracy and general capability scores. The results show that MoFO achieves comparable performance in the MedQ while significantly mitigating forgetting of general capabilities. Bold values denote the best results among these methods.
Ceneral Capability
Method
MedQ
CR
IFEval
HvmanEval
Avg.
0.55
Llama-2-7D-Chat
49.8
65.6
41.4
24.3
43.8
MedO
0.53
Default FT
Default FT
54.3
32.1
20.6
39.1
0.50
HFT
LlamaZ- 2-7b-chat
HFT
54.4
65.2
33.5
23.1
40.6
LoRA
LoRA
54.2
64.4
33.9
23.5
40.6
0.48
MOFO
MoFO
54.3
65.6
38.6
25.0
43.1
0.375
0.400
0.425
0.450
General capability
on the PMC- LLaMA-Instruction. dataset using various bascline methods and present the experimental results medical question answering (MedQ) and general capabilities in Table 5 Since the MMLU benchmark already contains medical-related instances (Hendrycks et al. 2021), which may lead to improved performance after fine-tuning, we instead use IFEval to assess general capabilities.
Results of fine-tuning on PMC-LLaMA-Instee
We fine-tune
MoFO performs well on the fine-tuning lask medical QA. It achieves compatible performance compared to Default FT and HFT. In terms of general capabilities, MoFO demonstrates the least degradation compared other baselines, with an average accuracy reduction of only 0.2%. Specifically, on the EvEval benchmark, our method only exhibits minor reduction of 0.3%, while Default FT, HFT, and LoRA experience significant degradations ranging from 7.5% to 9.3%. On code generation (HumanFval) tasks and commonsense reasoning (CR) benchmarks, our method also only exhibits a minor reduction less than 0.2%.
The performance of the fine-tuning task (math), measured by GSM8K, and the general capability scores of Gemma-2B-IT after fine-tuning on the MetaMathQA dataset. The figure on the right visualizes both GSMSK accuracy and general capability scores. The results show that MoFO achieves comparable performance in the fine-tuning task, while significantly mitigating forgetting of general capabilities. Bold values denote the best results among these methods.
Table 6:
General Capability
GSMBK Ac Accuracy
0.4
Method
GSMSK
Default FT
CR
[eval
HumauEval
Avg.
0.3
HFT
Gemma-2B-IT 2B-IT
11.4
57.6
33.6
31.5
40.9
LoRA
Default FT
42.0
52.1
24.3
20.6
32.3
0.2
MOFO
HFT
41.5
53.9
24.1
21.2
33.1
Cemmm-2BBIT
LoRA
40.6
G4.4
26.1
29.8
36.8
0.1
0.325
0.350 0.375 0,400 General capability
MoFO
42.1
55.0
28.7
29.1
37.6
47

---

Published in Transactions on Machine Learning Research (10/2025)
Results of fine-tuning Gemma-2B-IT MetaMathQA. We also explore how MoFO performs in other I.LMs. Specifically, we fine-tune Gemma-2R-IT MetaMathQA using various baseline methods and present the experimental results on mathematical reasoning (GSM8K) and general capabilities in Table 6 The experimental results demonstrate that MoFO achieves comparable performance of the finc-tuning task to Default FT and HPT across different models. terms of general capabilities, MoFO exhibits significantly less forgetting compared other baselines. This result demonstrates the versatility of the MoFO algorithm.
Table 7: The performance of the fine-tuning task (math), mcasured by GSM8K, and the general capability scores Tlamm-2-7BB-cha after fine-tuning on the MetaMathQA dataset. The figure on the right visualizes both GSMSK accuracy and general capability scores. The results show that MoFO achieves comparable performance in the fine-tuning task, while significantly mitigating forgetting of general capabilities. Bold values denote the best results among these methods.
0.5
General Capability
GSMBK Accuracy
Method
GSM8K
CR
IFval
HumanEval
Avg.
0.4
Default FT
HFT
Llama-2-T7 1anna-2-BB-haan
13.7
65.6
41.4
24.3
43.8
0.3
LoRA
Default FT
48.4
62.8
30.7
15.6
36.4
MOFO
HFT
46.9
63.4
31.8
20.0
38.4
0.2
LoRA
45.3
63.9
35.6
21.0
40.2
-lama-2-7B-Chatt
MoFO
47.1
64.0
37.1
21.7
40.9
0.375
0.400
0.425
General capability
Results of finc-tuning Llama-2-7B-Chat on MetaMathQA. We also finc-tune the Llama-2-7B. Chat on the MetaMathQA dataset. The results are presented in Table The results demonstrate that our approach achieves performance comparable to Default FT and HFT while exhibiting less forgetting compared to baseline methods.
Table 8: The performance of the fine-tuning task (coding), measured by HumanEval, and the general capability scores of Llama-2-7B-Chaa after fine-tuning the Eaggicodar-Evo--nnsttee dataset. Here we choose the three benchmarks exhibiting the most significant forgetting. We set the rank of LoRA 256, and of MoFO is set as 20%. The results show that MoFO achieves comparable performance in the fine-tuning task, while mitigating forgetting of general capabilities. Bold values denote the best results among these methods.
General Capability
Method
HumanEval
0.50
ARC-E
ARC-C
IFEval
Avg.
HumanEva
Default FT
Llama-2-7B-Chal
24.2
74.5
46.3
41.1
54.0
0.40
HFT
Default FT
56.2
71.2
45.2
33.5
50.0
LoRA
HFT
50.7
71.5
45.6
36.3
G1.1
0.30
MOFO
70 Chat at
LoRA
48.6
72.1
45.1
33.4
50.2
0.50
0.52
0.54
MoFO
53.3
72.1
46.0
36.1
51.4
General capability
Results of fine-tuning Llama-2-7B-Chat on Nagicodor--vo--nsstuee We also finc-tune the Llama-
2-7B-Chat on the Magicoder--Evol-nssttue dataset. We use ARC--ass (ARC-E) and ARC-Challenge (ARC-C) scores to measure general capability. The results in Table g demonst that our approach outperforms the baselines the fine-tuning tasks and exhibits less forgetting compared to baseline methods.
In summary, our MoFO algorithm shows competitive performance in instruction fine-tuning while preserving the general capabilities, effectively alleviating forgetting.
48

---

Published in Transactions on Machine Learning Research (10/2025)
0.50
0.45
GSMBK
0.35
HFT
(ytegullrication (g-regularization MOFO
0.25
0.20
HMA
0.15
0.63
0.64
0.65
0.66
Figure 13: The performance on the math task (GSM8K) and the scores in Commonsense Reasoning of Llama-2-7B after fine-tuning on the MetaMathQA dataset. The results show that the MoFO algorithm achieves a better Pareto front. The pink triangle represents the model obtained through HMA.
E.2 Experiment the Combination of MoFO and LoRA
In addition to using LoRA as baseline for comparison, can also view it as orthogonal method that can be integrated with MoFO. In the fine-tuning stage, LoRA restricts the trainable parameter space to a low-rank subspace. We note that the comparison of LoRA and MoFO (PEFT version) essentially evaluates MoFO and Adam within this same low-rank subspace defined by LoRA. To investigate this further, we conduct a comparative experiment following the setup in Table of Section 4.2 Table implies that MoPO ToRA effectively mitigates the forgetting issue that arises when using LoRA alone.
Table 9:
The performance of the fine-tuning task (math), measured by GSM8K, and the general capability scores of Llama-2-7B after fine-tuning on the MetaMathhQA dataset. The results show that MoFO LoRA preserve more pre-training knowledge than using LoRA alone.
General Capability
Method
GSM8K
Avg.
CR
MMLU
HumanEval
LoRA
43.3
65.1
37.7
26.4
43.1
LoRA + MoFO
43.4
65.5
39.4
26.7
43.9
E.3
Comparison with More Fine- Tiie-TTning Methods
Experiments on Heterogeneous Model Averaging (HMA)
We compare proposed method with the Heterogencous Model Averaging (HMA) (Ln et al. 2024a.. HMA approach evenly divides the L.M into three
input part, the middle part, and the output part- aart-and averages these parts with different ratios. To facilitate a comprehensive comparison, following the setting in Section 4.2 we evaluate the fine-tuning and forgetting mitigation performance for different HMA strategies. We select 15 different combinations of averaging ratios for different parts as follows: {(0.05, 0.2, 0.35), (0.1, 0.2, 0.3), (0.2, 0.2, 0.2), (0.3, 0.2, 0.1), (0.35, 0.2, 0.05), (0.3, 0.5, 0.7), (0.4, 0.5, 0.6), (0.5, 0.5, 0.5), (0.6, 0.5, 0.4), (0.7, 0.5, 0.3), (0.65, 0.8, 0.95), (0.7, 0.8, 0.9), (0.8, 0.8, 0.8), (0.9, 0.8, 0.7), (0.95, 0.8, 0.65)). We plot the results to construct Pareto front in Figure 13
Results show that our proposed method, MoFO achieves more effective Parcto front compared to the baselines.
Experiments on CoFiTune and Soft-masking
Zhang al. (2024a) introduces CoFiTune, a coarse-to-fine framework that balances specificity and versatility in LLMss by selectively updating specific modules and employing soft-masking mechanism, which is introduced
49

---

Published in Transactions on Machine Learning Research (10/2025)
by Ke al. 2023b a). We have compared MoFO with CoFiTune (with and without soft-masking) and the vanilla. soft- molt-ng method alone, following the setting in Table Tff Section 4.2 The results, presented in Table 10 below, demonstrate that MoFO outperforms these methods in both fine-tuning performance and mitigating forgetting. The results demonstrate that
CoFiTune achives similar forgetting mitigation performance as MoFO, but underperforms MoFO on fine-tuning lasks.
Vanilla Soft-masking exhibits slightly reduced performance in both fine-tuning tasks and mitigating forgetting than MoFO. These findings underscore the advantages of our proposed method.
Table 10:
The performance of the fine-tuning task (math), measured by GSM8K, and the general capability scores of Tlama-2-7B after fine-tuning on the MetaMathQA dataset. The results show that MoFO achieves comparable performance the fino-tuning task, while significantly mitigating forgetting of general capabilities.
General
Capability
Method
GSM8K
CR
MMLJU
HumanEval
Avg.
MoFO
47.7
65.7
42.7
24.6
44.3
Vanilla-SoftMask
46.4
65.6
42.9
23.2
43.9
CoPiTune w/o SoftMask
37.7
42.1
25.8
44.4
CoFiTune w/ SoftMask
34.4
65.0
41.5
25.6
44.0
From the results, we can see that MoFO achieves higher scores on the fine-tuning tasks while effectively reducing knowledge forgetting, demonstrating its superiority over these methods.
E.4 Comparison with More Parameter-Efficeene Fine- Muning Methods
Parameter- Efficient Fine-Tuning (PEFT) encompasses a collection of fine-tuning methods designed reduce the computational cost required for model training. In this subsection, mainly focus three famous PEFT methods:
Adapter (Houlsby et al. 2019: This approach involves inserting trainable "adapter' modules into every layer of model. During fine-tuning the parameters of these adapter modules are updated for downstream tasks, while the majority of the original model's parameters remain frozen.
BitFit Zaken et al. 2021 This method reduces the number of trainable parameters by fine-tuning the bias terms of the model on given downstream task, keeping other weights frozen.
LoRA Hu et al. 2022 : is a widely-used, parameter-efficient tuning method. LoRA trains low-rank matrix adaptations on the base model''s weights. Recent work (Biderman et al., 2024) demonstrates that LoRA can mitigate forgetting.
We compared MoFO with Adapter, BitFit, and LoRA, following the setup in Table I of Section 4.2 For the Adapter method, we set the learning rate to 1e-4 and performed grid search for the adapter size over the values (16, 64, 1283. We report the best-performing hyperparameter configuration for each method on the fine-tuning task.
As shown in Table 11 all three PEFT basclines (BitFit, Adapter, and LoRA) achieve higher average general capability scores than Default FT, indicating that they are indeed effective at mitigating catastrophic forgetting. However, they still lag behind MoFO in this regard.
Morcover, compared MoFO, the three PEFT methods perform markedly worse on the fine-tuning task (GSM8K), with the gap being especially pronounced for Adapters and BitFit. We conjecture that this may stem from the fact. that Adapters and Fit were originally proposed and evaluated relatively simple
50

---

Published in Transactions on Machine Learning Research (10/2025)
classification QA benchmarks (e.g., GLUE) with masked language models. While these methods were effective in such settings, they may be less suited to today's s more challenging domain-specife LLM tasks, leading to the weaker performance observed here.
Table 11: The performance of the fine-tuning task (math), measured by GSM8K, and the general capability scores of Llama-2-77 2-7B after fine-tuning on the MetaMathQA dataset. The results show that MoFO outperforms three PEFT methods.
General Capability
Method
GSM8K
CR
MMLU
HumenEval
Avg.
Llama-2-7B
13.7
65.6
42.0
24.2
43.9
Default FT
49.4
62.3
36.6
16.1
38.3
MoFO
47.7
65.7
42.7
24.6
44.3
BitFit
15.1
36.1
24.4
Adapter
24.4
63.2
32.3
21.8
39.1
LoRA
43.3
65.1
37.7
26.4
43.1
51

---

Published in Transactions on Machine Learning Research (10/2025)
Pre-trained
1.5
Fine Runed model (MoFO)
tuned (Adam)
1.354
1.5
Fine-tuned model (MoFO)
model (Adam)
2.636
0.684
2.432
2.228
000115
0.5
Suadwons
0.5
2.024
-0.655
1.819
0.0
-1.324
SACTO
1.615
1.993
-0.5
1.207
-26663
-1.0
-1.0
1.003
0.4
.0.2
0.0
0.2
0.4
0.5
0.7
-0,4
-0.2
0.0
0.7
(a) Loss landscape on fine-tuning dataset
(b) Loss landscape pre-training dataset
The loss landscapes of Pythia-160m after fine-tuning on subset of the FLAN dataset using Adam optimizer and MoFO. We plot the loss landscapes on (a) the fine-tuning dataset and (b) the pre-training dataset (Pile). We visualize a 2D weight-space plane spanned by the vector from the pre-trained model to the MoFO-tuned model (x-axis) and to the Adam-tuned model (y-axis). Axes are normalized so that one unit equals the length the pre-trained-> Adam vector. The color bar indicates the loss value-(a) finettnnig loss and (b) pre-training loss. A logarithmic scale is applied the loss values for better visualization. We find that MoFO, reaching closer point to the pre-trained model, has minimal fine-tuning loss and lower pre-training loss, compared to Adam.
Figure 14:
F More Explorations on MoFO
This section aims to provide deeper understanding of MoFO through series of experiments. In Appendix F.2 we conduct an elficiency analysis of MoFO. In Appendix F.3 we present. additional comparalive experiments on different filtering strategies. In Appendix F.4, we investigate why the momentum- -ilttrred update rule in MoFO achieves optimal fine-tuning performance compared to other update strategies. Appendix F.5 we present preliminary analysis on why MoFO might compare favorably to L1/L2 regularization. In Appendix F.6 investigate the performance of MoFO under alternative parameter partitioning strategy. Finally, in Appendix F.7 explore the application of MoFO to the Lion optimizer.
F.1 Validating MoFO's Impact on Preserving Pre-training Knowledge through Proximity
In this section, we empirically examine whether MoFO achieves its intended goal of converging to a minimum closer to the pre-trained model and mitigating forgetting mentioned in Section
Our exploratory experiment shows that MoFO indeed converges to minimum closer to the pre-training model. As shown in Figure 14(a), both MoFO and the Adam optimizer achieve minimal fine-tuning loss, indicating that switching from Adam to MoFO does not lead to performance degradation. Moreover, the distance from the pre-trained model to the minimum reached by MoFO is approximately 20% of that reached by the default Adam optimizer.
Our experiment demonstrates that the reduced parameter movement achieved by MoFO effectively mitigates the forgetting of pre-training knowledge. As shown in Figure 14(b), the fine-tuned model using MoFO experiences smaller increase in pre-training loss. Additionally, Table 12 shows that MoFO achieves higher accuracy commonsense reasoning tasks, indicating less forgetting.
52

---

Published in Transactions on Machine Learning Research (10/2025)
Table 12: 1rthia-660m'ss performance on common sense tasks, after being fine-tuned with the Adam optimizer and MoFO. The results indicate that MoFO significantly mitigates catastrophic forgetting. Bold values denote the best results among these optimizers.
HellaSwag
ARC-casy
ARC-challenge
Average
Pythia-1660 160m
30.1
39.6
23.8
31.2
Adam
28.3
37.4
22.1
29.3
MoFO
29.9
42.0
22.9
31.6
F.2 Efficiency Analysis on MoFO
We claim that MoFO does not lead to significant reduced fine-tuning efficiency. We provide an efficiency analysis by comparing the total training time between MoFO and Default FT on three LLMs with different sizes. The parameter update fraction of MoFO is set as 10%. The experimental results show that although MoFO requires computing a filter to select parameters with the largest momentum magnitudes, the additional computational overhead is minor. As shown Table 13 the additional training time incurred by MoFO is approximately 4% 5%, which relatively minor and manageable in practical applications.
Table 13: Comparison of total training time for Default FT and MoFO various LLaMA models and datasets. The additional time incurred by MoFO is around 4 5%, which is relatively minor in practical applications. LLaMA3-SB is fine-tuned on the UltraFeedback dataset (Cui et al. 2024
Model
Default FT
MoFO
Additional Training Time
LLaMA3.2-1B
49m22s
5lm24s
4.1%
LLAMA3.2-3B
1h30m18s
1h34m24s
4.5%
LLMMA38B
5h2m0.69s
5h17m34.00s
5.0%
F.3 Further Comparative Experiments on Filtering Strategies
To further substantiate the claim in Section
conduct an additional comparison of filtering strate- coop-complementary to the results in Table 3 using different dataset and model. Specifically, we fine-tune Gemma-2B-IT on the IFEval-like dataset using several BCD variants. The 1F'val-like dataset contains instruction response pairs in the style of the 1FEval benchmark. For these experiments, we randomly sample 39.5k instances from the dataset for training, set the learning rate to 1x10-5, and train the model for 2 epochs. We evaluate finc-tuning performance with the IFEval benchmark and assess general capabilitics with CR (common-sense reasoning), HumanEval, and BBH (O-shot) (Suzgun et al. 2022
As shown in Table 14 all BCD variants mitigate forgetting: their average general-capability score exceeds that of Default fine-tuning by at lcast 2.6%. Although MoFO is lower than MV BCD by only 0.1% in average general capability, it achieves the strongest performance on the target fine-tuning task (IFEval), outperforming Random BCD, Grad BCD, and MV BCD by 5.8%, 2.6%, and 1.3%, respectively.
Taken together, these results indicate that while all filtering strategies are broadly effective at mitigating forgetting, they exhibit distinct differences in task-specific fine-tuning performance. In our setting, MoFO the strongest methods forgetting mitigation while delivering the
offers favorable highest IF score among the tested strategies.
TThe "filtered subset from http:///wagggggscccccddaaaaaaalt
53

---

Published in Transactions on Machine Learning Research (10/2025)
Table 14: The performance on the instruction-folloing task (IFEval) and general capability scores of Gemma-2B-IT after fine-tuning on IFEval-like dataset using different updating strategies MoFO. Here we choose the three benchmarks exhibiting the most significant forgetting. For all the update strategies, we set the parameter update fraction as 10%. Bold values denote the best results among the BCD methods.
General Capability
Method
IFEval
CR
HumanFval
BBH
Avg.
Iemma-2R-IT
33.6
57.6
31.5
32.7
40.6
Defanlt FT
56.7
56.9
22.9
32.5
37.4
Random BCD
51.4
57.5
28.0
33.9
39.8
Grad BCD
54.6
57.3
28.1
34.1
39.8
MV BCD
55.9
57.5
28 4
34.6
40.2
MoFO
57.2
57.3
28.5
34.4
40.1
F.4 Insights of the Choice of Filtering Strategy
In this section, we attempt to address the question: What makes the momentumn-fitered update rule optimal among the candidates?
We hypothesize that the good performance of the momentun-filtered updating rule ariscs from ability promote more stable and consistent updates throughout training. Specifically, we hypothesize that:
Utilizing momentum instead of gradient filtering leads to more stable updates. Momentum accumulates historical gradients, promotes stability by smoothing out fluctuations in the gradient updates. Thereby, during the training process, the moment un-filtering mechanism chooses updating parameters in more stable manner.
2. Excluding the introduction of in the filtering mechanism contributes to more stable updates. The 2nd-order moment i may normalize gradients based on their magnitudes, potentially averaging out the importance of individual parameters within the filtering mechanism. Thereby, during the training process, not incorporating may help choose updating parameters a more stable manner.
For the ablation study, we add the GV-filtered BCD methods a bascline, which replaces MoFO's filter by FLT, /v)). Following the setting of experiments in Appendix B.2 and Figure 1 with the updating fraction = 3% over approximately 200 steps. To assess how many parameters change significantly during training, we calculate the percentage of weight parameters whose absolute change exceeds a threshold of 26-6.
all four methods
Table 15: Percentage of weight parameters with significant changes (absolute change x 10-6) during training.
Method
Percentage of Significant Updates
MoFO
29.8%
Gradient BCD
35.7%
MV-filtered BCD
83.6%
GV-filtered BCD
87.1%
Table 15 indicates that the parameter updating process of MoFO is more stable, which we think may contribute to its better fine-tuning performance.
54

---

Published in Transactions on Machine Learning Research (10/2025)
F.5 Preliminary Analysis Why MoFO Might Compare Favorably to L1/L2 Regularization
As shown in Figure (Section 4.2 when line-tuning Llama-2-7B MelaMathQA, MoFO yields a more favorable Pareto front balancing fine-tuning performance and forgetting mitigation than L1/L2 regulariza- tion and LoRA across hyperparamecerr configurations. This section presents a preliminary analysis offering one possible explanation for why MoFO may compare favorably to L2/L2 regularization. Importantly, we reuse the exact checkpoints from Figure no additional training runs are introduced.
Setup. We reuse the Llama-2-7B MetaMathQQ sweeps underlying Figure including LI, L2, and MoFO across the same hyperparameters. For each checkpoint, we compute the (unregularized) fine-tuning loss and the l2 norm of the gradient on 512 examples. To factor out severe forgetting, we report results for the subset whose CR scores fall within an acceptable range (same CR metric as in the main text). Figure 15 a) and Figure 15 plot mne-tuning loss and gradient norm versus CR, respectively, for this subset checkpoints drawn from Figure 5
Observations. For comparable CR scores, MoFO checkpoints generally achieve lower fine-tuning losses than those trained with In or L2; moreover, they also exhibit smaller gradient norms. Together, these appear to suggest that MoFO converges better than L1/L, L2 regularization.
One plausible explanation. Penalty methods promote proximity to the pretrained parameters, but they also modify the training objective and its gradient field, thereby possibly hindering convergence to a local minimum of the original fine-tuning loss. MoFO instead constrains how updates are applied (via momentum filtering) while contimuing to optimize the original fine-tuning loss, which may allow the optimizer follow task-relevant directions more effectively.
0.20
Gradient norm of fine- fine-tuning tiee-tuning task
L1 reg
L1 reg
0.18
L2 reg MOFO Llama2-78
L2 reg
0.16
MOFO
rre-tuning loss
Llama2-7B MoFO's average
0.14
MoFO's average
10-1
0.12
0.10
0.08
0.06
0.645
0.650
0.655
0.660
0.645
0.650
0.655 CR
0.660
CR
(a) Fine-tuning loas
(b) Gradient norm
Figure 15: (a) Fine-tuning loss and (b) gradient norm versus CR score for Tlama2-7B fine-tuned on MetaMathQA with MoFO, L1, and L2 regularization. The results show that MoFO generally achieves a lower fine-tuning loss and gradient norm than L1 and Lg while effectively resisting forgetting.
F.6 A Different Parameter Partition Scheme
In this section, we propose a different strategy for partitioning model parameters. Different from the default. partitioning scheme PyTor'h's Transformer implementation (Appendix C.4), our alternative approach partitions parameters at the granularity of individual attention heads. Recent studies ŞZhang et al. 2024d e) show that the Hessian matrix in Transformers is nearly block-diagonal, with several dense principal sub-blocks. In particular, Zhang et al. (022ch finds that within the same layer, distinct Q (or K) heads form different. blocks in the Hessian.
Motivated by this finding, we treat each head's Q, K, and V weights as separate partitions and apply our momentum-based filtering mochanism to these finer-grained partitions individually. To evaluate our method, conduct experiments the LLaMA2-7B models, comparing the default partitioning approach against our alternative head-level partitioning scheme.
55

---

Published in Transactions on Machine Learning Research (10/2025)
Table 16: Performance on the fine-tuning task (GSM8K) and general capabilities of Llama-2-7B after fino-tuning. Bold values denote the best results among these methods. The updating fraction is 15%.
General Capability
Method
GSMSK
CR
MMLU
HumanEval
Avg.
MoFO with default partition
47.7
65.7
42.7
24.6
44.3
MoFO with individual head partition
46.6
65.7
41.6
24.1
43.8
Algorithm MoPO + Lion
Input: Filtering Uhreshold a, number of partitions with the k-th partition of size dx, hyperparameters B1, 1.,82,A Ả of Lion optimizer, learning rate schedule {ne}.
Initialize mo zero tensors.
3: for iteration from 1, until converge do
4: for partition from to B do
(k)
Dr Va(a C finetune (0,-1)
C (k) = m (1 B1)9
for entry index i from 1 dg do
FLLT& (ma-1)l if [m Del is within the top-a of Im-1l's values else
8:
9:
end for
0(4)) = (A)) 7// (s (1)) 2 (m,-1) 20
10:
# Momentum Fi Filtering
mi (k) 32m + +1-B))
(k)
11:
12: end for
13: 0. = Concat (0. ,..., !!..,)())) (18)
14: end for
Table 16
shows that both partition schemes yield similar performance.
From the above initial experiment results, we believe that there is room for further exploration in better partitioning strategies.
F.7
Momentum Filtering Mechanism on the Lion Optimizer
A Algorithm Lion Optimizer
1: Input: Number of partitions with the k-th partition of size d,, hyperparameters 31,82, of Lion optimizer, learning rate schedule {nc)-
Initialize my as zcro tensors.
for iteration from 1,2.... until converge do for partition from 1 to B do
gt Vocn C Cinetune(B.. (ft-1)
())
Ct (b) 3/mm-1
(1)) (1 }e (k)
7:
mi 32m + B2)
8:
end for
0, = Concat 10() ,...,, ((B))
10: end for
In this section, we extend our investigation by integrating our proposed MoFO into the Lion optimizer et al. 2024). We first present the original formulation of the Lion optimizer in Algorithm
Chen
Building upon this foundation, we propose Algorithm where momentum filtering is applied to refine the parameter update in every iteration (Lines 7-10).
56

---

Published in Transactions on Machine Learning Research (10/2025)
Following the experimental setup in Appendix B.2 and Figure | conduct comparative experiments to evaluate the iveness of 'MoFO + Lion" in mitigating forgetting. As shown in Table 17 incorporating MoFO into Lion improves the average accuracy of preserving old knowledge by 2.6% compared to using Lion alone. However, 'MoFO + Lion' still lags behind 'MoFO + Adam' by 2.5%.
Table 17: Performance comparison on HellaSwag, eas- and ARC-challenge, along with their average.
Method
HellaSwag
ARC-easy
ARC-challenge
Average
Pythia-160m
30.1
39.6
23.8
31.2
Adam
28.3
37.4
22.1
29.3
MoFO (for Adam)
29.9
42.0
22.9
31.6
Lion
26.5
29.0
24.1
26.5
Lion + MoFO
27.5
36.7
23.2
29.1
57


Reviews:
[{"review_id": "rAgkNIeWcf", "reviewer": "Reviewer_ChzC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is described well. It makes sense to have tried it, and it sounds easy to re-implement.\n- It is nice that the authors targeted something uniquely suited to Adam, by far the most popular LLM optimizer.\n- The authors provide a proof of convergence.\n- Experiments are thorough and convincing. The method has a wide range of competitors, and many of those are represented in at least one experiment.\n\nWeaknesses:\n\n- The motivation section is nice, but it spends a lot of time talking about how crucial it is for a finetuned model to find a local optimum close to the original model parameters. Once we reach the experiments section, and see that both L1 and L2 regularization (which explicitly optimize for proximity to the original model) are pareto-dominated by the proposed technique, it would be nice to spend some amount of time speculating as to why that should be the case. Did FoMO actually do a better job of optimizing the L2-regularized objective? Or is it possible to be too close to the original parameters? Honestly, this isn’t much of a weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes a new variant of the Adam optimizer called Momentum-Filtered Optimization (MoFO) that is designed to allow language models to finetune to new tasks while retaining more of their original capabilities, that is, while avoiding forgetting. The optimizer works by selecting a small percentage (typically between 5 and 15%) of parameters in each parameter block to be updated. Parameters are selected based on which have the largest Adam momentum values. Blocks can be practically defined to correspond to natural parameter groupings in popular implementations. The authors provide a proof that this optimization converges, adapting a recent proof for the Adam optimizer. They also provide experiments that show the method outperforms reasonable baselines drawn from optimization (half fine-tuning), architecture (LORA), regularization (L1 or L2 toward the original model), a host of continual learning techniques (GEM, EWC, Replay), and a number of block coordinate descent variants (Random, Grad). The experiments are on fairly small models (Llama-2-7B, Gemma-2B, TinyLlama-1.1B), but cover a nice collection of tasks. Results are convincing overall.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None. The paper seems fine to me.", "broader_impact_concerns": "None. An optimization paper such as this has no ethical concerns beyond those inherent to machine learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M4RFkLx9TZ", "reviewer": "Reviewer_NmJ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper includes a thorough theoretical analysis of the convergence behavior of MoFO.\n\n- The proposed momentum filter is conceptually simple and may have implications for other types of optimization algorithms and settings.\n\n- The proof of Theorem 1 is interesting and relies on a key norm condition on the momentum filter.\n\n- Theorem 2 is interesting. I think it would be useful to run some synthetic experiments on top of Example 1 to validate the claimed relation between MoFO and Adam.\n\nWeaknesses and Major Comments:\n- The additional technical contribution on top of the prior paper by Shi et al. (2021) can be more clearly spelled out. Specifically, I think the technical exposition in Section 3.1 can probably be further fleshed out given the extra page space in TMLR format. For instance, from equation (1) to equation (2), this is interesting and I think you can connect this step with the proofs of the appendix more clearly.\n\n- In Theorem 1, I wonder if it's possible to derive a bound in terms of the $\\ell_2$ norm of the gradient? Asking because from the infinity norm to the $\\ell_2$ norm one would incur an additional factor of $\\sqrt d$, which would trickle into the convergence rate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of preventing overfitting in fine-tuning, which is a common problem that practitioners encounter in the usage of large pretrained transformer models. Unlike prior work that often utilizes additional regularizations to prevent overfitting, this paper introduces a momentum filtering mechanism, called MoFO. The idea is to truncate the largest $\\alpha$ percentage of the momentum updates to zero before applying the update. This idea is similar to the shrinkage estimator in statistics where large weights are shrinked relatively to reduce overfitting.\n\nThe proposed algorithm is validated through both theoretical analysis of its convergence and empirical validations on several LLM fine-tuning experiments. The experiments clearly demonstrate that the proposed algorithm leads to a convergent minimum that stays close to the pretrained model, while delivering strong empirical performance compared with two momentum methods, i.e., Adam and Lion.\n\nOverall, I think the main idea of this paper meets the bar for acceptance to TMLR. I have several comments and suggestions below that I hope will further improve the quality of this work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "List of comments to address:\n\n- In the appendix, it would be useful to add a roadmap for quick reference to the proofs.\n\n- Some of the figures are a bit difficult to understand, and I think they could use some extra polish in a revision.\n\n    - For example, in Figure 1, it wasn't clear what the three axes are.\n\n- The proofs in the Appendix could use some extra polish in the revision. Some of the notations and expositions can be improved.\n\n    - In the proof of Proposition 1, page 21, what is the definition of $S'_k$ and $S''_k$? It would be good to clarify.\n\n    - At the top of page 22, first equation, the norms are lacking a right-side bracket.\n\n    - Appendix A.2, it would be helpful to recall the definitions of various notations from time to time so that would be easier to keep track.\n\n    - Lemma 2: The notation $\\alpha$% in C should be without the percentage symbol and instead define $\\alpha$ as the percentage value directly.\n\n- There are several earlier and recent references on distance-based regularization and regularization related to instruction fine-tuning. It would be good to add these to the related work discussion. For instance, it's plausible that the momentum filtering also leads to better stability of the loss surface (measured in terms of noise additions).\n\n    - Li, D. and Zhang, H., 2021. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems, 34, pp.27249-27262.\n\n    - Zhang, H.R., Li, D. and Ju, H., 2024. Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach. Transactions on Machine Learning Research.\n\n- I wonder how difficult it would be to extend the convergence analysis to nonsmooth functions. It would be useful to add a discussion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZWPY32W4By", "reviewer": "Reviewer_3rAc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(S1) A MoFo framework was proposed to only update the model parameters with the largest momentum magnitudes.\n\n(S2) The theoretical convergence result of the MoFO algorithm was derived.\n\n(S3) Experimental results demonstrated that MoFo achieved better performance than baselines in LLM fine-tuning.\n\nWeaknesses:\n\n(W1) One major concern is that the related works on parameter-efficient fine-tuning algorithms are not discussed. The parameter-efficient fine-tuning framework also aims to adjust a small number of parameters. It might be more convincing to explain how the proposed method outperforms parameter-efficient fine-tuning algorithms (e.g., LoRA, Adapters, BitFit etc.) in mitigating forgetting.\n\n(W2) The motivation that \"filtering the momentum is more effective than filtering the gradient\" is not well justified. Table 3 shows that Gradient-filtered BCD has comparable performance as MoFo. The benefits of MoFo can be further explained both theoretically and empirically.\n\n(W3) The forgetting mitigation in Subsection 3.2 involves the specifical cases of learning functions. It is more convincing to provide the theoretical analysis in general cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presented a novel Momentum-Filtered Optimizer (MoFO) framework for mitigating forgetting in LLM fine-tuning. Specifically, MoFo only updated the model parameters with the largest momentum magnitudes. The convergence of MoFo-based algorithm was analyzed. Experimental results demonstrated that MoFo achieved better performance than baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) The caption of Figure 2 states that \"Given that LLaMA2’s original training data is not publicly available, we use the RedPajama dataset (Computer, 2023) as a comparable alternative\". It is confusing why RedPajama is used here.\n\n(2) It states that \"Figure 2(b) indicates a strong negative correlation with MMLU accuracy\". However, it seems that MoFO involves a strong positive correlation. This phenomenon needs more explanations.\n\n(3) It is unclear how all model parameters can be partitioned into $B$ blocks in Algorithm 1.\n\n(4) The running efficiency of MoFo can be analyzed compared to baselines.", "broader_impact_concerns": "No concerns on the ethical implications of the work need to be addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)