PROFIT: A Specialized Optimizer for Deep Fine Tuning
Authors: Anirudh S Chakravarthy
Shuai Kyle Zheng
Xin Huang
Sachithra Hemachandra
Xiao Zhang
Yuning Chai
Zhao Chen
Venue: NeurIPS 2025 poster ()
Keywords: Temporal multi-task
deep learning
Primary area: deep_learning

Abstract: The fine-tuning of pre-trained models has become ubiquitous in generative AI, computer vision, and robotics. Although much attention has been paid to improving the efficiency of fine-tuning model, there has been less scholarship around fine-tuning specifically for improved model performance. To remedy this gap, we present PROFIT, one of the first optimizers designed to incrementally fine-tune converged models on new tasks and/or datasets. Unlike traditional optimizers such as SGD or Adam, which make minimal assumptions due to random initializations, PROFIT takes the properties of a converged model into account explicitly to regularize the optimization process. Employing a temporal gradient-orthogonalization process, PROFIT outperforms fine-tuning methods in various tasks, from image classification to multimodal language model training to large-scale motion prediction. Moreover, PROFIT is encapsulated as a modular optimizer, which makes it easy to integrate directly into any training pipeline with minimal engineering effort.

PROFIT: A Specialized Optimizer for Deep Fine Tuning
Anirudh S Chakravarthy
Shuai Kyle Zheng
Xin Huang
Sachithra Hemachandra
Xiao Zhang
Yuning Chai
Zhao Chen
GM Cruise LLC
Abstract
The fine-tuning of pre-trained models has become ubiquitous in generative AI, com- puter vision, and robotics. Although much attention has been paid to improving the efficiency of fine-tuning models, there has been less scholarship around fine-tuning specifically for improved model performance. To remedy this gap, we present PROFIT, one of the first optimizers designed to incrementally fine-tune converged models on new tasks and/or datasets. Unlike traditional optimizers such as SGD or Adam, which make minimal assumptions due random initializations, PROFIT takes the properties of a converged model into account explicitly to regularize the optimization process. Employing a temporal PROFIT outperforms fine-tuning methods in various tasks, from image classifi- cation multimodal language model training to large-scale motion prediction. Moreover, PROFIT encapsulated as a modular optimizer, which makes it easy to integrate directly into any training pipeline with minimal engineering effort.
process,
Introduction
Fine-tuning pre-trained models has become widely adopted in solving computer vision and robotics problems as well as in modern generative Al settings. As datasets and models increase in size, having train a new model for every new application and setting quickly becomes intractable. Imagine, for example, the cost of having to train a new model every time an autonomous vehicle needs to operate new city, or every time camera application needs to recognize new type of object, or every time custom LLM needs to update its knowledge cutoff. The shift towards fine-tuning within the deep learning community has also been accelerated by developments large foundational models that were trained on vast quantities of data, such CLIP Radford al. [2021). DINO Caron et al. [202T). and open-source LLMs Touvron et al. (2023a). Indeed, deep learning is steadily inching towards a new paradigm where very few practitioners are training models from scratch.
At the same time, fine-tuning model comes with its own set of challenges. For one, models are known to readily forget old information when fine-tuned with new information, in process known as catastrophic forgetting Goodfellow et al. (2013). Various mitigation methods have been proposed Li and Hoiem] [2017). but require data engineering and modifications to the model architecture. Common practice within fine-tuning still largely relies on training on new tasks/data with smaller learning rate or with frozen backbone (or both). Parameter-efficient fine-tuning methods such as LoRA Hu et al. 2022] introduce learnable adapters for transfer learning, but serve to improve fine-tuning efficiency rather than accuracy (Biderman et al. (2024). We aim to improve the accuracy of fine-tuning while keeping the process efficient by not introducing any additional parameters.
A ubiquitous object within deep learning that is modular and abstracted away from the practitioner is the optimizer. Many AI models set the optimizer to popular standards like Adam Kingma and Ba
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

6g
6
orthogenalie
PROximal Fine Tuning
O2
Standard Optimizer (line tuning)
PROFIT optimizer (fine tuning)
10
80
2 85
1.0
Figure 1: Schematic of PROFIT. Standard fine-tuning (middle) takes successive steps away from good starting state 0g. PROFIT (right): (1) take ner small reference steps with O(ref) to obtain a displaced state 01: (2) compute the displacement A = 01 0u: (3) orthogonalize the new-batch gradient 9 82 81 - -à (4) restore A < 8g and then apply the main optimizer O along the orthogonalized direction. Here, r denotes 'reference'. See Alg. [ for details.
2013], Adam Loshchilov and Hutter 2019], or Momentum Sutskever et al. [0133. However, all current optimizers are designed for training from scratch, they make minimal assumptions about problem setting and initial model state. contrast, fine-tuning setting usually starts from a well-trained, well-converged model that already performs well on some set of meaningful data. So we ask: how do we design an optimizer specifically to start from a converged model from similar domain? Given the ubiquity and modularity of optimizers within training pipelines, such an optimizer would be immediately applicable and easy to implement within any fine-tuning setting.
Works such Learning Without Forgetting (LWF) Li and Hoiem [017] have proposed mitigating catastrophic forgetting by enforcing proximity an old state of the model, but require additional data pipelining and model snapshots that serve as additional supervision to keep the model anchored to its old "good" state. Instead anchoring a model across tasks, we propose a different but equally valid anchoring across time rather than across lasks. For each iteration of the optimizer, we take step away from equilibrium and balance further steps away from equilibrium with the model's 's desire return equilibrium using techniques from the gradient-basee multi-task learning literature. The result is a system that mimics data-driver anchor methods such as Li and Hoiem [2017) without incurring any additional data processing, and which foregoes the rigid static anchors of more classic methods favor of a dynamically updated flexible one.
Specifically, model converged in some state 6g will proceed along the states 01... 6r with standard optimizer. further update would send the system to 0 However, because dn was a "good state." the model would also benefit by returning to Og (Sec. 3p. We thus have two potentially conflicting gradient directions (A := 0, >> On and g : 0, 1)+ which is a classic multitask learning problem. We borrow from Yu et al. (2020) and assign g + g J. A, where is the orthogonalization operator. We restore the model to state dg ("translate" operation on the top right of Figure and take step in the orthogonalized direction g.
The operation of PROFIT relies on some non-negligible distributional overlap between the fine-tuning and baseline settings, it aims to dynamically keep the model state close to the baseline state. This requirement precludes some settings that are common within fine-tuning like pre- pre-training and then fine-tuning on completely orthogonal datasets. We refer to our approach with this additional constraint as "proximal fine-tuning", where the fine-tuning dataset of similar distribution that of the pre-training dataset. Although this constraint may seem limiting, "proximal fine-tuning" is prevalent in machine learning applications. For example, a self-driving car's trajectory prediction network may need to train a prediction head for new scooter type using the same sensor, while

---

maintaining performance on old tasks. We will also later show how to overcome this constraint even non-proximal settings by introducing a warmup phase.
PROFIT (PROximal Fine Tuning) is shown schematically in Fig. 1 To the best of our knowledge, PROFIT is among the first optimizers explicitly designed for fine-tuning.
Our main contributions are as follows.
We introduce PROFIT, optimizer for fine-tuning converged models, easily integrated into any deep learning framework.
We show that PROFIT allows unsupervised training as if the original data were available.
We show that PROFIT outperforms standard fine-tuning methods various tasks, from image classification to VLM fine-tuning to large-scale motion prediction for autonomous driving.
2 Related work
Multi-Task Learning For detailed background context multi-task learning (MTL), we refer the reader to Zhang and Yang (02T]. MTL Zhang and Yang [2021] is an optimization problem in which we train a model on multiple tasks simultancously to take advantage of the structure of shared neural networks, thus improving generalization and efficiency. One direction is to use gradient descent methods to optimize the joint multi-task learning problem. Our setting shares some similarities with MTL, with the key difference that problem is temporal multi-tast learning. Ozan [Sener and Koltun [2018) formulates the MTL problem as a multi-objective optimization problem and learns the loss weights that change dynamically. GradNorm Chen al. [2018) attempts to normalize gradients to balance learning of multiple tasks. PCGrad Yu et al. [020] suggests that to mitigate gradient direction conflicts, we should project a task's gradient onto the normal plane of the gradient of any other task where a gradient conflict is present. Our proposed technique is similar to PCGrad, but modifies the core designs specifically for the "proximal fine-tuning" setting.
Parameter-efficient fine-tuning Parameter-efficient fine-tuning reduces computational and mem- ory requirements by updating fewer parameters, which simplifies the adaptation of large models to new tasks. An adapter Houusby et al. [2019] introduces light-weight learnable parameters to help transfer learning. The trainable parameters in the adapter are much smaller than in the model, which makes them attractive in practice. Several strategies have been proposed. such as visual prompt tuning Jia al. [2022). side adapters Zhang al. [2020). bias tuning Cai al. [2020). and residual adapters Rebulfi et al. [007a]] for efficient learning. LoRA Hu et al. [2022] decomposes the matrices of an attention module into a low-rank matrix. contrast to our method, adapters require additional parameters and may also assume a specific class of model architectures for their success (e.g., transformers).
Optimizer-based Fine-Tuning Fine-tuning on particular datasets Kornblith et al. [2019), Chen al. [2020] common technique in the era of deep learning. Practitioners often use standard optimizers such as Stochastic Gradient Descent (SGD) Bottou| [2010), Adam Kingma and Ba [2013]. and AdamW Loshchilov and Hutter 2019 Recent architectures such as ViT's Dosovitskiy al.
[2021]. Caron et al.] [2021] Radford et al. 2021 or ConvNeXts Liu et al. [2022 use Adam W fine-tuning, while is also common to use SGD for fine-tuning models like ResNets He al.] [2016]. Kolesnikov et al. [220) due to the optimizer's efficiency. However, SGD and Adam W do not assume that we want to stay close to our model's start state, and thus lead models that tend to compromise performance on old data when fine-tuning on new data, also known catastrophic forgetting McCloskey and Cohen 1989]. Our method serves as regularization approach to bridge the gap in existing optimizers mitigate this forgetting issue.
Continual Learning Various approaches have been developed Kirkpatrick et al. [017]] Chaudhry et al. (2020). J.ng et al. [2020). Titsias et al. (2020). Mirzadeh et al. [202]]. in which the goal is to keep the information learned from the past tasks during continual learning. Learning Without Forgetting (LWF) Li and Hoiem| 2017 stores the response of the old model on new tasks/data and supervises the new model on these responses using distillation to prevent catastrophic forgetting Masana et al. 2022). Our work draws inspiration from LWF's key idea of storing pre- pre-trained model's 's response,

---

Algorithm PROFIT: A fine-tuning optimizer
Require:
Converged model M(x; 0) with trainable weights 0 and input to be trained on data from similar domain X' with loss L.
Require: Initialize reference model weights O
Require:
Initialize batch size B, reference steps Ryef, training steps napps standard optimizer with learning rate Amain and reference optimizer O with learning rate Aref. Each optimizer takes as arguments the current weights and gradient update direction, producing updated weight values.
1: for nsstep steps do: Oet f-
2:
D Save the model state.
3:
for Rhet steps do
4:
Take new B examples from X' and calculate gradients g g= VoL. 5: Take one step with reference optimizer 0 (f, g).
6:
end for
7:
D Calculate total displacement during reference steps. Find g = VoL for a new batch in Line 4. Calculate dot product w (A,g).
Calculate = - Oref-
8:
9:
if then:
11:
g 1
a 1 b denotes orthogonalizing a with respect to
end if
13:
f- Oze-
b Restore original state. Take step with main optimizer.
4 (8,gg
15: end for
the network's response at the initial state of each iteration. Other directions for addressing catastrophic forgetting
but aim to avoid the storage of old data/checkpoinnsstatiiiisss Instead,
rehearsal- based methods Rebufi et al. (017b], Chaudhry et al. [2019a] Lopez-Paz and Ranzato
[2177] Chaudhry et al. [2019b]. (aaaa et al. [2022] that directly make use of the old data source, and
architectures Mallya and Lazebnik [2018), [Serrà et al. [2018) [i et al. [2019), Wortsman et al. [2020), Wu et al. 2019]] These methods add substantial infrastructural overhead, while our approach attractive in its simplicity. Since PROFIT does not require replay buffer or architectural changes, we primarily benchmark against optimizers and fine-tuning techniques that practitioners would otherwise use.
aechitecture-basee methods that minimize inter-task interference through
Method
3.1 The PROFIT Optimizer
Implementing PROFIT (Algorithm [ involves defining optimizer wrapper that takes two standard optimizers O and (ref) as inputs. The latter "reference" optimizer perturbs the system from equilibrium, while the former "main" optimizer uses this perturbation to make the final update. The entire logic of PROFIT can be encapsulated within the logic of this optimizer wrapper class, making the method very portable, modular, and invariant to the choice of "main" optimizer.
Given a model M with weights 0 trained on data X corresponding to an old task, and a data source X' from similar domain corresponding fine-tune M to work well on both old and new tasks. We make strict assumption about having converged model M available as input PROFIT: untrained weights will lead to poor performance. Furthermore, our approach also requires two optimizers, a standard optimizer O and reference optimizer O (ne), The user can tune O as per their needs, while we recommend using SGD for (ref)
new task (with task loss L). our objective to
A step through optimizer works follows. First, store the current state of M by saving 0 in 8iet- Next, we draw Noat batches from X', one at time, and iteratively minimize L with the reference optimizer (ref) We have now perturbed the system from equilibrium and must decide the best way to restore the said equilibrium.

---

To do so, we first calculate : O We reason that if the stored equilibrium state corresponds to a good critical point of the original model, then A corresponds to the benign gradient direction that will restore the original critical point, as gradient descent would send us in the - := Oet - 0 direction. We thus have two potentially conflicting gradient updates: the one corresponding to A, and the one corresponding to the next queried update by (teef, which we call g. We need to decide on how take a single gradient step that is consistent with both options.
:= 0 - the displacement vector after nott steps of the optimizer
We then borrow an idea from PCGrad Yu et al. [2020), which reconciled conflicting gradients by orthogonally projecting them onto each other in a pairwise fashion. Crucially, we choose to project only g onto A, and not the other way around, because A represents a gradient towards the old dataset that may no longer be accessible and therefore must be treated with more care. We end with the two gradient updates g and g 1. A, and take both steps by first restoring 0 6er and then allowing O to take a step in the g 1 A direction. This process is repeated until training is complete.
This formulation allows us to view fine-tuning temporal multi-task learning, with the two tasks being: (1) pretraining (A), and (2) fine-tuning (g). To the best of our knowledge, this the first time it has been viewed through this lens, and this insight may pave avenues for future research.
3.2 Theoretical Considerations
We present key theoretical properties of PROFIT with proof sketches. First, show that PROFIT is "correct"' by decreasing the loss on the old task/setting, We also discuss potential failure cases by identifying all stable points of PROFIT, arguing that these are not problematic in practice.
Theorem 3.1. (Correctness on old data) Take a model M(x; f) converged on data Xois with loss Lodd which we would now like to fine-tune on data Naow with loss Lnow- Suppose that ((re)) takes O O'. A single step of PROFIT in batch Xnew with sufficiently low learning rates Amain Deef will decrease Lotd( (Nat) from its value at 0'.
Proof. At convergence point local minimum Xo, the first-order gradient vanishes, and to leading order the loss surface will look like L(x) = L(xx) + 0.5(x - Ho,' - xn) for positive definite
hessian Ho The gradient within this region is therefore Lx( Rs Ho(x- xo) HA) Because
Ho is positive definite, we conclude that (-A)'Ho(A) < O, which means that moving the system in the A direction is valid gradient descent direction.
The prior theorem establishes that PROFIT accomplishes precisely what it seeks to do: even if the old data are no longer available, PROFIT allows us to train as if we can still compute the full loss function of the old data. As the system moves further away from the old equilibrium, we are able to restore some of the function of that equilibrium through these regularized updates. In particular, there is case where PROFIT leads to trivial update. Intuitively, since the model converged, moving away from the minimum creates a gradient (-A) that points back toward it. PROFIT leverages this implicit gradient to regularize the update on the new task.
Theorem 3.2. (Stable points) Suppose a model has weights O and O maps 0 + 0' and O(ref) is SGD. If we are not at a critical point of Tnow, PROFIT will result in zero change in the model weights 0 if and only if L V Lnew where V refers the unit vector corresponding V.
Proof. In this case A would point in the direction - Lnew at which point Lnen 1 A = 0, and the total update from PROFIT will be just a restoration 0. If the equality of the gradient does not hold, Vo F-eww
A + will lead to a non-zero total update.
Corollary 3.3. (Linearity forces a stable point) In the situation defined by Theorem 3.2 PROFIT will encounter stable point if the loss surface is perfectly linear between 0 and 0'.
The prior theorem and corollary demonstrate that PROFIT will fail to move the system when the loss surface becomes exactly locally linear at point. This almost never occurs for high nigh-dimensional loss surfaces that exist for deep models.
Theorem 3.4. (Convergence) For model trained with SGD Ouee), any slundard optimizer for O, and loss under PROFIT, the model is guaranteed to converge to either (1) a stable point of the system as defined by Theorem 3.2 or (2) convergence point of O.
5

---

Proof. If the system does not converge to a stable point as defined in Theorem 3.2 then it will necessarily produce a valid gradient descent direction because is an orthogonal projection of valid gradient direction VoL. As such, it inherits the same stable points as O.
In summary, PROFIT enjoys the following theoretical guarantees:
1. By Theorem 3.1
it mitigates catastrophic forgetting relative to prior methods.
2. By Theorem 3.2 it introduces only rare failure modes, which require all higher-order gradients to vanish.
3. By Theorem 3.4 it inherits the stable points of its parent optimizers, allowing the use of state-of-the-ar art and future optimizers.
3.3 Hyperparameter Discussion
PROFIT introduces three main hyperparameters for fine-tuning: Nhets Ofref, and Aya- Are controls the degree of exploration of the reference optimizer, while Ave controls the step-size at each iteration of the reference step. The choice O can be set by the practitioner according to their needs, but (reef) should be set to standard SGD, as the gradient calculations that drive PROFIT are the cleanest when the reference updates are simple.
A primary concern is the cost of setting Tief, it requires nhet additional optimization steps per training step. In practice, the practitioner is encouraged to start with Rie 1, which works well in practice, and to only increase nout if performance is lacking. However, we note two reasons why the potentially increased compute of PROFIT not large issue: (1) we find that PROFIT generally converges faster, possibly due to the positive regularization effects of the method, and (2) fine-tuning generally run for much fewer steps than training from scratch, so the effects of this additional training time are often still very tractable. For all experiments,
Thet = 1.
Mathematically, Age encodes how quickly we travel from equilibrium study the general shape of the loss surface, so that we can make an informed decision on where to go next. In practice, we find that there is often a sweet spot somewhere between Amain/ '10 and Amain/ /10000, but the exact value is highly dependent on the loss surface for the specific problem. In fact, it may turn out that the optimal value of Age tells us something about the fundamental properties of the loss surface of that particular setting, but such analysis falls outside the scope of the present work.
3.4 Memory Considerations
One of the primary difficulties in implementing PROFIT is that storing the reference weights Our as defined in Algorithm [ will increase the memory footprint. In memory profiling experiments (Appendix D.I). we found that a vanilla implementation of PROFIT will on average increase the memory overhead of model training by approximately 25%, which then decreases to 0% if remove the additional memory consumed by Oze-- In many fine-tuning scenarios, this additional memory consumption may not pose a problem, as model backbones may be largely frozen during fine-tuning. However, it is still worthwhile to address this substantial memory footprint.
In general, modern optimizers such as AdamW Loshchilov and Hutter [2019] keep reference statisties of model weights in memory which also incurs substantial memory penalties. It may be possible to implement memory-efficient approximate versions of PROFIT for most popular deep learning optimizers by using the already-ald alloccate extra memory within those optimizers to also perform the PROFIT computations. For example. such an efficient implementation for momentum might involve replacing 0er with 0 km for momentum vector m and some appropriate normalization constant k || - ||m] This implementation would reduce the memory footprint of PROFIT to single additional scalar (Il@ l) We treat these memory-efficient implementations of PROFIT as high-priority directions for future work.
3.5 Assumptions
The key assumptions made by PROFIT are:

---

Method
Original Data Erur (0)
New Data Error (4)
Baseline Trained Original Data
0.0054
1.907
Fine-Tune New Data (Pall Model)
0.705
0.504
Fine-Tune on New Data (Head Only)
0.110
0.572
PROFTT
0.046
Table 1: Results 2D toy problem. A baseline is trained on just the original data domain and then fine-tuned at a lower learning rate on a new data domain (both the full model and the head only). This compared to PROFIT, which trained the full model. Fine-tuning only provides some protection against performance regression on the original data, but also less able to adapt the new data. PROFIT outperforms both baselines and shows impressive resilience in maintaining performance on the original split. All results have standard error S 0.01.
the model head weights
1. Proximal Fine-tuning: pre-training and fine-tuning datasets come from similar distributions modalities.
2. Well-trained: the initial network is converged and not randomly initialized.
Orthogonalizing between and A implies that are interested in resolving the conflict between the original and fine-tuning datasets. If the original and fine-tuning distributions are significantly different, the orthogonalized gradient direction may lead destructive interference. Therefore, interpretation holds only if the original and tne-ttnnig datasets are from proximal distributions. Assumption may differ from typical fine-tuning literature and appear greatly limiting, but is exceedingly common in practice. For example, a self-driving car may train a classification head for a new scooter type while maintaining performance existing vehicles. Assumption 2 is required since our method is exclusively designed for fine-tuning (which a guiding principle). Nonetheless, we include here emphasize the reader that PROFIT cannot be used for general model optimization.
4 Experiments
We now detail a number of experiments for PROFIT in diverse settings: image classification, visual task adaptation, and large-scale motion prediction. We primarily focus comparisons standard fine-tuning with commonly used optimizers (on either the full model or just the model head), as those by large margin still the most commonly used their known performance and ease of implementation. We will show that PROFIT, while easy to implement, provides significant performance boost in all cases.
methods in the industry due to
4.1 A Simple Toy Example
We first apply PROFIT to simple, 2D regression problem with MLPs toy example. We choose the 2D function fxx =
)) as radially symmetric and periodic functions pose some challenge for neural networks models. We also add ('0 1) noise the output. Even though want fit to low-dimensional example, is still important for the problem to be difficult enough to see interesting behavior within models, especially given the dimensionality requirements discussed in Section 3.2 of the main paper. The original dataset consists of input-output pairs where the input coordinates drawn independently from [1[ 1.0,, 1.0), while the new dataset wish fine-tune has input coordinates drawn from (/.0.8 1.5). This is clearly also challenging because, although the domains of the two datasets overlap in the interval [0.8, 1.0). they are largely non-overlapping.
Our MLP consists of three layers with weights of shape [2, 500] + [500, 500] + [500, 1]. and we use RMSPPrp our baseline optimizer. The baseline model is trained the original data split only for 10000 steps at a learning rate le - 2, with fine-tuning runs trained at a learning rate of 5e for 1500 steps. For exact details on the training procedure, please refer the Appendix C.
The results are visualized in the bottom half of Figure]] with more granular visualizations in Appendix Visually, the benefits of PROFIT are pronounced; standard fine-tuning creates warped shape bearing minimal resemblance the original ground truth. PROFIT effectively remembers the original shape with only minimal regressions along the steep edges of the distribution.
7

---

Method
SGD
Adam
Lookahead
Adam
PROFIT
ResNet-18
24.49/77.59
34.64/33.11
26.10/73.00
77..70
35.26/74.70 74.70
35.26
ViI-Timy
53,63/61 98
56.00/ 58.99
35.64/61.33
53.04/61.89
58.53/62.22
ViT-Small
57.93/65.04
58.60/63.95
57.81 /65.27
53.85/ /4.09
59,02/ 65.44
Table 2: Image classification accuracy (%) CIFARIO/ / CIFAR 100 after fine-tuning on CIFAR100 Each cell shows (CIFAR1O/ / CIFAR (00). PROFIT achieves the best balance between transfer and forgetting. Compared methods include Adam (Kingma and Ba [2015), Lookahead (Zhang et al. [2019)) and Adan (Xie et al. 2024)).
Natural
Specialized
Stractund
Canshhon
Retisspaly
Cleys Cuzes
Clev-DDe
KITTL DiA
XNORB Abii
SNORB El:
DTD
ETowes
SWINN
Son.99
400ml
NON
68.9
K7.7
64.3
K7.2
17.4
38.8
79.7
95.2
73.9
56.3
65.5
57.5
46.7
25.7
29.1
64.4
65.0
63.2
97.0
16.3
36.6
51.0
78.5
87.5
68.5
26.0
54.3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
VPT 1a
78.
90.8
65.3
96.0
18.3
7X.
1966
61.8
W6.1
k34
18.8
65.5
00.9
46.5
72.8
736
479
32.9
37.8
PROFTT
589
55.6
51.4
92 8
90.
19.6
37.7
79.3
538
43.5
73.5
12.8
37.8
21.1
77.
16.2
286
23.6
16.7
PROFIT (wartsup)
07.4
90.5
60.6
98.9
$9.3
$9.4
$2.0
84.9
95.2
85.6
74.3
88.5
64.7
51.9
80.6
8233
48.3
32.2
35.1
Table Image classification accuracy on VTAB-IK. Bold best, Underline second-best. Ties for best are all bolded; no second-best is shown in such cases. We compare PROFIT against standard fine-tuning strategies, including full fine-tuning and linear fine-tuning (denoted by Full and Linear). A naive PROFIT violates the proximity condition and performs poorly, but with Adam warm-up it achieves consistently higher accuracy.
Numerically, as shown in Table I after fine-tuning with different techniques, it is clear that PROFIT outperforms the baseline in not forgetting the original dataset distribution. Even though PROFIT modifies all model weights, it mitigates forgetting relative to the head-only fine-tuning baseline by a sizable margin, even though PROFIT acts on significantly more weights. Fine-tuning of the full model leads to disastrous results, with significant deformations of the predictions. PROFIT allows us the flexibility of mull-model fine-tuning without the drawbacks of catastrophic forgetting.
4.2 Image Classification
Next, we demonstrate the effectiveness of PROFIT for image classification. CIFARIO and CIFARION 100 (Krizhevsky and Hinton| (209]) ) are the facto benchmarks for image classification. The dataset consists of 32 x 32 images and the task is to classify image into one of K categories.
We first train a network on CIFARIO (pre-training) and fine-tune the network on CIFARI00. As discussed, PROFIT assumes proximality, i.c., both the pre-training and fine-tuning datasets need be from a similar domain distribution. This assumption is met for our choice of CIFARIO and CIFAR100, as they are both labeled subsets of Tiny Images Torralba et al. 2008
We experiment with various backbones, including R8 (He al. (2166), ViT-Tiny, and "TT-Small((Dooviiski et al. (2021), and compare PROFIT against popular optimizers like SGD, Adam (Kingma and Ba 2015 Lookahead (Zhang al. 2019 ), and Adan (Xie et al. (2024)
Table shows that PROFIT outperforms standard fine-tuning across all backbones and optimizers. CIFARIO accuracy after fine-tuning is also higher, indicating better retention of the original task. For example, with ViT-Tiny, Adam fine-tuning results in 55.64% (CIFAR10) and 61.35% (CIFAR100), compared to PROFIT's 58.53% and 62. 62.20%. Similarly. fine-tuning ViT-Small using Adam yields 58.60% (CIFAR10) and 63.93% (CIFAR100), while PROFIT achieves 59.02% and 65.44%.
4.3 Visual Task Adaptation Benchmark
The VTAB- 1K Zhai al. [2019] dataset is popular representation learning benchmark that evaluates generalization in 19 diverse classification tasks. It aims to learn feature representations effective for all tasks, with performance measured by classification accuracy on each.
We use ViT-Base Jia et al. [2022) with ImageNet pretraining as the backbone for our experiments. This setting is example of setting in which the original (ImageNet) and fine-tuning distributions
8

---

Backlene
Opimina
Auarnyy
March (oo
ROU0F44
Firal
Admm PROFIT
70
96.19
0.541
7.66
W
72.12
37.37
0.557
7.28
59.16
Table 4: Results on the DriveLM Benchmark. We fine-tune LLaMA-Adapter-e Gao et al. 2023 and compare PROFIT with Adam W, the de-facto method for fine-tuning VLMs. Our method shows improvements on all metrics, showcasing the applicability of PROFIT to large foundational models.
(VTAB-IK) differ significantly, violating Assumption I (Sec. 3.5 Although our method is not well tuned for these fine-tuning settings, this situation is common practice.
First, we show our results in Table and compare our method with Adam W Loshchilov and Hutter [219)] fine-tuning (Full - row 1) as well as final layer fine-tuning (Linear row 2). Full fine-tuning using PROFIT does not work well, showing poor performance across all 19 tasks in the benchmark. For example. PROFIT achieves 12.6% accuracy on the Clevr-Count benchmark while Visual Prompt Tuning Jia et al. [022] achieves 68.5% accuracy.
However, we provide a training recipe on how PROFIT can be used in such settings. First, we fine-tune the model towards the target distribution with an AdamW warmup for 10 epochs, and then apply PROFIT to fine-tune for the remaining 90 more epochs (PROFIT (warmup)). Intuitively, this moves the model's distribution somewhere between the pre-training and fine-tuning distribution, which makes it amenable to the structure of PROFIT. This approach outperforms the full model fine-tuning with Ad So, while our method as expected is not recommended for non-proximal settings (as discussed in Section 3.5 using a short warm-up phase with another optimizer allows PROFIT to substantially outperform full fine-tuning with standard optimizers.
4.4
Multimodal Vision-Language Models (VLM)
VLMs (Hwang et al. 2025 trained on web-scale data have helped to improve generalization of end-to-end driving systems. DriveLM (Sima et al. (2023]) is a visual question-answeriig (VQA) benchmark to evaluate VLMs autonomous driving. The DriveLM-nuScent
dataset contains prompts which probe perception, prediction, and planning capabilities respectively.
sensor inputs and text prompt, and the network aims to provide accurate textual responses for
We use a pre-trained LLMMA-Aaapeer--v (Gao et al. (2023), D. which performs bias tuning on LLaMA- 7B (Touvron et al.. (2023b)) We compare PROFIT with AdamW (Loshchilov and Hutter (2019)) the de facto method for fine-tuning VLMs. We report results on the validation set in Table PROFIT improves accuracy over AdamW by 5.6%, GPT score by 1.5%, Match Score by 1 1,, and the Final Score by 2%. addition, PROFIT provides better results for VQA, as seen by improved performance metrics (BLEU-4, ROUGE-L, CIDEr). For further explanation on metrics, we refer the reader to the Sec. C.4 of the appendix.
4.5 Large-Scale Roboties Motion Prediction
We evaluate PROFIT the Waymo Open Motion Dataset (WOMD) Ettinger et al. [022] large- scale driving dataset. The task is to predict agent trajectories over 8 seconds using multi mulli--oddl observations from the last second, including agent histories, map data, and traffic light states. Our model builds on the state-of-the-ard Wayformer Nayakanti et al. [2022) by fusing multi-modal inputs with self-attention transformer and predicting future trajectories using learned latent queries.
We first train a model on car trajectory prediction, and then fine-tune the model on data from the same class (car) well as different classes (pedestrian). For baselines, use the AdamW Loshchilov and Hutter 2019] optimizer. WOMD allows us to evaluate PROFIT on (I) fine-tuning on the same data and (2) different domain-shifics tasks. For example, CIFAR100 is similar to CIFARIO, whereas pedestrian trajectories differ from car trajectories.
Figure and 5] show the results. Average Distance Error (ADE) measures the mean distance between ground truth and predictions at each point, while Final Distance Error (FDE) assesses only the final point. PROFIT consistently outperforms the baseline, especially in car-to-pedestrian fine- tuning. Fine-tuning the head alone performs poorly on car-to-pedestrian tasks, and full fine-tuning better, but still inferior to PROFIT. Thus, PROFIT effectively repurposes models for related settings.

---

Baseline Fine Tune
WOOD manFBE
WOMD MINDDE
134
The Tirk PO0e)
Only
034
IROFIT
125
1224
8500
044
2.22
322
644
5:2
343
PROFIT
WOOD MEI5EE
1.7
NHe kee H&L Mhor
0.30
Tamn PuB Made
1.9
1.2
699 2200
(a)
(b)
Figure 2: Results for PROFIT on Waymo Open Motion Dataset. (a) Training error curves for FDE at 3s and 8s (top) and ADE at 3s and (bottom). PROFIT outperforms both fine-tuning baselines by sizable margin. (b) Visualizations of motion prediction outputs for both the baseline fine-tune model (top) and PROFIT (bottom). Trajectory ground truth is shown as a shaded bar and denser lines represent more confident predictions. PROFIT (bottom) produces more confident predictions that align better with the ground truth (shaded bar) compared to the baseline (top). Best viewed in color.
Method
Target Class
ADE@3s
ADE (m)
FDE@3s
FDE@8s
Baseline
1.024
2.581
fine tune (F)
Car
0.458
1.322
1.021
2.548
fine-tune
Car
0.456
1.303
1.009
2.507
PROFIT
Car
0.454
1.299
1.008
2,489
tine-tune (F)
Ped
0,214
0.621
0.465
1.242
fine tune (II)
Ped
0.232
0.724
0.508
1.544
PROFIT
Ped
0.203
0.579
0.427
1.145
Table 5: Results on Waymo Open Motion Dataset. F stands for full-model fine-tuning and H stands for head (last layer) only. Standard errors are within 0.005m for 38 metrics and 0.015m for 8s metrics. There is minor but noticeable improvement for PROFIT on the car-to-car benchmarks and a substantial improvement for PROFIT on the car-to-ped benchmarks.
We also minor improvements in the car-to-car benchmark, suggesting that PROFIT can be used to extract more performance from any model that has already converged. We could even imagine scenario in which additional training using PROFIT is standard model maintenance practice.
Conclusion
We introduce PROFIT, an optimizer that improves robustness against catastrophic forgetting during fine-tuning by using confidence in the model's prior converged state as regularizer. PROFIT excels in various settings: (1) fine-tuning to CIFAR 100 from CIFAR 10, (2) fine-tuning on new distribution in VTAB-IK by providing train recipe, (3) fine-tuning large Vision- Lasiuu-L Models for autonomous driving and question-ansween answering, and (4) fine-tuning on both new and identical tasks large-scale motion prediction. In all cases, PROFIT outperformed standard fine-tuning methods. Importantly, the modularity of PROFIT as an optimizer allows it to integrate easily into training pipelines. We believe PROFIT is a valuable tool for practitioners and encourages the development of new optimizers that support fine-tuning the primary deep learning paradigm.
10

---

References
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sébastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese Tarun Gogineni, Adam P. Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei. Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park. Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev. Max Schwarzer, D. Sculley, Harshit Sikchi. Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet. Sam Toizer, Foivos Tsimpourlas. Nikhil Vyas, Eric Wallace, Xin Wang. Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedrie Whitney, Hannah Wong, Lin Yang. Yu Yang. Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card. CoRR, abs/2508 10925, 2025. doi: 10.48550/ARXIV.5508.. 10925. URL https://doi. http:////oooooogg///// 10. 48550/ arviv 2508 10925
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey F. Hinton, Layer normalization. In International Conference on Learning Representations (ICLR), 2016.
Dan Biderman, Jacob P. Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA Learns I Less and Forgets Less. Trans. Mach. Learn. Res., 2024.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, pages 177-186, 2010.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 11285-1 11297, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Leurning Representations (ICLR), 2019a.
Arslan Chaudbry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. In International Conference on Learning Representations (ICLR), 2019b.
Arslan Chaudhry, Naeemullah Khan, Puneet K. Dokania, and Philip H. S. Torr. Continual learning in low-rank orthogonal subspaces. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9900-9911, 2020.
Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv-adapter: Exploring parameter efficient transfer learning for convnets. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pages 1551-1561, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020.
11

---

Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing deep multitask networks. In Proceedings of the International Conference Machine Learning (ICML., 2018.
DriveLM conttributoss. Drivelm: Driving with graph visual question answering. ittps://githul canm/DpeaDrrivellaa//ziivvel 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ils.. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshomn Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller. Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olanee Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak. Ke Li, Kenneth Hcafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/2407.21783. 2024.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R. Qi, Yin Zhou, Zoey Yang, Aurelien Chouard. Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 97 10- 9719, 2021.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parfmeter-effficien visual instruction model. arXiv preprint ar arXiv:2304.150 15010, 2023. httpsstss///axxi org/
abs abs/2304.115 15010
Ian Goodfellow, Mchdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 131iv 2013.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR), 2022.
12

---

Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji. Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp. Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving. Trans. Mach. Learn. Res., 2025.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 709-727, 2022.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 3647-3658, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
James Kirkpatrick. Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114(13)33521-355 -35266, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Simon Korbblith, Jonathon Shlens, and Quoe V. Le. Do better imagenct models transfer better? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Univ. Toronto, 2009.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Learning (ICML), pages 3925-3934. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935 2947, 2017.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 71- 2004.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Infommaion Processing Systems (NeurIPS), 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7765-7773, 2018.
Marc Masana. Xialei Liu, Bartiomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classifica- tion. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Academic Press, 1989.
13

---

Seyed- Iman Mirzadeh, Mehrdad Farajtabar, Dilan Görür, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. International Conference on Learning Representations (ICLR), 2021.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2980-2993 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 511-318, 2002.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017a.
Sylvestre-Alvin Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2001-2010, 2017b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the JEEF/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2022.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations (ICLR), 2022.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Joan Serrà, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning (ICML), pages 4548-4555 PMLR, 2018.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens BeiBwenger, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with graph visual question answering. In Proceedings of the European Conference Computer Vision (ECCV), 2023.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In Proceedings of the International Conference Machine Learning (ICML), pages 1139-1147 147. PMLR, 2013.
Michalis Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations (ICLR), 2020.
Antonio Torralba, Robert Fergus, and William T. Freeman. 80 million tiny images: large data set for nonparametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 30(11), 2008
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arX19:2302.13977, 2023a.
14

---

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, liyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models. CoRR, 2023b.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus- image description evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566 2015.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 15173-15184, 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Trans. Puttern Anal. Mach. Intell., 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/1910.04867, 2019.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 698-714, 2020.
Michael R. Zhang, James Lucas, Jimmy Ba, and Geoffrey E. Hinton. Lookahead optimizer: k steps forward, step back. In Advances in Neural Information Processing Systems (NeurIPS). volume 32, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: open pre-trained transformer language models. In arXiv: preprint 2205.01068. 2022.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586-569, 2021.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. IEEE Trans. Pattern Anal. Mach. Intell., 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The authors believe that the paper's contributions and scope are accurately presented the introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discussed the limitations of the proposed method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

The authors have tried their best to provide the full set of assumptions and complete and correct proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The authors tried their best to provide the full information needed to reproduce the main experimental results of the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The submission provided sufficient details for other to reproduce the algorithm, but the code is not immediately released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ Staeeees//ooeeSSSeiaiiiiiioiioi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//bdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper reused the popular benchmarks to validate the claims, and they all publicly available.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The authors have tried their best to report the statistically significant results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The authors provided the sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this paper follows the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper provides new optimization algorithm that can help train the deep learning model more efficiently, and whether it has positive or negative societal impacts depends on the application, and this paper has not tied to particular application.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in this paper are cited and referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper has provided sufficient discussion so that authors can reproduce the idea on their own, and the paper does not release new assets on it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This is research paper focusing the validation of new optimization method, and has nothing to do with crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This is a research paper focusing on the validation of new optimization method, and it has nothing to do with crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
22

---

A Appendix: Introduction
In the appendix, we provide the following:
A theoretical discussion of PROFIT (Sec. B
Detailed setup, hyper-parameters, and ablations (Sec.
Limitations of PROFIT (Sec. D).
B Additional Theoretical Discussion
Here, we provide a bit more exposition on the theoretical properties of PROFIT. In the main text (Section 3), we proposed two properties of PROFIT: (1) that PROFIT updates will reduce the loss value of the old loss on the old data, despite making no assumptions on access to the old data, and (2) that PROFIT has stable points on linear loss surfaces.
The implications of statement (I) are fairly straightforward, as it implies that we can train with the settings of the old system even when that old system falls out of scope. This the primary feature of PROFIT and is the main proof that PROFIT is a meaningful regularization method. But the implications of (2) are more interesting. The fact that PROFIT works despite not functioning properly on linear loss surfaces implies that PROFIT relies on nontrivial values of second-orde gradients and curvature within the loss surface to function. Thus, any critical point which the system converges while under PROFIT updates must have been reached through nonlinear path from the model's s starting point. This requirement may have robustness implications for the convergence points found by PROFIT, as any such convergence points must have alternative nonlinear paths leading to it.
However, there is an important discussion be had regarding the convergence of PROFIT. We did not expand on this convergence in the main text because talking about convergence within a fine-tuning setting is potentially ambiguous, as is difficult to deconvolve convergence on the new dataset with convergence on the old setting (for which we may no longer have available data). Especially when the number of incremental training stages increases, it becomes secondary converge on the new (potentially small) dataset on which the model is fit, and more important to maintain the efficacy of the base model. In that case, the correctness property (1) becomes the main property of importance.
In general, a proof of convergence of PROFIT is difficult for two reasons: (1) The orthogonalization procedure we use is asymmetrie versus the symmetric procedure in Yu al. [2020). due our prioritization of mitigating regressions, and (2) the restoration step to the old state after steps of (reff) is discrete rather than treated as a separate incremental gradient step. Both of these design decisions were made support the main goal of regression mitigation for applications of PROFIT. So, although we do not provide a complete proof of conditions under which PROFIT converges in this work, we make this omission precisely because convergence the new data is secondary goal in our setting, and the majority of our important design decisions were not made in service to that particular goal, but rather to the more difficult goal of regularization during fine-tuning.
C Experiments
C.1 A Simple Toy Example
We provide more granular visualizations of toy example results in Figure 3 fine-tuning on the head layer only produces sizable shifts in the overall height of the output shape. while PROFIT more effectively remembers the original shape.
We note that
Our toy example ground truth is the function f(x) = sin(10[xl) with input in R2. This function was chosen because of its extreme nonlinearity and difficulty in fitting by standard neural networks. To further increase the challenge, for the training data, normal noise of size A0, 1) is added, while no noise is added to the test data. The "original dataset" consists of 50000 points with both dimensions between -I and 1, while the "fine-tune dataset" consists of 50000 points with both dimensions between 0.8 and 1.5.
The model itself is a 3-layer MI.P, consisting of weight layers [2, 500], [500, 500], and [500, 1]. LayerNorm (Ba et al. (0166) is applied after every layer except for the last. RMSProp is used with
23

---

Original Dataset
New Dataset
Full Dataset
Ground Truth
03
14
on Original Only
Tune, Head Only
Fine Tune, Model
Tune,
Model
Model Predictions
15
Figure 3: Toy example visualizations. The top row shows ground truth distributions for the original, new, and combined datascts, while the bottom row depicts model predictions for different training strategies: training on original data only, head-only fine-tuning, full model fine-tuning, and using the proposed PROFIT optimizer for full model fine-tuning. This highlights the PROFIT optimizer's effectiveness in retaining old task knowledge while adapting to new tasks. Best viewed color.
Backbone
CIFARIO Accuracy
ResNet18
Tiy ViT-Small
82.38%
86%
Table 6: Accuracy of pre-trained models the CIFARIO dataset (before fine-tuning).
default PyTorch hyperparameters (a = 0.99, c le 8) and the learning rate 1e 2 for fitting to the original distribution, with a learning rate decay of 0.9 every 500 steps. After fitting the original distribution, we fine-tune the new distribution for 1500 steps at a learning rate of 4, with decay factor of 0.95 every 100 steps. PROFIT is with nre 1.
C.2 Image Classification
C.2.1 Pre-trained model performance
We show the performance of the pre-trained backbone on CIFARIO datasets in Table ResNet-18 achieves an accuracy of 91.06%, ViT-Tiny achieves 82.38% and ViT-Small achieves 83.86% accuracy.
C.2.2 Comparison to Adapters
Adapters provide an efficient mechanism to fine-tune large pre-trained models with a fraction of trainable parameters. Although PROFIT does not introduce any additional parameters, compare our method with commonly used parameter-efiie
efffcciett fine-tuning methods in Table
Despite using fewer trainable parameters, our method marginally outperforms LoRA Hu et al. 2022 and VPT Jia et al. [2022] for ViT backbones and Conv Adapter (Cen et al. [2024] for ResNet. Furthermore, our method is complementary to adapters and shows improvements when used as the optimizer for training adapter-based methods.
C.2.3 Ablation Study
We ablate Whet Table general, we observe that increasing Ther leads to better performance in the original task while compromising performance in the fine-tuning task. This follows from our discussion in See 3.2 of the main paper, since The controls the degree of exploration away from the stable point. However, we recommend Ree starting point.

---

Backbone
Adapter
Optimizer
CIFARIOO Accuracy
ResNet18
PROFIT
74.70%
ConvAdapter
Chen et al.
2024
Adam
74.81%
ConvAdapter
Chen
PROFIT
75.26%
VIT-Tiny
PROFIT
62.20%
LORA
Hu CL a
2022
Adam
61.02%
LORA
PROFIT
61.55%
VPT
ma el 11
0122
Adam
60.63%
VPT
21727
PROFIT
61.24%
ViTll
PROFIT
65.44%
LORA
PROFIT 65.02%
VPT
Ja et
al
Adam
63.98%
VPT
PROFIT
64.57%
Table 7: CIFAR- 100 accuracy for different backbones, adapters, and optimizers
Method
Pett
CIFARIO Acc (1)
CIFAR100 Acc (1)
ResNet-18 18
1
35.26%
74.70%
2
32.55%
73.57%
5
39.27%
71.42%
ViT-Tiny
56.75%
62.35 %
2
53.10%
61.67%
VIT-Small
$9.02%
65.44%
58.58%
64.91%
56.54%
63.75%
Table Ablation study on Tieer for image classification for different backbones. The results demon- strate that varying nhet can have significant impact model accuracy when using PROFIT.
We also ablate performance on Amn in Table which shows that the best choice may vary for the choice of backbone. In general, larger values Aye promote new task accuracy, while smaller values effectively mitigate catastrophic forgetting. These results are reasonable, as smaller values of Xe correspond to the reference point lying closer to the original model state. However, we were always able to beat the baseline on both old and new task accuracies simultaneously.
Aut
We also use this setting substantiate Assumption 2 (Sec 3.4) which states PROFIT does not work well without a converged model. When we train a ResNet-18 from scratch on CIFAR100 using PROFIT, we get 1.05% accuracy, which is good as random guessing.
C.2.4 Regularization effects
We plot the training and validation losses for CIFARI00 fine-tuning in Figure We observe similar trends for all network choices, where the validation losses illustrate that PROFIT is able to achieve better generalization across both the pre-training (CIFARIO) and fine-tuning (CIFARIO0) tasks. Such results indicate that PROFIT may have positive regularization effects during the course of fine-tuning, allowing the network to converge quickly while achieving better performance.
C.2.5 Implementation Details
For CIFAR10 pre-training, we use Adam optimizer with a learning rate of 1e 4. ViT-Tiny and ViT-Small are trained for 400 epochs, while ResNet-18 is trained for 200 epochs. For the Lookahead optimizer, use 0.5 and 1: for fair comparison against our method which uses neet 1. We perform a parameter sweep to obtain the best performance for each method on CIFAR100 fine-tuning, and list the best obtained syper-parameet in Table T0
C.2.6 Memory Footprint
As highlighted Section 3.4 in the main paper, one of the main limitations our work is the additional memory consumed. We use 4 Tesla T4 GPUs for all our experiments and quantify the increase in GPU memory consumption and train time by PROFIT in Table Fine-tuning ResNet- 18 with PROFIT consumes additional 2.23 GB of GPU memory, while fine-tuning ViT-Tiny and
25

---

Method
Amin Avaf
CIFARIO (1)
CIFAR 100 (1)
ResNet-18
10
35.26%
74.70%
100
35.16%
74.45%
74.27%
10000
35.83%
74.41%
Vir-iny
10
57.32%
61.99%
100
56.75%
62.35%
58.53%
62.20%
10000
53.78%
62.05%
VIT-Small
10
58.41%
66.29%
100
59.02 %
65.44%
1000
56.96%
65.53%
10000
56.84%
65, 55.95%
Table 9: Ablation study for PROFIT in image classification for different choices of backbone
Aut
architectures. For ResNet-18, 18, smaller values of Aman
yield relatively stable accuracy, while ViT-based models show a more pronounced change, with peak performance observed at specific ratios. This result highlights the importance of fine-tuning Awia to achieve optimal performance for different Aut
'vd
backbone architectures in image classification tasks.
train, loss
vall, cwfarr0, loss
loss
2500
2000
200
1500
155
1000
0.5
900
50
Step
100
200
303
300
100
200
300
Figure 4: Training and validation losses for fine-tuning ViT-Small on CIFARIO0.
ViT-Small consumes 1.3 GB and 1.1 GB extra. This increase is consequence of loading the reference optimizer states in memory. However, our method does not introduce any new learnable parameters.
C.3 Large-Scale Robotics Motion Prediction
The motion prediction model follows a standard encoder-decoder transformer architecture, as in Wayformer Nayakanti et al. [202].
The encoder takes multi-modal inputs the target agent's history, nearby agent histories, map information, and traffic light states. Each input modality is encoded by a separate MLP to an embedding with dimension of 64. The input embeddings fused through concatenation input tokens to a self-attentin transformer. The encoder transformer includes 2 attention layers, 8 heads, 256 hidden dimensions, and 1024 feedforward dimensions. We add learned positional embeddings, initialized as a Gaussian vector with zero mean and standard deviation of 0.02, to each token.
The decoder is cross-attention transformer that attends six learnable latent queries. initialized with zero mean and standard deviation of 0.02, to encoder embeddings. The decoder transformer includes 8 attention layers, 8 heads, 256 hidden dimensions, and 1024 feedforward dimensions. The output queries are mapped to a weighted set of six trajectory samples through an MLP. Each sample includes (x. positions for the next 80 timesteps and weight scalar.
The model is trained end-to-end by a smooth L1 loss on trajectory predictions and a cross-entropy loss on the predicted weights. The Adam optimizer is used with default Py Torch learning rate 1e - 3, /8 = (0.9, 0.999), weight decay le The base model is trained on the WOMD training set for 60 epochs with batch size of 256.
26

---

Method
Optimizer
Learning Rate
Epuchs
ResNet-18
Adam
lo-4
200
Lookahead
le-4
100
PROFIT
le.4
100
ViT-Tiny
Lookahead
le.4
200
PROFIT
le-4
200
ViT-Small
Adam
3c-4
400
Lookahead
3e-4
PROFTT
3e.4
200
Table 10:
for CIFARTOO results (Sec 4.1 in the main paper).
Method
Optimizer
Time (sec epochh
GPU: Memory (GB)
ResNet-18 18
Adan PROFIT
185
5.50
248
7.73
VIT-Tiny
135
4.45
PROFIT
185
5.86
VIT-Small
Adam PROFIT
198
4.17
269
5.28
Table 11: Training time and GPU Memory utilization of different methods for CIFAR100 fin-ttuning.
For car-to-car fine-tuning experiments, learning rate is dropped by factor of 100 from the original and training is performed for only 1500 steps (because original training run already converged, training for too long in this setting leads to overfitting). For aa-tt-peedestrian fine-tuning experiments, learning rate also is dropped by a factor of 100, but training is allowed to run for the same number of steps as the original model.
For PROFIT, Arr set to 3 and Ayel set to 1/10 of the learning rate. We note that the value of nee is quite high in this setting, but training is relatively fast and you can get better results fairly early on in training so the number of steps can be cut down considerably (see, for example, the curves in Figure of the main paper).
C.4 Multimodal Vision Language Models
C.4.1 Ablation Study
The DriveLM benchmark leverages variety of VQA metrics for evaluation. BLEU Papineni et al. (002] evaluates precision by measuring n-gram similarity between generate and reference texts. ROUGE-L [2004) evaluates recall by measuring the longest common subsequence between the generated and reference texts. CIDEr Vedantam et al. [0055] evaluates quality by computing the cosine similarity between the n-gram TF-IFF vectors for the generated and reference sentences. GPT Score aims to captures semantic nuances missed by aforementioned metrics by using ChatGPT (GPT-3.5-Turb0) as an evaluator. using the ground-truie objects in the scene, while prediction accuracy is evaluated over discretized future states. Match score evaluates whether the VLM correctly understands the order in which to attend to other agents in the scene. The final score for this benchmark is weighted average of these metries. For further details, we refer the reader to the evaluation criteria for the DriveLM Challenge contributors [023].
addition to these metrics, perception accuracy is evaluated
We provide inference visualization of our method in Figure 5 The baseline, fine-tuned with AdamW, is unable to detect the traffic light and black sedan in the scene. This leads to the suggested plan of running the red light, which is potentially catastrophic error. On the other hand, PROFIT successfully identifies the red light and follows traffic regulations by planning to remain stationary. We conclude PROFIT useful tool in the era of VLMs extract better performance on new setting.
We ablate performance on Ami key hyper-paramest to PROFIT, Table In general, larger values encourage exploration and improves fine-tuning task accuracy, as discussed See 4.3 of the main paper.
27

---

Baseline
Metthod
Hacurpllet imporatt expedtin in Phy
hore rigo waricle
taffs light of
Prediction wixt folt sttu
Turning
Paaniigg the ago volide nhd?
Berna coltraaty
Figure 5: We compare the baseline with PROFIT on an example from DriveLM. The model fine-tuned with PROFIT is able to perceive the traffic light and black sedan in the scene, while the baseline (fine-tuned with AdamW) does not detect the traffic light and hallucinates the presence white truck. Consequently, the baseline suggests running the red light, while method follows traffic rules by staying stationary. Best viewed in color.
Anais Ard
Accuracy (%) f
GPT Score (%)
Match Score (%)
BLEU-4
ROUGE-L 1
CIDEr 1
Final Score 1
10
71.99
37.21
0.550
0.029
100
65.83
72.19
37.16
0.550
7.22
0.029
58.64
1000
65.74
72.07
37.02
0.550
7.22
0.030
58.54
10000
67.88
72.12
37.37
0.557
7.28
0.035
59.16
Table 12: Ablation study on Aum for PROFIT on the DriveIM Benchmark. Higher values of Anin Ayt
improves perception and prediction accuracy and language scores, showcasing the strength of PROFIT in generalizing both VQA and scene understanding tasks.
C.4.2 Implementation Details
We closely follow the training recipe from DriveLM contributors [2023]. For Adam we use a learning rate of le to fine-tune. PROFIT uses learning rate of le 2 for fine-tuning, while we set A to 1/ 1/10000 the learning rate and nyef = 1. We use H100 GPUs for experiments.
C.5 Visual Task Adaptation Benchmark
The Visual Task Adaptation Benchmark (VTAB-IK) Zhai et al. [2019) is popular representation learning benchmark to evaluate generalization across diverse set of image classification tasks. The 19 datasets can be grouped into categories: Natural, Specialized and Structured groups. Each dataset contains 800 training examples and 200 validation examples. The domains for each dataset vary significantly, making a challenging benchmark for PROFIT.
C.5.1 Implementation Details
We follow the fine-tuning hyper-parametent backbones, and classification heads parameters using the same setting VPT Jia al. 2022). For experiments using PROFIT, we fine-tune the model with an initial learning rate of 0.01 and employ a cosine decay learning rate schedule, consistent with
the approach in Jia al. 20022. We set Apef to one-tenth of the learning rate and use nlytf 1.
we conducted PROFIT experiments in two different directions. One is to apply the PROFIT directly to fine-tune the models the VTAB- KK dataset, this is denoted as "PROFIT w/o warmup" which yields poor performance, because the model starting point is poorly optimized for the target datasets. We also conducted another experiment, The "warmup" is conducted using the same Adam W optimizer and the same parameters as the FULL VPT described in the table 6 in Jia et al. Jia et al. [2022).
As shown in Table 13
We also include in Table 13 comparisons against LoRA and NOAH, which are both popular current fine-tuning methods that allow for efficient fine-tuning. These methods both add additional trainable
28

---

Natural
Specialized
Structured
Cifer
Caltch
Flonet
SVHN
Sun
Camelyn
EUNSAT
Rodiee5
Retinopathy
Clove Clov Count
ClorrDis
DMLab
KITT--NM
d5pe dSpe-Lox
d5pr-On
ANORB-Azio
ANORB-EN
DTD
Pots
Pall al. 2023
87.2
86.0
57.4
38.8
79.7
95.2
73.9
65.5
57.5
46.7
29.1
Lincan Dia
64.4
85.0
63.2
97.0
36.3
36.6
51.01
78.5
57.5
08.5
74.11
34.3
50.6
33.2
55.4
12.5
20.0
9.6
19.2
VPT Ree dil Suree
7888
90.8
65.8
98.0
58.1
78.1
49.6
81.8
96.1
83.4
68.4
68.5
50.0
46.5
72.8
73.6
47.9
32.9
17.8
LoRa
67.1
01.4
60.4
98.8
90.4
65.3
54.0
84.9
95.3
54.4
73.6
82.9
69.2
49.
78.1
75.7
47.1
31.0
44.0
NOAH
2025
84.4
44.2
PROFT
51.4
92.4
19.6
37.7
79.3
73.5
12.6
57.9
77.5
10.2
24.0
10.7
PROFT (warmup)
09.4
90.5
09.6
98.9
39.2
89.4
32.0
84.9
95.2
85.6
74.3
80.5
54.7
51.9
90.6
82.3
49.3
32.7
25.1
Table 13: Image classification accuracy on VTAB-IK Bold best, Underline = second-best. Ties for best are all bolded; no second-ber best is shown in such cases. We compare PROFIT against standard fine-tuning strategies, including full fine-tuning and linear fine-tuning (denoted by Full and Linear). A naive PROFIT violates the proximity condition and performs poorly, but with Adam warm-up it achieves consistently higher aceuracy.
weights the model architecture, and PROFIT is competitive with these baselines. For future work we would be interested to see how to apply PROFIT on top of LoRA and NOAH implementations. as we suspect these methods would be synergistic and would further improve downstream model performance.
D Limitations
As discussed in Sec 3.2, PROFIT requires inference on Nret + 1 batches (2 for fre = 1) per iteration, hence being slower than standard optimizers for fine-tuning. In practice, we observe that PROFIT converges earlier than corresponding baselines (which are trained longer for fair comparison). which may mitigate this. We also note that typical fine-tuning settings are shorter and not as involved as training from scratch, which may further alleviate this concern. Another limitation is that our method would consume slightly more memory as a result of instantiating two optimizers (tef) and O).
D.1 Memory Profiling
In training generative models, memory consumption is bottleneck, and as mentioned in 3.4 the vanilla version of PROFIT induces some training-time memory overhead. For completeness we include those memory profiling experiments here. As model parameter counts and activation footprints grow, so does the GPU/ VRAM requirement, often leading to training failures, batch-size reductions, and suboptimal hardware utilization. Memory profiling plays a critical role in assessing whether particular, when proposing
proposed method remains practical under realistic resource constraints. new optimizer like PROFIT, we must ensure that any additional memory overhead is manageable and that itcration time remains comparable. To validate the generality of our method, we profiled a diverse set of architectures that included both language and generative models.
We conducted a series of memory profiling experiments to compare PROFIT with the standard AdamW W optimizer in a range of widely used architectures. These models were selected to represent different domains of interest for future research in fine-tuning large models. Table 14 summarizes the peak memory consumption and the per-iteration time for each setup.
Midel
Peak Men. (MB) AdamW Poak Mom. (MB) PROPIT Itee Time (ms)
AdamW Ther. Time PROPIT
OPT 1.30 10.00 222gg a
k022
5532.5
6324.
105.6
115.7
OPT-2.78 TB 2 Some FiR
86012.4 4
10076.1
164.2
174.3
LLaMAS ki Dolee 2021
2034.3
25693.4
474.7
490.4
OSS 200 2iun K055
14502.5
17992
117.7
139.9
Suble Diffesihi1 20mme:
2022
5603.5
7134.5
130.3
142.9
Table 14: Peak memory usage and iteration time for AdamW vs. PROFIT. PROFIT shows modest additional memory overhead with comparable iteration latency. Note that we did not fine-tune the models with PROFIT for the accuracy here, and the experiments are for memory profiling only.
For OPT (Zhang et al. 2022 ). I.LaMA (Dubey et al. (0244) ) and Stable Diffusion v1-5 (Rombach) et al. 20222) ), we adopt QLoRA (Dettmers et al. 2023 for fine-tuning, while LoRA (Hu al.
29

---

2022 is used for GPT-OSS-20B, This profiling highlights PROFIT's s generality and efficiency across both language and vision-language models. As shown Table 14 have conducted the memory profiling experiments. The experiment shows that PROFIT is on par with AdamW, with an additional memory increase. This is because we treat both main and reference optimizers completely separately in order to better explore the hyperparameter space. As we discussed in the Section 3.4 it expected that the memory overhead the model training is increased approximately 25% over the AdamW (Loshchilov and Hutter (019)) baseline.
Impact Statement
The methods presented in this work represent general advancements in the field of machine learning, and are intended to be applicable in wide range of machine learning applications. We hope that this work will help in preventing catastrophic forgetting and overfitting to potentially harmful data, which may be positive ethical/societal effect. Other than that consideration, although there may be broader impact considerations in any downstream applications using our work, we feel that this work taken in isolation does not engender any additional societal impacts worth noting here.
30


Reviews:
[{"review_id": "qxMRKipbBd", "reviewer": "Reviewer_HVkg", "summary": "The paper proposes a novel optimization algorithm for “proximal fine-tuning” where fine-tuning dataset is from a similar distribution to pre-training. The objective of this work is to fine-tune a fully pre-trained model to perform well on both fine-tuning and pre-training tasks. To this end, it employs two optimizers: a standard optimizer and reference optimizer. The proposed algorithm first uses reference optimizer to “explore” loss landscape and updates the standard optimizer using orthogonal projection of the gradient onto the change of parameters in the reference optimizer, not to catastrophically forget the parameters learned from pre-training dataset. The paper provides extensive experimental results on  image classification, visual question answering and robotics motion prediction tasks on datasets that does and does not satisfy the proximal fine-tuning assumption.", "questions": "* What is the difference between Section 4.1 and 4.2? I do not see if this distinction is necessary.\n* Can the authors provide the formula for the orthogonal projection explicitly for better readability?", "limitations": "Limitations of the methods related to assumptions and additional hyperparamters were noted and discussed in Section 3.3 and 3.4.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper proposes a simple but effective method tailored for fine-tuning.\n2. The assumptions (which can be limitations) were clearly noted and discussed in the paper to reduce any potential confusion readers may have.\n3. It provides some theoretical derivation that supports the effectiveness of the proposed method.\n4. It provides extensive experimental results on various fine-tuning tasks.\n\n\n**Weaknesses**\n1. Some parts of the writing lack clarity. \n\t1.  In line 154-156, it is not clear how “because…” is a justification for projecting g onto $\\Delta$ not the other way around.\n\t2. In line 157-158, “take both steps…” and “to take a step” sound like $\\theta$ is updated three times in total but it seems like it is updated once after projection.\n\t3. Minor: the term equilibrium was used without sufficient explanation, making its meaning unclear for a while.\n2. The proximal fine-tuning assumption could be a strong assumption. Although the authors explicitly mentioned this and provided an example of perception in autonomous driving, there are obviously more cases where this assumption does not meet than the cases where it does. To have this assumption satisfied for more cases, the pre-training dataset needs to be large enough to be similar to any potential fine-tuning datasets.\n3. In addition, it is not clear how to measure if the proximal fine-tuning assumption is satisfied or not between pre-training and fine-tuning datasets. \n4. It is not clear why the standard SGD is recommended for $O^\\text{ref}$. Line 208-209 sounds like it is solely for the sake of mathematical simplicity. Is it a correct interpretation? Can the authors provide more fundamental reason?\n5. Experiments are done with only one random seed. Without standard deviation, it is hard to know how significant differences are.\n6. Bold in Table 2 does not seem to be for “the highest accuracy are highlighted in bold.” which may mislead readers. \n7. The 10 epoch warmup with AdamW seems like a heuristics based on extensive experiments. How should practitioners choose the number of epochs for an arbitrary pair of pre-training and fine-tuning datasets?\n8. The authors provided two justifications for additional computation using $n_\\text{ref}$ in Section 3.3. However, I cannot find experiment results that support these claims. \n9. There is no formal proof for Theorem 3.1 (I couldn’t find any proofs in the appendix). Could the authors provide the formal proof for Theorem 3.1? In particular, I do not really understand “If the learning rate is sufficiently small, then $\\nabla_{\\theta’} L_\\text{old}$  must point in the $\\Delta$ (as defined in Alg 1) direction”", "was_revised": true, "final_justification": "The authors’ rebuttal addresses some of my initial concerns. However, I remain unconvinced that the proximal assumption: the fine-tuning dataset comes from a distribution similar to that of pre-training, is generally realistic or justified in practical scenarios. In addition, the results are based on a single random seed, which weakens the credibility of the findings, particularly when performance differences between methods are relatively small.\nFinally, while the writing of the paper needs improvement, I did not observe substantial effort or willingness to address this in the revision. Taken together, I do not believe this work meets the acceptance standards of NeurIPS.", "rebuttal": "Thank you for your review and finding our method “simple and effective” with “extensive experimental results”. \n\n> If proximal fine-tuning assumption could be a strong assumption\n\nThe proximal setting is very common in machine learning applications. In computer vision, this can be applied when the sensor modality stays constant through the lifecycle - expanding classification/segmentation taxonomies in self-driving, localization and action in new indoor environments for humanoid robots, new rooms/layouts/maps for AR/VR applications. In language models, this can be used to fine-tune a base model to support new tool-calls or function calls, or in post-training to ensure pre-trained knowledge isn’t forgotten while instruction-following capabilities are being introduced. In general, any setting where a model has to be updated due to collection of new data would also fall into the proximal setting.\nWe note that even nonproximal fine-tuning settings can be made proximal with a mild warmup stage. In Section 4.2, we presented experiments on VTAB where naive application of PROFIT leads to poor results, but a simple warmup period (10 out of 100 epochs) transforms the VTAB fine-tuning setting into a proximal setting, and at that point PROFIT leads to significantly improved results. In other words, even non-proximal fine-tuning settings can be easily transformed into proximal settings, making PROFIT actually a fully general algorithm for fine-tuning. \n\n> How to measure if the proximal fine-tuning assumption is satisfied\n\nIn our setup, the pre-training and fine-tuning datasets are known to be sampled from the same underlying distribution. We also do show situations where proximality fails - for example in VTAB (Sec 4.2). However, we also noted within our VTAB experiments that a short warmup cycle (10 out of 100 epochs) moves the VTAB setting into the proximality range and at that point PROFIT provides significant training benefits. While we find measurement of the proximality condition to be complex and very dependent on dataset and model properties, we at least provide a method to induce proximality in any setting which should work for general use-cases.\n\n> How to choose the number of epochs for an arbitrary pair of pre-training and fine-tuning datasets\n\nThe VTAB strategy was a heuristic chosen to warmup for 10% of epochs and fine-tune for the remaining epochs. We tried some variants with slightly higher warm-up epochs and found similar performance. Therefore, when a practitioner is faced with finetuning on arbitrary distributions, we recommend starting with a warmup for 10-20% of the total epochs.\n\n> Why the standard SGD is recommended \n\nThe reference optimizer is used to perturb the model away from the local optima to estimate the local loss landscape. By construction, $\\Delta$, the pseudo-gradient (L155), minimizes the loss on the old task. Using optimizers which rely on other statistics would mean that $\\Delta$ does not faithfully represent the direction to minimize old task loss, since it may be influenced by information gathered at another point in the loss surface. This property only holds true for vanilla SGD, and hence the reference optimizer should be set to SGD.\n\nIn other words, PROFIT’s core algorithm is powered by simple orthogonality relationships between the main and reference gradient updates. If the reference update is calculated by any other optimizer than SGD, it is too complex for the simple orthogonality calculus to still hold true.\n\n> Experiments are done with only one random seed\n\nExperiments were run multiple times and we report error bars here. These will be added in a revision if we are allowed to proceed. CIFAR experiments have a standard error <= 0.01, DriveLM experiments have standard error within 0.2 on accuracy, VTAB experiments shows the mean accuracy over 3 repeats (as discussed in the VTAB paper), Waymo Open Motion Dataset experiments have standard errors within 0.005m for 3s metrics and 0.015m for 8s metrics. \n\n> Formal proof for Theorem 3.1\n\nAt a convergence point $\\mathbf{x}_0$, the first-order gradient vanishes and to leading order the loss surface will look like $L(\\mathbf{x}) \\approx \\mathbf{x}_0 + 0.5(\\mathbf{x} - \\mathbf{x}_0)^t\\mathbf{H}_0(\\mathbf{x} - \\mathbf{x}_0)$ for positive definite hessian $\\mathbf{H}_0$. The gradient within this region is therefore $\\nabla L(\\mathbf{x}) \\approx \\mathbf{H}_0(\\mathbf{x} - \\mathbf{x}_0) \\equiv \\mathbf{H}_0\\Delta$. Because $\\mathbf{H}_0$ is positive definite given that $\\mathbf{x}_0$ was a critical point, this means that $(-\\Delta)^t\\mathbf{H}_0\\Delta < 0$, which means the $-\\Delta$ direction is a valid gradient descent direction. We apologize for the confusion - by “$a$ points in the direction of $b$” in the part of the proof you referenced we intended the meaning to imply “pointing in the same direction as,” which implies $\\langle a, b\\rangle > 0$ and shows that $\\Delta$ is a valid gradient descent direction. We hope this more formal proof clarifies any ambiguity and will include it in the final version of the paper. \n\n> Difference between Sec 4.1 and Sec 4.2\n\nIn Sec 4.1, we demonstrate results on image classification when the pre-training and fine-tuning datasets are sampled from the same distributions. In Sec 4.2, we illustrate a case where this assumption may not hold, and demonstrate a strategy to use PROFIT in such cases."}, {"review_id": "Jisc9eVSIg", "reviewer": "Reviewer_FYXs", "summary": "This abstract presents PROFIT, a novel optimizer targeting improved model accuracy for fine-tuning pre-trained models, addressing a perceived shift in the field towards efficiency over accuracy. The authors position PROFIT as one of the first optimizers specifically designed for incrementally fine-tuning models that have already converged on prior tasks/datasets.", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. PROFIT uniquely resolves catastrophic forgetting by projecting new-task gradients onto a plane orthogonal to pretrained model displacements\n\n2. The method integrates as a plug-and-play optimizer wrapper requiring no architectural changes or extra parameters.\n\n3. Demonstrates consistent gains in image classification, autonomous driving VQA, and motion prediction .\n\nWeaknesses:\n\n1. The caption of Table 2 states that bold values indicate the best results, yet there are ​​two bold entries per column​​. Most of the proposed method’s results are bolded, which is confusing.\n\n2. Table 2 compares the proposed method with classic PEFT (Parameter-Efficient Fine-Tuning) algorithms, but ​​image classification alone does not sufficiently demonstrate PEFT’s applicability to general vision tasks​​. For real-world applications (e.g., autonomous driving, robotics), ​​more complex tasks like object detection or instance segmentation​​ are more practical and have been benchmarked in recent works [1-2]. Given this is NeurIPS, I suggest the authors to ​​expand experiments to validate the method’s generality​​. \n\n[1] 5% > 100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks\n\n[2] 1% vs 100%: Parameter-Efficient Low-Rank Adapter for Dense Predictions\n\n3. Lines 172, 183, 185, 194, etc., exhibit ​​encoding artifacts​​ at their line endings.", "was_revised": true, "final_justification": "Thanks to the authors. I will keep my score.", "rebuttal": "Thank you for your thoughtful review and recognition that our work “uniquely resolves catastrophic forgetting” and “integrates as a plug-and-play optimizer”.\n\n> Two bolded entries per column in Table 2\n\nWe include results from adapter methods such as VPT, NOAH, etc for completeness and to show that PROFIT is competitive with these SOTA methods. However, we emphasize that these methods induce additional trainable parameters and the comparison is not apples-to-apples. We will add a column on Table 2 to indicate when a method induces additional trainable parameters, format the table to show clearer grouping between different methodology classes, and also include discussion of this point in the main text. \n\n> Suggest the authors to ​​expand experiments\n\nWe thank the reviewer for the suggestion. We believe the experiments reported in Sec 4.1, 4.2,  4.3, and 4.4  are comprehensive, and although we are time-constrained and cannot run the requested experiments in time for this response, we are confident they will show the same trend as the other experiments presented. We note that benchmarks added for VTAB are a wide variety of classification tasks (Sec 4.2), and we also demonstrated positive results on VLMs in Sec 4.3, which represents a bleeding-edge modern computer vision task. Our experiments on DriveLM also show improvements in a setting that is tangential to computer vision and very high-dimensional. However, we will plan to add additional experiments in time for the camera-ready."}, {"review_id": "EOZuTG6GdC", "reviewer": "Reviewer_4J8q", "summary": "This paper proposes PROFIT, a specialized optimizer designed to improve fine-tuning of pre-trained deep learning models. Unlike conventional optimizers such as SGD or Adam, which are designed for training from scratch, PROFIT leverages the structure of an already converged model by treating its current state as a reference point. By framing fine-tuning as a form of temporal multi-task learning, PROFIT enables accurate and efficient adaptation to new but related tasks without introducing additional parameters.", "questions": "- Given this proximal setting, why is a specialized optimizer like PROFIT necessary instead of simply applying standard proximal optimization techniques (e.g., L2 regularization around the pre-trained weights)? A direct comparison or ablation would help clarify whether the performance gain stems from the temporal orthogonalization approach itself or merely from staying close to the initial weights.\n- PROFIT introduces new hyperparameters (e.g., reference steps $n_\\text{ref}$, $\\lambda_\\text{ref}$), which influence the computation cost and optimization behavior. What is the actual impact of $n_\\text{ref}$ on performance and forgetting? An ablation study varying $n_\\text{ref}$ (e.g., 1, 2, 4, 8…) would help determine whether the reference optimization phase is truly beneficial or just an overhead.\n- The proposed method assumes a proximal relationship between pre-training and fine-tuning data distributions, which is central to its effectiveness. However, in many real-world applications, this assumption may not hold strictly. Could the authors provide more concrete and diverse examples where proximal fine-tuning naturally arises in practice? Some clarification would help assess the general applicability of the method beyond narrowly defined scenarios like autonomous driving or sensor-specific tasks.\n- The current method focuses on proximal fine-tuning, where pre-training and fine-tuning distributions are assumed to be similar. Can the authors clarify whether PROFIT could be extended or adapted for domain adaptation scenarios, where the source and target distributions differ moderately or significantly?\n- Figure 1 appears to be central in conveying the intuition behind PROFIT’s update mechanism. However, the diagram’s intent and the plots are not clearly explained and may be confusing to readers. Could the authors revise or elaborate on the figure to more explicitly show how PROFIT differs from standard optimizers in both conceptual and operational terms?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. Strengths\n- PROFIT is one of the first optimizers explicitly designed for fine-tuning pre-trained (converged) models, addressing a key gap where traditional optimizers like SGD and Adam fall short.\n- The use of orthogonal projection to decouple new-task gradients from the restoration direction (toward the original state) is a simple and principled strategy inspired by multi-task learning, allowing stable and targeted adaptation.\n\n2. Weaknesses\n- PROFIT is only effective when the fine-tuning dataset is from a distribution similar to the pre-training dataset. It fails or performs poorly in cases of significant domain shifts, limiting its general applicability.\n- PROFIT introduces a reference optimization phase ($n_\\text{ref}$ steps) for each training iteration, which can increase training time, although this may be offset by faster convergence.\n- Please also refer to the following questions.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed and thoughtful response to my questions. While several points were clarified, some of my core concerns remain unresolved. Specifically, the method is framed from a proximal optimization perspective, yet no comparison is provided against a standard L2-based proximal optimization baseline, which could be a natural and principled reference in this context. Including such a baseline is crucial to properly assess the contribution of the proposed approach.", "rebuttal": "Thank you for your positive reading and recognizing our work as “one of the first optimizers explicitly designed for fine-tuning pre-trained deep learning models”.\n\n> How PROFIT compares to standard proximal optimization techniques (e.g: L2 regularization around the pre-trained weight)?\n\nAn L2 regularizer around the pre-trained weights relies on a maximally uninformative Bayesian prior that will indiscriminately pull updates back towards the pre-trained point. However, L2 regularization does not take into account weight updates towards the fine-tuning task at all. We already know that such indiscriminate preference of the original convergence point is suboptimal, because in all fine-tuning/post-training settings the old convergence point is assumed to be a poor solution (otherwise we would not fine-tune at all). PROFIT resolves this issue by continuing to update the model “anchor” point after each step, which allows for dynamically useful updates throughout model training.\n\nWe also note that L2 regularization around the pre-trained weight would necessitate keeping a copy of the model weights around to be able to perform this regularization step. This both induces compute/memory overhead but also adds ambiguity when a model needs to go through multiple iterative stages of fine-tuning. L2 regularization also adds a magnitude hyperparameter while PROFIT handles update magnitude balancing automatically and with theoretical grounding. These restrictions make this type of L2 regularization scheme less practical. PROFIT does not require any additional data, which is why the comparison itself is less meaningful (just like it would be difficult to compare the L2 regularization algorithm to a situation where we have access to all the original dataset). We chose to primarily focus on comparisons between PROFIT and settings that assume access to the same cardinality of information to ensure comparisons are all apples-to-apples.\n\nThus, we do not believe L2 regularization is an effective proximal optimization technique but also is only possible in a privileged setting that assumes higher levels of information access than we have in the more general PROFIT setting. To the best of our knowledge, we are the first to examine the “proximal fine-tuning” setting with these data restrictions, which also is the most general setting for fine-tuning. \n\n> What is the actual impact of hyper-parameters on performance and forgetting?\n\nWe quantify the impact of hyper-parameters in the supplementary. In particular, we point the reviewer to Table 4 and 5 where we ablate the critical hyper-parameters of our method. We leverage these hyperparameters in all the discussed experimental settings.\n\n> Examples where proximal fine-tuning arise in practice\n\nThe proximal setting is very common in machine learning applications. In computer vision, this can be applied when the sensor modality stays constant through the lifecycle - expanding classification/segmentation taxonomies in self-driving, localization and action in new indoor environments for humanoid robots, new rooms/layouts/maps for AR/VR applications. In language models, this can be used to fine-tune a base model to support new tool-calls or function calls, or in post-training to ensure pre-trained knowledge isn’t forgotten while instruction-following capabilities are being introduced. In general, any setting where a model has to be updated due to collection of new data would also fall into the proximal setting.\nWe note that even nonproximal fine-tuning settings can be made proximal with a mild warmup stage. In Section 4.2, we presented experiments on VTAB where naive application of PROFIT leads to poor results, but a simple warmup period (10 out of 100 epochs) transforms the VTAB fine-tuning setting into a proximal setting, and at that point PROFIT leads to significantly improved results. In other words, even non-proximal fine-tuning settings can be easily transformed into proximal settings, making PROFIT actually a fully general algorithm for fine-tuning. \n\n> Whether PROFIT could be extended where source and target distributions differ\n\nWe point the reviewer to Section 4.2, where we show our method can be mitigated in practice by using a short warm-up phase with standard fine-tuning before switching to PROFIT. Furthermore, we show this approach works well in Table 2.\n\n> Revise or elaborate on Figure 1\n\nWe discuss how PROFIT differs from standard optimizers in the caption of Figure 1. Standard optimizers which take successive steps through parameter space, while the reference optimizer (superscript “r” - for “reference” - in the figure) keeps track how the system tries to move away from good state ($x_0$). The good state is then restored ($x_0$) and then reference steps are translated to $x_0$. The main optimizer then takes a step and the gradient update is orthogonalized wrt to the translated reference vector. For further details, we point the reviewer to Alg 1 and Sec 3.1 where we elaborate the intuition and operational mechanism behind our method."}, {"review_id": "xQfo3OoBXN", "reviewer": "Reviewer_Fqcd", "summary": "This paper introduces PROFIT (PROximal Fine Tuning), a novel optimizer designed to mitigate catastrophic forgetting during the fine-tuning of pre-trained models. The core idea is to reframe the optimization process as a temporal multi-task learning problem, where the model must learn a new task while not forgetting its original one. PROFIT achieves this by treating the model's pre-fine-tuning state as an \"anchor.\" It then uses gradient orthogonalization to adjust the update steps for the new task, specifically nullifying any component of the gradient that directly conflicts with returning to the anchored state. The paper positions PROFIT as a better alternative to standard full-model fine-tuning using optimizers like AdamW. Experiments across various domains, including image classification and motion prediction, demonstrate that PROFIT improves performance over standard baselines.", "questions": "In Table 2, the paper includes results from several state-of-the-art parameter-efficient methods (e.g., VPT, NOAH) which often outperform  PROFIT (warmup). Since the paper's text explicitly defines its comparison against standard Full and Linear fine-tuning , could you clarify the intended role of these other SOTA methods in your analysis? What is the key takeaway for a reader when comparing PROFIT to these methods?\n\nPROFIT (warmup) strategy is presented as an effective solution for the non-proximal VTAB benchmark . Was the choice of a 10-epoch AdamW warmup a result of a systematic hyperparameter search, or was it a heuristic? How sensitive is the final performance to the duration and the specific optimizer used for this warmup phase?\n\nI will be happy to adjust score if the author can address my concern (plus appropriate response to the weakness).", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n+ Paper is well writen.\n+The core idea of framing fine-tuning as a temporal multi-task problem and using gradient orthogonalization as a mechanism is reasonable and presents an interesting approach to the challenge of catastrophic forgetting.\n\n+The paper's central claim—that PROFIT outperforms standard full-model fine-tuning—is supported by empirical results across a decent breadth of tasks, including image classification, visual task adaptation, and motion forecasting. \n\nWeakness\n- The presentation in Table 2, which compares PROFIT to many advanced optimizers, could be clearer. The paper does not fully disclose the reason for including results from several state-of-the-art methods (e.g., VPT, NOAH) that often show superior performance. While the text specifies that its direct comparison is to standard fine-tuning, the table's structure may confuse readers about the paper's positioning relative to the broader SOTA landscape.\n\n- The paper claims PROFIT is easy to integrate but lacks a comparative analysis of implementation complexity. It is unclear how the effort to implement PROFIT compares to setting up the other advanced optimizers listed, particularly parameter-efficient techniques like LoRA or VPT which also aim to simplify model adaptation \n\n- The paper should disclose the spatial (memory) overhead. Storing reference weights for large-scale models could have significant memory implications, which is a crucial factor for practical deployment, but this is not measured or discussed.", "was_revised": true, "final_justification": "Based on the authors' response, I feel the problem of having another copy of optimizer would be a bit of waste, considering the marginal improvement. However, the authors have provided evidence this can be avoided after making the decision of using certain types of optimizers. I am happy to raise score to borderline accept.", "rebuttal": "Thank you for finding our paper “well written” and “supported by empirical results across a decent breadth of tasks”.\n\n> Was the choice of a 10-epoch AdamW warmup a result of a systematic hyperparameter search?\n\nThe VTAB strategy was a heuristic chosen to warmup for 10% of epochs and fine-tune for the remaining epochs. We tried some variants with slightly higher warm-up epochs and found similar performance. Therefore, when a practitioner is faced with finetuning on arbitrary distributions, we recommend starting with a warmup for 10-20% of the total epochs.\n\n> Paper does not fully disclose the reason for including results from several state-of-the-art methods (e.g., VPT, NOAH)\n\nThis is an astute observation. We include these results for completeness and to show that PROFIT is competitive with these SOTA methods. However, we emphasize that these methods induce additional trainable parameters and the comparison is not apples-to-apples. We will add a column on Table 2 to indicate when a method induces additional trainable parameters, format the table to show clearer grouping between different methodology classes, and also include discussion of this point in the main text. \n\n> How the effort to implement PROFIT compares to setting up other advanced optimizers\n\nWe outline the algorithm of PROFIT in Alg 1. This logic can be encapsulated in an PyTorch optimizer, and our implementation is ~20 lines of code.\n\n> Paper should disclose the spatial (memory) overhead\n\nWe discuss the memory complexity for our method in Table 8 of the supplementary. Storing an additional copy of states for the reference optimizer does increase memory complexity, which is a limitation of our method as we point out in Sec 2.2.6 of the supplementary.\n\nWhile PROFIT is implemented with two optimizers in our experiments for generality and flexibility, this is not a requirement. In practice, one can reuse a single optimizer instance (e.g., SGD, AdamW, etc.) for both the main and reference model, provided care is taken to isolate their parameter updates. This reduces the additional memory footprint to effectively zero.\n\nMoreover, PROFIT is compatible with parameter-efficient fine-tuning techniques such as adapters (e.g., LoRA, VPT). In Table 3 of the supplementary, we show that combining PROFIT with these methods yields consistent performance improvements while keeping memory usage low."}, {"review_id": "ciK7JCnfMX", "reviewer": "Reviewer_97b1", "summary": "Fine-tuning pretrained deep models often sacrifices accuracy on the original task.\nPROFIT is an optimizer that leverages the fact that a finetuning run starts from a converged model. By anchoring updates to the reference state and orthogonalizing gradients, PROFIT regularizes weight updates and yields better performance with minimal engineering overhead.", "questions": "In Section 4.3, could you also report zero-shot accuracy on a standard LLM benchmark suite (e.g. MMLU or LAMA) before and after fine-tuning to quantify how much general knowledge is retained?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. PROFIT can be easily dropped into an existing deep learning framework and it avoids the overhead of adapters or LoRA.\n1. Evaluations cover images, vision-language, and motion trajectories. Waymo results are convincing.\n1. When the pre-train and fine-tune distributions diverge, authors present an AdamW warm-up fix.\n1. Theoretical analysis explains why the projection step limits weight drift.\n\n**Weaknesses**\n1. Although the authors mention that \"Since our method is an optimizer by design and does not require additional storage (unlike the continual learning paradigm), we do not benchmark against continual learning methods.\", it would be great to see LWF(Li & Hoiem 2017) included as a baseline, since it is a frozen model copy with soft target distillation. This would give more context to PROFIT's gains.\n\n**Minor**\n1. line 187: occrs -> occurs\n1. line 189: oprtimizer -> optimizer\n1. line 315: during fine-tuning. by leveraging -> during fine-tuning by leveraging\n1. labels in figures are hard to read, especially in Table 2 and Figure 2. Consider splitting them or move some parts to the appendix.", "was_revised": true, "final_justification": "I keep my score unchanged for the following reasons:\n1. Authors gave a conceptual argument that PROFIT can be seen as an online variant of LWF and commit to add an empirical comparison in the camera-ready. The current comparisons already cover the optimizers that make the same resource assumptions, and the theoretical clarification shows why LWF is disadvantaged.\n1. Authors promised MMLU results in the final version.", "rebuttal": "Thank you for your positive assessment of our work. We are very much aligned that the positives you pointed out (our convincing results and our method’s lightweightness) are principles at the heart of what we hoped to accomplish with PROFIT.\n\n> Including LWF as a baseline\n\nWe will plan to run an analysis against LWF for a revision, but for now can offer the following theoretical analysis: for each step of PROFIT, we update the model by two signals: the signal $-\\Delta$ which is equivalent to the soft distillation target update within LWF, and an additional update $g\\perp \\Delta$ which is a regularized version of the standard new task update within LWF. Therefore, one can think of PROFIT as a flexible version of LWF where we constantly update the “frozen” model to the model state at $t-1$. This allows PROFIT to surpass a few notable limitations to LWF that limited its practical utility:\n\n1. LWF requires a separate inference step on a model copy that must be kept around.\n2. LWF is difficult to configure when a model has to be maintained through multiple fine-tuning iterations, as it is unclear what to use as the frozen “base” model through multiple steps. PROFIT always sets the frozen model to the model at timestep $t-1$, which removes ambiguity on frozen model choice.\n\nIn addition, since PROFIT is a drop-in optimizer, it is significantly easier to configure and run.\n\nUltimately, it is reasonable to think of PROFIT as a practical evolution of LWF which removes standard barriers and ambiguity, while preserving the core idea. Importantly, PROFIT accomplishes this while removing assumed access to material data (i.e. frozen model weights) that LWF relies on. As such, we originally felt the comparison would be not too instructive, similarly to how comparing LWF to fine-tuning a model with the original dataset intact would also be difficult to interpret. Comparison to standard optimizers which make the same set of assumptions on access to data is more apples-to-apples. However, we are aware that given how much our methodology is inspired by LWF, a comparison may be of interest and we are working hard on completing that comparison to include in the supplementary. We hope that in the meantime this additional context at least clarifies how PROFIT is theoretically positioned with respect to LWF and why we decided to omit the comparison in the main paper to keep the analysis clean.\n\n> Zero-shot accuracy on a standard LLM benchmark suite\n\nWe will plan to run experiments on MMLU for the camera-ready version. One benchmark we ran that provides insight into the amount of general information retained are the CIFAR baselines we present in Section 4.1. Although not an LLM baseline, these experiments show that PROFIT beats all other methods in terms of knowledge retention and performance on the transfer set. We note that PROFIT is not intended as a silver bullet against catastrophic forgetting, but is mainly designed to mitigate forgetting in such a way that optimizes learning on the transfer set. This design principle explains why we largely reported transfer set accuracy results. However, we agree that these results could provide additional context on another baseline for PROFIT and will begin to run those experiments."}]

Author rebuttal:

Meta-review: This is a nice paper that introduces a novel and practical optimizer, PROFIT, specifically designed for the ubiquitous task of fine-tuning pre-trained models. I am pleased to recommend this paper for acceptance.

The reviewers and I were impressed by several key strengths. The work addresses a clear gap in the literature, as PROFIT is one of the first optimizers tailored for fine-tuning already converged models. Its core idea of framing fine-tuning as temporal multi-task learning and using gradient orthogonalization to mitigate catastrophic forgetting is both simple and effective. A major advantage is its ease of integration—it's a "plug-and-play" wrapper that doesn't require architectural changes, making it highly accessible to practitioners. The authors back up their claims with extensive and compelling experiments across a diverse range of applications, from image classification and VQA to large-scale motion prediction, with the Waymo results being particularly noteworthy.

During the review process, some valid concerns were raised. One key point was the "proximal fine-tuning" assumption and how to determine if it holds. The authors have convincingly addressed this by demonstrating that a simple warm-up phase can adapt the method for non-proximal settings, and by highlighting numerous practical scenarios where this assumption is naturally met. Another important point was the lack of direct comparisons to methods like L2 regularization or continual learning approaches such as "Learning without Forgetting." While this is a valid critique, it does not diminish the core contribution of the paper. I strongly encourage the authors to include these comparisons and further discussion in their camera-ready version to provide a more complete picture for the community.

Overall, PROFIT presents a valuable, well-motivated, and empirically validated contribution that will be of great interest to the machine learning community. Its simplicity and effectiveness make it a promising tool for practitioners. This work is a solid addition to the NeurIPS program.
Decision: Accept (poster)