ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Authors: Tao Feng
Wei Li
Didi Zhu
Hangjie Yuan
Wendi Zheng
Dan Zhang
Jie Tang
Venue: ICML 2025 poster (2025)
Keywords: Catastrophic Forgetting
Continual Learning
Incremental Learning
Primary area: deep_learning

Abstract: Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Optimizers such as SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. However, access to gradient information is not always feasible in practice due to black-box APIs, hardware constraints, or non-differentiable systems, a challenge we refer to as the gradient bans. To bridge this gap, we introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting. ZeroFlow examines a suite of forward pass-based methods across various algorithms, forgetting scenarios, and datasets. Our results show that forward passes alone can be sufficient to mitigate forgetting. We uncover novel optimization principles that highlight the potential of forward pass-based methods in mitigating forgetting, managing task conflicts, and reducing memory demands. Additionally, we propose new enhancements that further improve forgetting resistance using only forward passes. This work provides essential tools and insights to advance the development of forward-pass-based methods for continual learning.

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Tao Feng' Wei Li
Didi Zhu
Hangjie Yuan?
Wendi Zheng'
Dan Zhang'
Jie Tang'
github.
Abstract
K
Backpropagation provides a generalized configu- ration for overcoming catastrophic forgetting. Op- timizers such SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. However, access to gra- dient information is not always feasible in prac- tice due to black-box APIs, hardware constraints, or non-differentiable systems, a challenge we re- fer to as the gradiens bans. To bridge this gap. we introduce ZeroFlow the first benchmark de- signed evaluate gradient-free optimization al- gorithms for overcoming forgetting. ZeroFlow examines a suite of forward pass-based methods across various algorithms, forgetting scenarios, and datasets. Our results show that forward passes alone can sufficient to mitigate forgetting. We uncover novel optimization principles that high- light the potential of forward pass-basd pass-based methods in mitigating forgetting, managing task confficts, and reducing memory demands. Additionally, we propose new enhancements that further improve forgetting resistance using only forward passes. This work provides essential tools and insights to advance the development of forward- forward-pass- based methods for continual learning.
Trais
Trais
Trais
Randienly
DD
D
Model
(e)
Aa(
2g
Permizsion denied
Forword Paes
Gradient Entimatioon
Figure 1: Illustrations of ZeroFlow. New tasks (or down- stream tasks) arrive sequentially, the gradient bans block the model from learning and memorizing using backpropaga- tion. ZeroFlow overcome this issue via forward passes.
continual pre-training (CPT) (Shi al., 2024; Zhu al., 2024b), etc. Among them, optimization algorithms play crucial role, e.g., SGD has become the default choice during CL (van de Ven et al., 2022), while Adam is frequently seen in fine-tuning FMs (Luo et al., 2023; Zhu et al., 2024a). These optimization algorithms in tandem with various meth- ods (ranging from regularization and rehearsal strategies to architectural changes) rely on gradient information avoid forgetting (Zhou et al., 2023c; Bian et al., 2024). Nonethe- less, in real- real-world scenarios, gradient information is not always available or computable (i.e., the gradient bans), like, Scenario i: large language models as a service (LL.MaaS) and black-box APIs. Scenario ii: hardware systems that do not support principled backpropagation. Scenario ii;: Al for science with non-differentiable underlying systems.
1. Introduction
Catastrophic forgetting remains one of the major challenges on the path to artificial general intelligence (AGI) (Hadsell et al., 2020; Zhou et al., 2023b), i.e., models tend to forget previously learned tasks when trained on new ones on time- evolving data flow (Feng et al., 2022b). This phenomenon commonly seen across various tasks, including continual learning (CL) (Wang et al., 2023), fine-tuning of foundation models (FMs) (Sun et al., 2025; Yuan et al., 2024), and
In other words, Scenario i implies that pretrained models are monetized (Miura et al., 2024) (model owners do not publicly release their pretrained models but instead the ser- vice), i.e., only the inputs and outputs are accessible (Gan et al., 2023; Sun et al., 2022). Scenarios ii/ili implies that the limitations prevent or restrict the execution of backprop- agation (Lillicrap et al., 2020), i.e., extremely high memory demands (Mangrulkar et al., 2022), unsupported systems and hardware (Jabri & Flower, 1992), or non-differentiable functions, etc (Tavanaci et al., 2019; Gu et al., 2021). The above means that typical methods for overcoming forgetting are not available because backpropagation is banned, as Figure This yields the primary question to be explored,
Core contribution 'Tsinghua University ZZhejiang University. Correspondence to: Jie Tang <jietang @ susinnhua.cdu.cn>...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Forward
Forward
Forward
Forwarti
1.0
0.20
1.0
0.15
20-Adam-S So
08
SGDD 20-Adam-S Sk
0.10
-GDD-Sign 20-Adam- Si
SGD--Sgu 20-Adam-Sin
0.30
DC-SGD-Sig
0.70
04
0.2
20-Adam
O.SGD 20-Adar
O.SGD 2d
0.SGG 20
00-9GD
20-Adam Com
SGD-CCos 20-Adam-Con
20-Adam- SGD-CCoos
Cot
SSDDCCom 20-Adam-Col
SCD-Cons
CIBAR-100
CUB
ImogeNict
OmmiBenchmark
(a) EASE on average accuracy
(h) EASE on forgetting
(c) APER on average accuracy
(d) APER on forgetting
Figure 2: ZeroFlow Evaluation Results of Catastrophie Forgetting. We visualize the evaluation results of 2 models (EASE (Zhou et al., 2024b) and APER (Zhou et al., 2023a)) in several ZeroFlow dimensions (average accuracy over all tasks and a forgetting metric). For comprehensive numerical results, please refer to Table 1.
pealing property is that need only forward passes to be enough to overcome forgetting. Maybe, once is all it takes!
(Q) Could establish a benchmark under gradient bans for overcoming catastrophic forgetting, and explore the overlooked optimization principles?
To sum up, our contributions are listed below,
To tackle (Q), a natural idea is to use the forward pass-based method (Hinton, 2022; Baydin et al.. 2022; Ren al., 2022) instead of backpropagation to overcome forgetting. The zeroth-order (ZO) optimization methods (Flaxman et al., 2004; Nesterov & Spokoiny, 2017; Malladi et al., 2023; Ghadimi & Lan, 2013), as representative methods, are well- suited to this issue due to their relaxed information require- ments, as they rely only on function values rather than gra- dients. Under gradient bans, DECL and DFCL (Yang et al., 2024) first attempt to overcome forgetting from stream of APIs, but they focus synthetic data level rather than opti- mization. Therefore, it remains elusive whether benchmark studies using gradient-free methods can mitigate forgetting.
(i) We propose the first benchmark ZeroFlow for over- coming forgetting under gradient bans. This benchmark includes our investigations into 7 forward pass optimization algorithms, several forgetting scenarios and datasets with varying complexity, and task sequences (as Figure 2).
(ii) Through this benchmark, we uncover overlooked opti- mization principles and insights into how forward passes can mitigate forgetting. These include the role of forward passes in managing task conflicts and the trade-offs between forgetting and memory efficiency. We proved that catas- trophic forgetting can be overcome in easier way!
(iii) Apart from a comprehensive evaluation of catastrophic forgetting, we introduce three enhancement techniques, which further improve the performance and efficiency of just forward passes to overcome forgetting.
In this work, we explore several Zeroth- order optimization methods on dynamic data Flow (as shown in Figure 1), examining their performance across various forgetting sce- narios, model types, and evaluation metrics. Through detailed analysis, we reveal the overlooked potential of forward passes and various ZO methods in overcoming catastrophic forgetting. This benchmark study offers an easier way to overcome forgetting and helps reveal the pros and cons of these methods in alleviating forgetting. Ex- tended from the gained insights, we introduce three new enhancement variants that further improve 70 optimization to overcome catastrophic forgetting. Simply put, we can mitigate forgetting more effectively and efficiently using only forward passes.
2. Literatures
Catastrophic forgetting. Catastrophic forgetting occurs across various tasks, including CL. fine-tuning of FMs, and CPT (Zhou et al., 2023b; Wang et al., 2023; Zhuang et al., 2022a: Luo et al., 2023). To mitigate this issue, various methods have been proposed (Aojun et al., 2025; Jeeveswaran et al., 2023; Sun et al., 2023b; Li et al., 2024). In CL. methods range from regularization and rehearsal strategies to architectural changes (Zhuang al., 2023; Bian et al., 2024; Lu et al., 2024). Lately, pre-trained models (PTM) further advanced these methods due to their strong generalization (Yuan et al., 2022; Feng et al., 2022a), as seen in PTM-based CL (Zhou et al., 2024a). All these meth- ods share a common goal: achieving an optimal balance between learning plasticity and memory stability (Wang et al., 2023). In FMs, catastrophic forgetting often arises from overfitting to small fine-tuning datasets during CPT or
Our rationale for choosing the ZO optimization algorithms to overcome forgetting for the following two key consid- erations: (i) implementation cost minimization, that is, we expect minimal modifications to existing optimizers. (ii) theory of diversity, that is, we expect to cover diverse op- timization methods. These considerations ensure that our benchmark is comprehensive and simplified. And, an ap-
2

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
fine-tuning (Luo et al., 2023: Zhu al., 2024a). Common techniques to address this include learning rate adjustment, parameter-efficient fine-tuning, mixed data strategies, and instruction tuning (Luo et al., 2023; Zhang et al., 2025). Ad- ditionally, as foundational models increasingly gain multi- modal capabilities, the complexity of catastrophic forgetting also intensifies (Zhao et al., 2024a; Zhu et al., 2024a).
Algorithm 1 Genetic formulation of 0O optimization
Require: Initialized model parameters 0g E Q S R2, train- ing dataset D = {(,, y: y)) E X x y, empirical loss function C. learning rate Te. gradient perturbation vector 5. and descent direction computation d6.)
1: while 0, not converged do
2: Sample mini-batch B from D
Optimization for catastrophic forgetting. Two broad cate- gories of optimization methods exist for overcoming forget-
3: Step 1. ZO gradient estimation:
4: g: = V E;
ting, (i) Standard Optimization. SGD and the Adam family are frequently employed to investigate catastrophic forget- ting (Hadsell et al., 2020; Masana et al., 2022). For instance, in CL. various CL methods predominantly utilize the SGD optimizer for standard evaluations (van de Ven et al., 2022; Sun et al., 2023a; Zhou et al., 2024c). In fine-tuning the LLM, the Adam series is commonly used to observe forget- ting phenomena (Luo et al., 2023; Zhu et al., 2024a). Some works explored orthogonal spaces with these standard opti- mizers to alleviate forgetting (Lopez-Paz & Ranzato, 2017; Feng et al., 2022c; Saha et al., 2020), such as OGD (Fara- jtabar et al., 2020), and GPM (Saha et al., 2020). Moreover, other works (Farajtabar et al., 2020; Chaudhry et al., 2018; Lopez-Paz & Ranzato, 2017) modified the gradients in the standard optimization process to align the learning spaces of new and old tasks, such as Uni-Grad (Li et al., 2024). The core of these efforts (Deng et al., 2021; Shi et al., 2021) is to find an cquilibrium between learning and forgetting in optimization. (ii) Sharpness-aware Optimization. This series of methods (He et al., 2019; Foret et al., 2020; Zhong et al., 2022; Zhuang et al., 2022b) has gained attention due to the effeectiveness of the flat minimum in mitigating for- getting (Li et al., 2024; Kong et al., 2023; Cha et al., 2021; Mehta et al., 2023). Methods such as FS-DPGM (Deng et al., 2021), F2M (Shi et al., 2021), DFGP (Yang et al., 2023), SAM-CL (Tung et al., 2023) overcome forgetting in the flatness areas of different configurations. C-Flat (Bian et al., 2024) proposed a CL-friendly general optimization framework, that holds promise as baseline optimizer for overcoming forgetting.
5: Step 2. Descent direction computation:
6: h; = Φb (1)))=
Step 3. Parameter updating:
8: 0.+1 0, "):-" h,
9:
L 1=1+1
10: end while
Ensure: Updated model 0z
tive solution for estimating descent directions in scenarios where first-order (FO) gradients are either inaccessible or infeasible to compute. Considering a deep learning model parameterized with 0 E e C Rdd and given a mini-batch B extracted from the training dataset D {[22, 9:) Let (0; B) denote the empirical loss, then the genetic formula- tion of ZO optimization follows Algorithm 1.
1) ZO gradient estimation. Randomized Gradient Estima- tion (RGE (Nesterov & Spokoiny, 2017)) and Coordinate- wise Gradient Estimation (CGE (Berahas et al., 2022)) per- turb the model using &, which is generated either from random unknown distribution (in RGE) by modifying in- dividual coordinates (in CGE), and then observe the changes in the loss function L after each perturbation, step by step, to provide a reliable gradient estimate. However, due to their reliance on slow single-direction perturbation, these meth- ods are not well-suited for deep learning tasks, as perform- ing full perturbation in high-dimensional parameter spaces is time-consuming For instance, typical vision models like ResNet trained on ImageNet have over 25 million param- eters. Performing per-dimension perturbations over such a large parameter space renders Z0-based querying highly inefficient. Standard Simultaneous Perturbation Stochastic Approximation (SPSA(Spall, 1992)) improves efficiency by generating pairs of symmetric vectors and perturbing in multiple directions simultaneously, as follows,
Our work. The works mentioned above are all rooted in a gradient feedback mechanism. Such mechanisms are powerless against catastrophic forgetting without explicit gradient information. Our work overcomes forgetting only via forward pass instead of gradient feedback.
A (() L(e + cE: B) - L(0 - cE: g-1 (I)
3. Exploring Zeroth-Order Optimization to Overcome Forgetting
26
Where is positive scaler and & is recommended to follow symmetric distribution with finite inverse moments (e.g.. the Rademacher distribution). The symmetric distribution ensures unbiased exploration of perturbations in both posi- tive and negative directions of parameters at each step. And the finite inverse moments property guarantees that the steps
C.1. Zeroth-Order Optimization
Zeroth-order (ZO) optimization has been extensively studied over the years within the realms of numerical computation and approximation algorithms. It functions as an alterna-
3

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
well-controlled, avoiding excessively large steps due to e-1 drawn from the distribution (e.g., E [I( for some large p), which would otherwise lead to an unstable opti- mization process. In practical implementations for models with a large number of parameters (e.g., Me70 (Malladi et al., 2023) in LLMs (Zhao et al., 2024b)), Gaussian noise with zero mean induces substantial perturbations, thereby enhancing exploration across the parameter space and fa- cilitating the escape from local minima. This methodology achieves gradient estimation with only two objective func- tion evaluations, rendering its computational cost inddpnn- dent of input dimensionality. Such computational efficiency has established SPSA as a preferred method for addressing the complexities of tigh-dimensional deep learning tasks. While increasing q in q-SPSA can improve stability in the update direction, setting q = 1 is sufficient for pretrained LLMs (Malladi et al., 2023).
nld
060
-10
-10
10
10 10
3
0
10
(a) FO-Adam
(b) ZO-Adam
Figure 3: Trajectory of FO and ZO Optimization during Overcoming Forgetting. The trajectory is taken when using the total loss from both tasks (cyan) and the gradients from each individual task fixed points during optimization (red and orange). The trends of ZO optimization hold the potential to manage forgetting and learning.
data based on learning progress.
2) Descent direction computation. In unconstrained op- timization for deep learning, the last gradients It gener- ally coincide with the estimated ZO gradients ÿi (e.g., ZO- SGD (Ghadimi & Lan, 2013), ZO-SCD (Lian et al., 2016)). To reduce approximation errors, ZO-SGD-Sign (Liu al., 2019) applies element-wise sign(.) operation. Addi- tionally, ZO-SVRG (Liu et al., 2018), inspired by variance reduction methods in first-order optimization, adjusts the update step by using estimated gradients from previous train- ing examples. CARS (Kim et al., 2021) adaptively selects the smallest function value in each iteration, which helps maintain monotonicity during optimization.
Liotal
Leur + (1
Creplays
(2)
Neentezt
Noontezt
where Neontert represents the number of contexts encoun- tered so far. In Experience Replay (Rolnick et al., 2019), both components use classification loss based on their re- spective data distributions, so ZO gradients can be expressed as Veer and Creplay respectively. However, in the emerging generative replay workflows (Shin al., 2017), Equation (2) may introduce additional loss for the training of generators. In this case, the generator can be trained using standard backpropagation or in conjunction with ZO training without FO gradients.
3) Parameter updating. Normally, for most 70 methods, parameters are updated in similar way with FO optimizers, and the learning rate 7): is set to constant. Except for the special design for achieving some constraint prerequisites, several methods make an effort to strike a balance between converge speed and accuracy. ZO-AdaMM (Chen et al., 2019) uses an adaptive learning rate and refines gradient es- timation by incorporating momentum from past information. This approach is particularly effective in handling complex and evolving optimization landscapes, where the function's 's behavior may vary over time or be hard to capture with straightforward gradient approximations.
ii) Extension based methods can be divided into fixed and dynamic architectures. Fixed architectures separate model parameters for specialized context learning, while dynamic architectures expand the model size during adaptation. Both approaches mitigate forgetting from the model's 's perspective and enable model-agnostic ZO solutions.
iii) Regularization-ban based methods penalize significant changes to parameters important for old tasks or maintain the output distribution with respect to previous inputs. The template loss function is given by
Liotal Leur + aCregs
(3)
C.2. Zeroth-Order Optimization for Catastrophic Forgetting
where is a coefficient hyperparameter. The FO gradients from dual objectives (Leur for adaptation and Lreg for preservation) drive optimization toward their respective op- tima, achieving inter-task equilibrium. Notably, Z0 gradient estimates, though obtained in noisy environment, exhibit comparable optimization behavior.
Rationality. ZO optimization leverages the function values of the forward passes to approximate FO gradients, making feasible to avoid gradient bans. This feature enables seamless integration into common forgetting scenarios, such as CL. We explore it in the following three categories.
As shown in Figure 3, we visualize and compare the op- timization trajectories of ZO and FO methods under the learning-memory memory trade-off dynamics in continual learning. The objective is defined over wwodiimensional parameters, with axes specified in Appendix A.2. The striking similarity
i) Memory-base based methods maintain a repository of exem- plars from previous tasks and dynamically adjust the overall loss function by combining these stored samples with new
4

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Table 1: ZeroFlow Evaluation on CIFAR-100, ImageNet-A, CUB and OmniBenchmark, This table compares average accuracy, final accuracy, and forgetting measures of 2 models, and 4 forgetting scenarios. More intuitive trend please see Figure All 70 optimizations use query budget of g = 1. Bold indicates the best accuracy achieved among ZeroF
CIFAR-100
CUB
ImageNet-A
Omnißenchmark
Method
Optimizer
Strategy
Avg
Last
Fgt
Avg
Last
Fgt
Avg
Last
Egt
Avg
Last
Egt
FO
91.23
85.96
7.32
89.31
83,76
9.61
61.24
51.02
10.84
74.73
67.40
15.11
ZO
78.62
68.40
15.64
88. 94
82.91
8.08
57.87
48.32
11.08
73.50
66.60
17.78
SGD
Sign
83.21
75,88
10.58
89.81
84.61
8.10
59.15
49.31
11.77
73.81
66.75
17.21
Conserve
82.22
75,88
8.93
89.21
83.42
10.31
58.61
48.58
12.41
77.07
70.73
14.87
EASE
FO
90.56
84,82
7.69
84.44
77.10
10.51
$9.60
47.20
19.08
74.27
66.28
15.63
Z0
83.36
76.09
10.16
89.49
84.14
8.67
58.90
48.72
12.35
76.15
69.69
15.87
Adam
Sign
83.14
76,01
10,44
89.82
84.65
8.21
58,97
48.85
12.20
77.12
71.08
14.68
Conserve
82.15
75.65
9.24
89.82
84.61
8.40
59.23
48.85
12.81
77.19
70.99
14.68
Forward
82,26
76.05
8.74
89.26
83.67
9.35
57.76
48.19
11.03
77.00
70.74
14.99
FO
82.31
76.21
7.33
90.56
85.16
5.19
59.50
49.37
9.91
78.61
72.21
7.87
ZO
82.33
76.21
7.36
90:53
85,20
5.12
59.58
49.51
10.02
78.60
72.21
7.85
SGD
Sign
82.32
76.23
7.32
90.42
85.28
4.96
59.65
49.77
9.89
78.60
72.26
7.78
Conserve
82.31
76,21
7.33
90.62
85,28
5.05
59.68
49.70
10.18
78.61
72.21
7.87
APER
FO
82.31
76.21
7.33
90.56
85.16
5.19
59.60
49.77
10.06
76.60
72.21
7.85
Z0
82.12
75.45
7.47
90.33
84.31
6.01
58.89
49.24
9.32
78.44
72.10
7.87
Adam
Sign
82,01
75.60
7.38
89.86
84,18
5.99
57,82
48.12
9.72
78.26
72.05
7.75
Conserve
82.21
75.98
7.34
89.96
84.48
5.90
57.86
47.53
10.00
78.61
72.21
7.87
Forward
82.32
76.22
7.32
89.47
83.38
6.24
58.25
47.99
9.62
77.61
71.45
7.87
between the two trajectories highlights the potential of ZO optimization in effectively balancing learning and forgetting, thereby motivating our further investigation.
takes 70.32 hours in DeepZero (Chen et al., 2023). Such computational demands render from scratch training imprac- tical for high-dimensional CL models, particularly those employing expansion- based architectures. Consequently, we focus our discussion leveraging ZO optimization to overcome forgetting within pre-training context.
Potential. The intrinsic optimization mechanism of ZO exhibits particular promise in continual learning scenar- ios. Intuitively, ZO perturbs parameters using random or coordinate-wis directional vectors and observes changes in the evaluation function, effectively optimizing within a noisy environment. This approach enables small parameter modifications to yield significant impacts on target objec- tives, resulting in distinctive gradient estimations compared to FO optimization. Notably, while ZO methods do not explicitly incorporate sharpness regularization terms, they naturally facilitate the exploration of flat regions in parame- ter space. The influence of optimizing flat regions with ZO approaches in continual learning can be summarized in two main manifolds: (i) For previous tasks, the noise-induced parameter robustness enhances resilience against perturba- tions from new task adaptation; (ii) For new tasks, empirical evidence suggests that convergence to flat minima generally leads to lower generalization error.
4. ZeroFlow Benchmark
This section delves into the empirical performance of 70 optimization in overcoming catastrophic forgetting. Our ZeroFlon benchmark evaluates average performance across incremental stages, final-stage accuracy, forgetting, and efficiency, while accounting for dataset complexity and model diversity.
D.1. Benchmark Setups
Forgetting scenarios, schemes, and models. We conduct evaluations under a standard catastrophic forgetting setting, namely class incremental learning. For this purpose, we investigate two state-of-ther tht--artt schemes: EASE and APER. Both models are initialized with ViT-B/16 pretrained on ImageNet- IK (INIK), and are subsequently fine-tuned on four downstream tasks of varying complexity-raniing from standard benchmarks such as CIFAR-100 and CUB, to more challenging datasets like ImageNet-A and Om- niBenchmark, which exhibit a large domain gap from the pretraining distribution (Zhou et al., 2024a;c). Follow- ing (Zhou et al., 2023a), each dataset is evenly split into 10
Risk. Although 7.0 demonstrates superior generalization abilities, its practical performance is limited by optimiza- tion strategies and the complexity the optimization setting. Despite significant efforts to reduce convergence error, op- timizing models from seratch in high-dimensional space remains challenging due to slow convergence speed (propor- tional to the parameter dimension d). For instance, origin CGE-based ZO training for model with 12k parameters
5

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Dew
-10
-10
10
-10
-10
10
-10
-5
10
-10
-10
-5
10
(a) FO-Adam
(b) ZO-Adam
(c) ZO-Adam (q
(d) ZO-Adam- -add--iiiin
(e) -70-edem-C
10
-10
10
-10
110
10
(f) FO-SGD
(g) ZO-SGD
(h) 20-00-SGD
(i) ZO-SSD--Sig (i)
Figure 4: The Trajectory of Different Optimization during Overcoming Forgetting. and denote the minima for the new, old, and both tasks, respectively. The trajectory is taken when using the total loss from both tasks (cyan).
incremental tasks by class. For instance, OmniBenchmark contains 300 classes, with 30 classes introduced each stage. No memory is permitted for storing past examples.
based (FO-SGD and FO-Adam) methods in typical forget- ting scenario (continual learning). We use two SOTA models examples (EASE (Zhou et al., 2024b) and APER (Zhou et al., 2023a)) and investigate SGD and Adam optimizers, forward
Benchmark setup and details. To evaluate the applica- tion of VeroFlow in forgetting scenarios, we include the methods described in Section C.I. specifically ZO (Ghadimi Lan, 2013), Sign (Liu et al., 2019), and Conserve (Kim al., 2021: Zhang al., 2024), in comparison with their FO counterparts using SGD) and Adam optimizers (Chen et al., 2019). Additionally, highlighted in (Zhang et al., 2024), Forward- (Baydin et al., 2022) which relies on forward mode automatic differentiation, potentially be- comes a missing but competitive forward pass baseline. a nutshell, ZeroF low covers 7 forward pass-based methods: ZO-Adam,
methods, and four commonly used datasets. Several observations are listed below,
First, the performance of ZO method is comparable to or even surpasses that of the FO method across almost all for- getting metrics and datasets. However, will be shown later, the FO method requires significantly more memory overhead. This suggests that forward passes alone can ef- fectively mitigate forgetting, and the ZO method offers a simpler, more efficient alternative. In some cases, such as with ZO-Adami and ZO-SGD on OmniBenchmark, ZO methods even outperform FO methods.
ZO-SGD, ZO-SGD-Siee Sign, Z0-Adam- 20-Adam-Conserve Forward-Gres Un- less otherwise specified, the query budget is fixed to for efficiency. Notably, here we consider generating one set of perturbation vectors for the entire model as one query. In other words, we usually require forward propagations for two-point finite difference gradient estimations.
Second, Forward Grad demonstrates competitive perfor- mance when compared to other ZO and FO methods. Un- like typical ZO methods, Forward Grad utilizes a unique forward pass mechanism, making it a promising baseline for future studies. more intuitive trend overcoming forgetting refer to Figure 6. These observations motivate further exploration into the effectiveness of ZO method.
Evaluation metries. Overall, adopt two categories of evaluation metrics in ZeroFlow: accuracy and efficiency. The accuracy metrics include average accuracy across all tasks, final-task accuracy, and forgetting score (BWT in Appendix B.5). The efficiency metrics encompass memory usage (GPU), query budget, and runtime. Together, these metrics provide insights into the resource demands ZO optimization for mitigating forgetting.
KeroFlow helps manage memory and runtime. In Ta- ble 2, we compare the efficiency of various 70 and FO optimizers in mitigating catastrophic forgetting, focusing on two key aspects: memory cost (in GB) and runtime cost (in seconds). First, naive ZO optimization reduces memory usage by approximately fivefold compared to FO optimization. Moreover, ZO methods reduce runtime per iteration by around 50% relative to FO, significantly improv- ing their practicality for overcoming forgetting. Notably, we regenerate the perturbation vectors for model parameters iteratively by storing random seeds. This degrades the vec-
D.2. Evaluation Results of ZeroFlow
ZeroFlow evaluation on continual learning. In Table 1, we evaluate the performance of different BP-free and BP-
6

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Table 2: Memory Cost (GB) and Runtime Cost (s) of Each Optimizer on 3 Forgetting Scenarios. The per-epoch runtime in seconds (s). ZO-SGD w/ query budget q = 1, and all other optimizers w/ query budget q
Porward
Forword
1.0
700-admm- Sim
0.
SiD-Sign Z0-Adam- Si
-SDD-Sig
0.3
0.3
022
Z0-Adan
0-SGD ZD-Adan
G-SGD
Optimizer
Memory 4 CIFAR- 100
CUB ImageNet-A
FO-SGD
12.08 GB
59.3s
16.1s
12.2s
20-Adan-Ca Con
20-Adan- SGD-Cou
Cot
SGD-Cuus
ZO-SGD (g- = 1)
2,41 GB
32.4s
8.3s
6.8s
CIPAR-100
CUB
ImageNet-A
OmniBeehharkk
ZO-SGD (q = 4)
2.41 GB
111.7s
28.7s
18.0s
ZO-SGD-Sige
2,41 GB
32.4s
8.3s
6,8s
(a) EASE last accuracy
(b) APER last accuracy
D0-SGD-Coonserve
2.41 GB
70.
15.7s
12.4s
Figure 6: ZeroFlow Evaluation Results for Forgetting.
Forward-Grad
3,94 GB
45.9s
11.1s
9.0s
We visualize the evaluation of 2 models in last-task accuracy.
60.0
14.0
increase across {1,2,4,8,16.32 suggesting that additional function evaluations enable more accurate gradient estima- tion. The results suggest that in scenarios where function evaluation costs are manageable, higher query numbers can yield substantially better performance, with Adam being particularly effective at leveraging the additional gradient information for enhanced optimization outcomes.
SGD Avg
SGD Fgt
59.5
Adam_Avg
Adam_ _igt
13.5
Average accuracy (%)
59.0
13.0
Fongeting
58.5
12.5
58.0
12.0
57.5
11.5
5. Insights and Discussions
57.0
11.0
16
32
As shown in Figure 4, we visualized the optimization trajec- tories of both forward passes and backpropagation methods. Our analysis reveals several key insights:
Query number
Figure 5: Performance Comparison under Different Query Mumbers. Both optimizers show improved per- formance as query numbers increase.
Convergence behavior across optimizer families. In Fig- ure 4, both FO and ZO methods demonstrate successful con- vergence to the minima of new and old knowledge spaces, regardless of whether they use Adam or SGD as their base optimizer. This convergence consistency validates our theo- retical foundation.
tor granularity from full-model to per-layer level, thereby further reducing the memory required for forward evalua- tions in ZeroFlow, at the cost of additional runtime for regenerating the vectors. Second, the ZO and Sign variants demonstrate comparable efficiency in both memory and run- time. Although increasing the number of queries impact runtime efficiency, it does not compromise memory advan- tages. Third, Conserve also demonstrates efficient memory management, although its runtime is approximately twice as long as that of naive ZO. This may partly explain its stronger performance in some scenarios, as shown in Table 1. Fi- nally, the Forward Gradient method requires more memory than other ZO-based approaches because it involves comput- ing gradients via the Jacobian-vector product (JVP), which necessitates storing all intermediate activations during the forward pass. For models like VIT, this includes large at- tention maps and other intermediate representations. In contrast, naive Z0 methods only require two forward passes on perturbed inputs and avoid storing these intermediate values, resulting in much lower memory usage.
Distinet trajectory characteristics of FO and ZO. FO ap- proaches (Figure 4a, 4f) show smoother optimization paths due to their access to exact gradient information. In contrast, ZO methods demonstrate varying degrees of exploration behavior through trajectory jitter. This exploration pattern is particularly pronounced in ZO-Adam variants compared to ZO-SGD variants, indicating that the base optimizer choice significantly influences the exploration exploration-exploitttio exploitation trade- off during optimization.
Path characteristics in Zo optimization. Comparing base ZO methods with their q 4 counterparts (Figure 4b vs 4c, Figure 4g vs 4h), we observe that increasing query num- bers leads to smoother trajectories, suggesting that more queries help provide more stable gradient estimates. The Sign variants (Figure 4d, 4i) demonstrate more pronounced oscillations in their trajectories, particularly visible in the ZO-Adam-Sign case. In contrast, the conservative variants (Figure 4e, 4j) maintain relatively stable paths that better balance between the old and new task minima.
Trade-off between performance and query number. As shown in Figure 5, we investigate the impact of query num- bers on optimization performance, comparing SGD and Adam optimizers in the zeroth- order setting. Both optimiz- ers demonstrate improved performance as query numbers
Distinct characteristics between optimizer families. Adam-based approaches (Figure 4a-4e) demonstrate more

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Task 1
Task
Task 3
Task 4
Task 5
80
FO
Accuracy (%)
ZO,, 170E
Z0, 140E
60
40
50
100 Epoch
150
2000
50
100
150
2000
50
100 Epoch
150
2000
50
100
150
2000
50
100
150
200
Epoch
Epoch
Epoch
Figure 7: Effectiveness of Hybrid Zo in Overcoming Forgetting, In Hybrid ZO, backward benefits from forward passes.
Table 3: Effectiveness of Historical Estimation in Miti-
20
gating Forgetting, Proportion of 0% denotes that the plain optimizer ZO-SGD. Bold indicates the best performance.
Function value
Funption value
0.5
1.0
20
Proportion
Metrics
-1.5
0%
20%
40%
60%
80%
10K
30K
Steps
Steps
Avg
57.87
58,90
58.76
58.34
57.83
(a) FO-SGD
(b) ZO-SGD
Last
48.32
49.04
48.84
48.42
48.10
Fgt
11.08
11.79
11.78
11.60
11.57
Figure 8: Variation in Function Values of Forward Passes. Function values for new tasks is highlighted in red, old tasks is highlighted in green.
oscillatory trajectories with frequent direction adjustments, indicating more dynamic exploration of the loss land- scape. In contrast, SGD-based methods (Figure 4f- hibit smoother and more stable trajectories, suggesting more gradual progression toward the optimization objective. These distinct optimization patterns could influence how each method balances between preserving old task knowl- edge and adapting to new tasks.
hybrid ZO. We first use FO coarsely optimize local minimum (first 140 or 160 epochs) and then refine the solu- tion by searching for flatter regions around it using ZO (last 30 or 60 epochs). As the first two subfigures in Figure 7, ZO provides only limited gains to FO. This is because FO inher- its strong generalization from the pretrained backbone but loses its generalization ability quickly after two incremental stages. In later stages, ZO helps remedy the vulnera- bilities of backbone trained by FO, leading to significant enhancements compared to the FO baseline.
6. New Enhancement to Mitigate Forgetting
In ZO optimization, the estimation of the gradients relies on finite difference of the objective function. We set query budget = 1 in the benchmark for efficiency. However, limited queries cannot capture the accurate ZO directions. When the model learns tasks sequentially, the high variance inherent in Z0 gradient estimation poses critical chal- lenge. Though increasing query numbers can stabilize the gradient estimates, it leads to prohibitive overhead Thus, ex- ploring cariance-reduced optimization algorithms is crucial for 20-based CI. Specifically, we propose enhancements to stabilize the ZO optimization process:
Enhancement 2: Leverage historical information to over- come forgetting. When learning new tasks, models lever- age previously learned parameters while prioritizing the preservation crucial parameters for old tasks. To mitigate interference from new tasks, propose reweighting old task gradients with historical gradients, which can stabilize perturbations caused by low query loops in ZO optimiza- tion. Figure 8 illustrates the function value trajectories for both old and new tasks. While FO optimization shows smooth convergence toward the global optimum, ZO opti- mization exhibits a more volatile path. Notably, objectives related to old tasks demonstrate smaller changes in both magnitude and variance. This observation motivates to stabilize the optimization by reducing changes to old gradi- ents through linear combination with historical gradients: Jold = (1 - a) + O'Dhistoricat- where larger a indicates greater reliance on historical information for stability, at the cost of reduced contrast with new task gradients.
Enhancement Hybrid ZO to overcome forgetting. While ZO methods does not explicitly minimize sharpness, it stabilizes optimization by approximating gradients and assessing the rate of change in loss function through pertur- bations. This indirect approach helps reduce the curvature of the loss landscape, steering the optimization away from sharp and unstable regions. This insight motivates us to investigate Hybrid ZO method. Figure? illustrates results
8

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Table 4: Effectiveness of Sparsity-induced Estimation in Overcoming Forgetting, Proportion of 0% denotes the plain ZO-SGD. Bold indicates the best performance.
Ratio
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Avg
57.87
59.17
59.46
59.29
59.39
59.45
59.26
59.39
59.38
59.47
Last
48.32
48.58
49.05
48.72
48.91
49.24
49.11
49.05
49.11
49.24
Fgt
11.08
12.65
12.17
12.76
12.53
12.37
12.36
12.54
12.46
12.33
Table 5: Ablation Studies on the Effectiveness of Combining Enhancements.
Optimizer
Hybrid
Historical
Sparsity
Avg
Last
FO-SGD
61.24
51.02
57.87
48.32
61. 61.40(+3.53)
51.34(+3.02)
70-SGD
58.90(+1.03)
49.04(+0.72)
59.47(+1.60)
49.24(+0.92)
62.07(+4.20)
51.94(+3.62)
In Table 3, validate the effectiveness of historical esti- mation in mitigating catastrophic forgetting. Modest pro- portions of historical information (e.g., 20%, 40%, 60%)) outperform ZO-SGD (0%), effectively controlling perturba- tions while maintaining a low query budget (q = 1).
fluctuations sometimes facilitate escaping local minima and encourage broader exploration, which in turn benefits FO convergence. Furthermore, incorporating historical gra- dients and sparsity perturbations contributes to mitigating forgetting and stabilizing the optimization process.
Enhancement 3: Sparsity-induced estimation helps to overcome forgetting. In ZO optimization, the gradients for new tasks often highly uncertain due to the approx- imation nature of the gradient estimation. To reduce this variance, we implement random sparsification by creating a seed-based mask and setting gradients outside the mask to zero. By reducing the number of non-zero gradient com- ponents, we aim to stabilize the optimization process and mitigate the noise in gradient updates.
7. Conclusion
This paper introduces ZeroFlow, benchmark study that probes a series of forward pass-based methods for overcom- ing catastrophic forgetting. This work resorts to an easier way (no need for backpropagation and activation storage) to overcome forgetting. Concretely, our benchmarks include various forward pass-based methods, forgetting scenarios, and evaluation metrics. We also reveal the overlooked opti- mization principles for overcoming forgetting via forward passes. Based on these insights, we propose two easier and better enhancement to overcome forgetting and extend the application of related methods easily.
In Table 4, we report the performance of sparsity- -nnducy--nduced zo in overcoming forgetting. The sparsity level is varied in this experiments, ranging from 10% to 90%. We observe that the sparse technique improves the average and last ac- curacy across all scales, which implies that forgetting is effectively controlled. The reduction in volatility can be attributed to the sparse strategy yielding smoother gradient estimates compared plain ZO-SGD, effectively bounding variance to low level and thus mitigating forgetting. More- over, the robust performance across different sparsity ratios provides strong evidence for the efficacy of variance control in addressing forgetting.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Complementary Enhancements: The results in Table 5 demonstrate that the proposed enhancements are not mutu- ally exclusive and can be effectively integrated. Specifically, FO training can substantially benefit from subsequent fine- tnning with hybrid ZO optimization, as illustrated in Fig- ure Notably, the inherent instability of ZO with large step
A cknnoleegmennts
This work was supported in part by the National Natural Sci- ence Foundation of China (NSFC) under Grant 62495063, This work was supported in part by the China Postdoctoral Science Foundation under Grant 2024M761677.
9

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
References
Feng, T., Yuan, H., Wang, M., Huang, Z., Bian, A., and
Zhang, J. Progressive learning without forgetting. arXiv 2022c.
Aojun, L., Hangjie, Y., Tao, F., and Yanan, S. Rethinking stability-plasticity plasticity trade-off in continual learning from the stability- an architectural perspective. ICML, 2025.
preprint arXiv:
Flaxman, A. D., Kalai, A. T., and McMahan, H. On-
line convex optimization in the bandit setting: gradient descent without gradient. arXiv preprint cs/0408007, 2004.
Baydin, A. G., Pearlmutter, B. A., Syme, D., Wood, F., and Torr, P. Gradients without backpropagation. arXiv preprint ar arXiv:2202.08582 2022.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Berahas, A. S., Cao, L., Choromanski, K., and Scheinberg, K. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foun- dations of Computational Mathematics, 22(2):507-560, 2022.
Gan, W., Wan, S., and Philip, S. Y. Model-as-a-service (maas): survey. In 2023 IEEE International Conference Big Data (BigData), 2023.
Bergou, E. H., Gorbunov, E., and Richtarik, P. Stochastic three points method for unconstrained smooth minimiza- tion. SIAM Journal on Optimization, 30(4):2726-2749, 2020.
Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 2013.
Bian, A., Li, W., Yuan, H., Yu, C.,, Wang, M., Zhao, 7.,, Lu,, A., Ji, P., and Feng, T. Make continual learning stronger via c-flat. NeurlPS, 2024.
Gu, J., Zhu, H., Feng, C., Jiang. Z., Chen, R., and Pan, D. L2ight: Enabling on-chip learning for optical neural networks via efficient in-situ subspace optimization. Ad- vances in Neural Information Processing Systems, 2021.
Cha, S., Hsu, H., Hwang, T., Calmon, F. P., and Moon, projection regularization for continual
T. Cpr: classifier classifier-projection learning. ICLR, 2021.
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, Embrac- ing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-1000 1040, 2020.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. arXiv preprint arXiv: 1111.812200022 2018.
He, H., Huang, G., and Yuan, Y. Asymmetric valleys: Be- yond sharp and flat local minima. NeurlPS, 32, 2019.
Chen, A., Zhang, Y., Jia, J., Diffenderfer, J., Liu, J., Parasyris, K., Zhang. Y., Zhang, Z., Kailkhura, B., and Liu, S. Deepzero: Scaling up zeroth-order op- timization for deep model training. arXiv preprint arXiv:2310.02025, 2023.
Hinton, G. The forward-forwan algorithm: Some prelim- inary investigations. arXiv preprint arXiv:2212.13345, 2022.
Jabri, M. and Flower, B. Weight perturbation: An opti- mal architecture and learning technique for analog vlsi feedforward and recurrent multilayer networks. IEEE Transactions on Neural Networks, 1992.
Chen, X., Liu, S., Xu, K., X., Lin, X., Hong, M., and Cox, D. Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization. NeurIPS, 32, 2019.
Jeeveswaran, K., Bhat, P., Zonooz, B., and Arani, E. Birt: Bio-inspired replay in vision transformers for continual learning. ICML., 2023.
Deng, D., Chen, G., Hao, J., Wang, Q., and Heng, P.-. Flat- tening sharpness for dynamic gradient projection memory benefits continual learning. NeurlPS, 34, 2021.
Kim, B., Cai. H., McKenzie, D., and Yin, W. Curvature- aware derivative free optimization. arXiv preprint arXiv:2109, N00:2119.13391, 2021.
Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp- 3762-3773. PMLR, 2020.
Kong. Y., Liu, L.,, Chen, H., Kacprzyk, J., and Tao, D. Over- coming catastrophic forgetting in continual learning by exploring eigenvalues of hessian matrix. IEEE Transac- tions on Neural Networks and Learning Systems, 2023.
Feng, T., Ji, K., Bian, A., Liu, C., and Zhang. J. Identifying players broadcast videos using graph convolutional network. Pattern Recognition, 124:108503 2022a.
Li, W., Feng, T., Yuan, H., Bian, A., Du, G., Liang, S., Gan, J., and Liu, Z. Unigrad-fs: Unified gradient projec- tion with flatter sharpness for continual learning. JEEE Transactions on Industrial Informatics, 2024.
Feng. T., Wang, M., and Yuan, H. Overcoming catastrophic forgetting in incremental object detection via elastic re- sponse distillation. In CVPR, 2022b.
10

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Lian, X., Zhang, H., Hsieh, C.-J., Huang, Y., and Liu, A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first- order. Advances in Neural Information Processing Sys- tems, 29, 2016.
Nesterov, Y. and Spokoiny, V. Random gradient-free mini-
mization of convex functions. Foundations of Computa- tional Mathematics, 2017.
Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. arXiv preprint arXiv:1904.092: 2019.
Lillicrap. T. P., Santoro, A., Marris, L., Akerman, C. J., and Hinton, Backpropagation and the brain. Nature Reviews Neuroscience, 2020.
Ren, M.. Korbllihh, S., Liao, R., and Hinton, G. Scal- ing forward gradient with local losses. arXiv preprint arXiv:2210.03311 2022.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. NewrIPS, 2021.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Liu, S., Kailkhura, B., Chen, P.-.., Ting, P., Chang, S., and Amini, L. Zeroth-order stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2020.
Liu, S., Chen, P.- Chen, X., and Hong. M. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2019.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. NeurIPS, 2017.
Shi, G., Chen, J., Zhang, W., Zhan, L.-M., and Wu, X.- M. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. NeurIPS, 2021.
Lu, A., Feng, T., Yuan, H., Song, X., and Sun, Y. Revisiting neural networks for continual learning: An architectural perspective. IJCAI, 2024.
Shi, H., Xu, Z., Wang. H., Qin, W., Wang, W., Wang, Y. Wang, Z., Ebrahimi, S., and Wang, H. Continual learning large language models: A comprehensive survey. arXiv preprint arXiv:2404. 16789, 2024.
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting large language models during continual fine-tuning. arXiv preprint XXi:22308.08747 2023.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural informa- tion processing systems, 30, 2017.
Malladi, S., Gao, T. Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning large language mod- els with just forward passes. NeurlPS, 2023.
Spall, J. C. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332-341, 1992.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter- efficient fine-tuning methods. https://githe: hom/hugg' i inngfacce face/ /eft, 2022.
Sun, H.-I.., Zhou, D.-W., Ye, H.-J,, and Zhan, D.-C. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.0711 2023a.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and Van De Weijer, J. Class-incremental learning: survey and performance evaluation on image classifica- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Sun, M., Wang, Y., Feng, T., Zhang, D., Zhu, Y., and Tang, J. A stronger mixture of low-rank experts for fine-tuning foundation models, 2025.
Mehta, S. V., Patil, D., Chandar, S., and Strubell, E. An empirical investigation of the role of pre-training in life- long learning. J. Mach. Learn. Res., 24:214:1-214:50, 2023. URL https: /
Sun, T., Shao, Y., Qian, H., Huang, X., and Qiu, X. Black- box tuning for language-modee www-e--d-----e--iiiiiie In Interna- tional Conference on Machine Learning, 2022.
Sun, Z., Mu, Y., and Hua, G. Regularizing second-order influences for continual learning. In CVPR, 2023b.
22-0496.. 000446.hee html.
Miura. T., Shibahara, T., and Yanai, N. Megex: Data-free model extraction attack against gradient-based explain- able ai. In Proceedings of the 2nd ACM Workshop on Secure and Trustworthy Deep Learning Systems, 2024.
Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masque- lier, T., and Maida, A. Deep learning in spiking neural networks. Neural networks, 2019.

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Tung, L. T., Van, V. N., Hoang, P. N., and Than, K. Sharp- ness and gradient aware minimization for memory-based continual learning. In Proceedings of the 12th Interna- tional Symposium on Information and Communication Technology, SOICT. ACM, 2023.
Zhong, Q., Ding, L.., Shen, L.,, Mi, P., Liu, J., Du, B., and
Tao, D. Improving sharpness-aware minimization with fisher mask for better generalization language models. arXiv preprint arXiv:2210.05499, 2022.
Zhou, D.-W., Cai, Z.-W., Ye, H.-J., Zhan, D.-C., and Liu, Z. Revisiting llass-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.073338 2023a.
van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. Three types of incremental learning. Nature Machine Intelli- gence, pp. 1185-1122 1197, 2022.
Wang, L., Zhang, X., Su, H., and Zhu, A comprehen- sive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 2023.
Zhou, D.-W., Wang, Q-W., Qi, Z.-H,, Ye, H.-., Zhan, D.-C., and Liu, Z. Deep class-incremental learning: A survey, arXiv preprint arXiv:2302.03644: 2023b.
Wang, L.. Zhang, X., Su, H., and Zhu, comprehen- sive survey of continual learning: Theory, method and application. TPAMI, 2024.
Zhou, D.-W., Wang, Q.-W., Ye, H.-J., and Zhan, D.-.. model or 603 exemplars: Towards memory class- incremental learning. ICLR, 2023c.
Yang, E.,, Shen, L.,, Wang, Liu, S., Guo, G., and Wang, X. Data augmented flatness- aware gradient projection for continual learning. In IEEF/CVE International Confer- ence on Computer Vision, 2023.
Zhou, D.-W., Sun, H.-L.,, Ning, J., Ye, H.-J., and Zhan, D.-C. Continual learning with pre-trained models: A survey. In IJCAI, pp. 8363-8377 1033-331, 2024a.
Zhou, D.-W., Sun, H.-L., Ye, H.-J., and Zhan, D.-C. Expandable subspace ensemble for pre-trained model- based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Yang, E., Wang, Z., Shen, L.., Yin, N., Liu, T., Guo, G., Wang. X., and Tao, D. Continual learning from a stream of apis. IEEE Transactions on Puttern Analysis and Ma- chine Intelligence, 2024.
Yuan. H., Jiang, J., Albanie, S., Feng. T., Huang, Z., Ni, D., and Tang, M. Rlip: Relational language-iman image pre-training for human-object interaction detection. In NeurlPS, 2022.
Zhou, D.-.,, Wang, Q-W., Qi, Z.-H., Ye, H.-J., Zhan, D.- C., and Liu, Z. Class-incremental learning: survey, IEEE Transactions on Pattern Analysis and Machine In- telligence, 2024c.
Yuan, H., Zhang, S., Wang, X., Wei, Y., Feng, T., Pan, Y., Zhang, Y., Liu, 7.,, Alhanie, S., and Ni, D. Instructvideo: instructing video diffusion models with human feedback. In CVPR, 2024.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. ICML. 2024a.
Zhang, D., Feng, T., Xue, L., Wang, Y., Dong, Y., and Tang, J. Parameter-efficient fine-tuning for foundation models. arXiv, 2025.
Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng. Y. Llama-moe: Building mixture-of-expprt from llama with continual pre-training. arXiv preprint 24ti:22006.15554 2024b. URL https:// /ttps://arxiv.. org/
Zhang, Y., Li, P., Hong, J., Li. J.. Zhang, Y., Zheng, W., Chen, P.-Y., Lee, J. D., Yin, W., Hong, M., Wang, Z., Liu, S., and Chen, T. Revisiting zeroth-order optimiza- tion for memory-efficient LLM fine-tuning: A bench- mark. In Forty-first International Conference on Machine net/
Zhuang, H., Weng, 2., Wei, H., Xie, R., Toh, K.-A,, and Lin, Z. ACIL: Analytic class-incremental learning with abso- lute memorization and privacy protection. In NeurIPS. 2022a.
Learning, 2024. URL ptttps://oppereeeeee forum? d=THP MM.2.......
Zhuang, H., Weng. Z., He, R., Lin, Z., and Zeng, Z. GKEAL: Gaussian kernel embedded analytic learning for few-shot class incremental task. In CVPR, 2023.
Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Zhang, K., Xu, S., Chen, D., Timofte, R., and Van Gool, L. Equivariant modti-mydality image fusion. In CVPR, 2024a.
Zhuang. J., Gong, B., Yuan, L., Cui, Y., Adam, H., Dvomek, N., Tatikonda, S., Duncan, J., and Liu, T. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXXv:2203.08065, 2022b.
Zhao, Z., Deng, L., Bai, H., Cui, Y., Zhang, Z., Zhang, Y., Qin, H., Chen, D., Zhang, J., Wang, P., and Gool, L. V. Image fusion via langualan model. In ICML, 2024b.
12

---

ZeroFlow: Overcoming Catastrophie Forgetting is Easier than You Think Supplementary Material
A. Experimental Details
In this section, we provide overview of zeroth-order optimization algorithms and the function settings used for the trajectory analysis.
A.1. Concise Overview of Zeroth-Order Estimation
Zeroth-order optimization aims mininize/maximize an objective function f RR - R. without derivative information. The core problem is formulated minøeR" L(0), where 9 denotes the optimization variable. To enable gradient- tradient-based updates, Simultaneous Perturbation Stochastic Approximation (SPSA(Spall, 1992)) is a commonly used technique to approximate gradients by perturbing the input variables. Specifically, the gradient (()) at point is estimated as:
B) I((O + cE; B) - T[O - <E; B) -
(4)
2e
where & ~ (0, I) is random perturbation vector, and 0 is small perturbation step size (typically adjusted during optimization).
ZO-SGD(Ghadimii & Lan, 2013); Using the gradient estimator (0, follow the iterative update rule:
E; B). zeroth-order algorithms, such as ZO-SGD,
0.11 = 0, - T7
Ex: Ec:
(5)
where 7/ is the learning rate at step t. ZO-SGD bypasses explicit gradient computation through local function evaluations, making it suitable for digeenimesii non-convex optimization problems.
ZO-SGD-Sign(Liu et al., 2019): A variant of ZO-SGD, known ZO-SGD-Sight improves upon the original approach by approximating the gradient direction using the sign of the gradient estimate. The update rule becomes:
o,+1 02 Th sign (L,, Ee; B)),
(6)
where sign(.) denotes the element-wise sign function. This approach often leads to faster convergence some problems where the magnitude of the gradient is not as important its direction.
et al., 2020): ZO- ZO-SGD-COne GGD-Conserve is another variant that conservatively selects the update direction by locally comparing three candidate points, rather than directly committing to a single gradient step. The update rule for this method is:
ZO-SGD-Conserv(Beeeee
Oc+1 arg min (v)), C. = {0r, 0: VL,,,, Ess B), 0, + tp &L yEC,
This method mitigates overly aggressive updates by evaluating possible directions and choosing the one that locally minimizes the objective function.
ZO-Adam(Zhang al., 2024): ZO-AdaMM (Chen et al., 2019) the first attempt apply the Adam family (specifically AMSGrad(Redas et al., 2019)) zeroth-order (7O) optimization algorithms, providing convergence guarantees for both convex and nonconvex settings. The update rule is given by:
0++1 = 02 T)t
mh
V. Diag(max(Vi,:2
(8)
VV E
mr 31m4-1 + (1-8I) E;; B),
0 1-11-1 + (1- B2) (V Er; B))²
13

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
In our implementation, we simply replace SGD with Adam for convenience, referring this variant as ZO-Adam, Neverthe- less, also provide reference implementation of the original oracle Z0-AdaMM algorithm.
Forward Gradient Descent (FGD)Baydinn et al., 2022): FGD replaces backpropagation with forward- automatic differentiation to estimate gradient directions using aacobian-vector products (JVPs), Instead of computing full gradients via reverse-mode automatic differentiation (AD), FGD samples probe vectors to construct unbiased estimators of the gradient direction. A typical FGD update step is:
(f) d0
O2+1 1 (v)) 02 -/t-"
where is random probe vector (e.g., Rademacher or Gaussian), and J (v)) represents the forward-mode gradient approximation in direction FGD enables training when reverse-mode AD is impractical or unavailable, and offers flexibility for hardware or software systems that only support forward execution. We denote Forward as FGD throughout this paper.
A.2. Function Settings
Following the setup in CAGrad (Liu et al., 2021), we visualize the optimization behavior of first-order (FO) and zeroth-order (ZO) methods in mitigating forgetting. Specifically, we consider a -ioeeimnniiona parameter space 0 = (01, 82) E RR, with the following task-specific loss functions: L, (0) ei (0)s((0) + (0) for the old task (orange), and l.2(0) ei (0) e2 (0)g:(0) for the new task (red). The parameter point is initialized at [-8.5, -5] be closer old tasks, facilitating better adaptation to them. The contour plot in Figure illustrates the overall objective function defined L(0) L1 (0) + (0). where the r- and y-axes correspond to 01 and 0g. respectively.
(0) = log (max (10.5(-0) 01 - (ann(a00)) 5 x 10-9) + 6,
(2(0) = log (max (10.5( ( 01 + 3) (aanh--02) 2) 5 x 10 10-9) + 6,
(0) = (-81 7)2 + 0.1 8)²
20.
10
g2 (-8, 7)2 + 0.1(02 8)2
20,
10
ci (0) max (tanh(0.5 02), 0), (2 max (tanh(-O0 (00.5 02), 0).
B. Additional Results
B.1. Comprehensive Analysis of Memory Usage on ZeroFlow
120
In this subsection, we provide a detailed comparison of memory usage during incremental learning to demon- strate the storage efficiency of ZeroFlow (ZO-SGD) compared to its counterpart, FO-SGD, Figure 9 illustrates the peak memory usage of MEMO when trained on the CIFAR-100 dataset. The backbone is fixed pretrained ViT-B/16 VIT-B/16-INIK model, which is subsequently fine-tuned with batch sizes ranging from 64 to 512. The experimental results highlight the following observations:
Memory usage (GB)
100
size Boundary
80
60
40
20
First, doubling the training batch size significantly in- creases the memory consumption of FO-SGD, requiring more GPU resources. For instance, completing the entire incremental training process on FO requires 1, 2, 3, and GPUs, respectively, for batch sizes of 64, 128, 256, and 512, with each GPU equipping with 24GB of memory. In contrast, ZO-SGD training consistently requires only one GPU resource.
Task
Figure 9: Comparison of Memory Usage between FO-SGD and ZO-SGD with Different Batch Sizes. A denotes the increase in memory usage of the final task compared to the initial one.
14

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Second, as training progresses, the memory demands for larger batch sizes increase rapidly. For FO, the memory consumption for 512 batches stage grows by 30.08 GB compared the initial stage. In contrast, ZO-SGD shows modest increase of only 3.92 GB, maintaining low growth rate. As training advances, the memory efficiency of ZO-SGD becomes more pronounced, especially for model- -edellexppnsioo based CL models.
B.2. More Observations Optimization Trajectories during Overcoming Forgetting
10 DeW old
10
10
10
10
new old
new old
Dew old
new
old
5
5
5
5
0-
0-
-
5
-5
-
-5
10
10
10
10
10
10
5
5
10
10
5
0
10
10
5
5
10
5
0
5
10
10
-5
0
10
(a) FO-Adam
(b) ZO-Adam
(c) ZO-Adam (q
(d) ZO-Adam-Sign
Adam-Conserve
10
new
new
10
old
old
-5
-5
-5
-5
-10
-10
-10
-10
-
5
10
5
0
10
10
-
5
10
10
10
-5
10
(f) FO-SGD
g ZO-SGD
(h) 20-SGD (g
4)
ii ZO-SGD-Sign
() 100-SGDDCConsee
10
10
10
10
10
DCW
new
new
DOW
new
old
old
old
old
5
5
0
0-
0
-5
-5
-5
-5
10
10
10
-10
-10
10
-5
5
10
10
5
0
10
10
-5
5
10
10
-
0
5
10
10
-5
0
10
(k) FO-Adam
()) ZO-Adam
(m) Z0-Adam (q = 4)
(n) ZO-Adam-Sigh
(0) Adam-Conserve
10
10
10
10
10
Dew old
new old
new old
Dew old
new
old
5
5
5
0
0
-5
-5
-5
-5
-5
10
10
10
10
10
-
5
10
10
5
0
10
10
-5
5
10
5
0
5
10
10
-5
0
10
(p) FO-SGD
(q) ZO-SGD
ZO-SGD (g
ZO-SGD-Sign
(1) -70-SGD-Conserve
Figure 10: The Trajectory of Different Optimization during Overcoming Forgetting. The first and last two rows trained for 100k steps with learning rates of 0.001 and 0.01, respectively. Red denotes the minimum of new task, orange denotes the minimum of old task. The cyan trajectory taken when using the total loss from both tasks.
In this subsection, we present a different scenario where the model initialized at a local minimum 61, 82 = (-4.0, 5.0). surrounded by intricate valleys, but training with different learning rate as shown in Figure 10. For learning rate of
15

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
0.001, the first-row subfigures demonstrate that Adam using both FO and ZeroFlow stagnate in the valley. Even with bias correction, the Adam optimizer still fails to escape the local region without sufficient momentum. However, Z0-Adam-Sigh seems successfully optimize towards the region around the global minimum. Unlike ZO-Adam, ZO-Adam-Sign applies the gradient using sign function, which outputs either +1 or - depending on the gradient direction. This discrete update method, which lacks continuous gradient information, causes 20-Adam-Sign to take larger, step-like jumps. Particularly in the early stages, where gradient information is sparse or noisy, this leads to more fluctuations and introduces greater randomness the optimization process, helping it to cross over the valleys. The second-row subfigures use SGD as the base optimizer. We observe that, except for ZO-SGD-Sigh Sign, both ZeroFlow and FO-SGD converge effectively. This can be attributed SGD's simple update rule based function values. Notably, FO-SGD escapes the valley by leaping to higher and flatter region, while ZeroFl Serooloowwo demonstrates the ability to traverse beneath valleys. With higher learning rate of 0.01, FO-Adam, 7.0-Adam with four queries, and TO-Adam-Sigg escape the local region more easily. However, 70-Adam still stagmates along the valley, demonstrating the stabilizing effect of multiple query loops. Similarly, ZO-Adam-Cone suffers from the risk of an overly conservative strategy. ZO-SGD also fails to converge to the optimum due to gradient explosion caused by the large learning rate. contrast, ZeroFlow shows minimal degradation despite its inherent randomness.
As result, the behavior of KeroFlow-sometine escaping the valley but failing to converge the optimum, and sometimes getting trapped with low query counts but not with higher ones-highlight the trade-off between randomness and stability during updates. With larger search loops, lower learning rates, and more stable update steps, the model becomes increasingly prone to getting stuck in local minima, especially in continual learning scenarios where balancing old and new tasks introduces additional complexity.
B.3. Extra Evaluation on Memory Replay Methods
We further evaluate the performance of ZeroFlow when applied to a representative replay-based method (MEMO (Zhou et al., 2023c), replay buffer = 2000). to demonstrate its broader applicability. As shown below, ZeroFlow consistently remains stable in mitigating forgetting. Notably, although the average accuracies exhibit slight gaps compared to FO methods, the accuracies the final stage progressively approach or even surpass those of the FO baselines on the CIFAR-100 dataset.
Table 6: Accuracy Results on MEMO.
CIFAR-100
ImageNet-A
Method
Optimizer
Strategy
Avg
Last
Avg
Last
FO
87.43
79.66
53.15
38.97
70
85.92
79.00
46.87
25.81
SGD
Sign
85.72
79.10
53.31
38.18
Conserve
85.86
79.20
47.20
28.51
MЕMO
FO
86.45
76.17
54.06
41.54
Zo
85.86
78.59
52.70
39.01
Adam
Sign
86.16
76.38
53.10
39.82
Conserve
85,89
77.71
53.20
39.57
Forward
84.63
76.32
53.59
40.64
B.4. Memory and Time Efficiency on Larger Transformers
To assess the scalability of ZeroFlow, we evaluated its efficiency on two larger vision transformers, ViT-L/16 and ViT-H/14. As shown below, ZeroFlow consistently offers substantial memory savings across all model sizes. Notably, even when using SO-SGD-Sign, the runtime remains faster than that of standard FO optimization.
B.5. Longer Task Sequence
To further assess robustness, we evaluate performance on an extended task sequence consisting of 20 tasks. As shown below, ZeroFlow continue deliver comparable performance. Additionally, following (Wang al., 2024), we additionally
16

---

ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
Table 7: Evaluation on lager transformers.
ViT-B/16
ViT-L/16
ViT-H/14
Optimizer
M
Runtimell
Memory
Runtime
Memory
Runtimel
FO-SGD
12.08GB
59.3s
33.27GB
65.0s
78.09GB
190.1s
ZO-SGD (q=1)
2.41GB
32.4s
3.77GB
47.0s
6.45GB
118.7s
ZO-SGD (q=4)
2.41GB
111.7s
3.77GB
178.3s
6.45GB
442.6s
70-SGDD-Sign
2.41GB
32.4s
3.77GB
48.7s
6.45GB
119.3s
ZO-sGD-Conserve
2,41GB
70.1s
3.77GB
108.9 9s
6,45GB
222.3s
Forward
3.94GB
45.9s
5.82GB
142.0s
9.85GB
372.58
Table 8: Additional Experimental Results of EASE 20 Sequential Tasks.
Method
Optimizer
Strategy
Avg
Last
FWT
BWT
FO
87.32
80.20
-6.89
-6.79
ZO
82.65
75.98
-8.33
-7.71
SGD
Sign
83.47
76.13
-8.01
-7.22
Conserve
82.20
75.94
-8.64
-7.93
EASE
FO
86.67
78.19
-7.17
-6.80
ZO
84.07
76.89
-7.92
-7.19
Adam
Sign
84.16
76.90
-7.95
-7.20
Conserve
83.82
76.76
-8.04
-7.07
Forward
82.84
76.32
-8.25
10.84
adopt the FWT and BWT metrics to assess the overall performance of ZeroFlow. FWT (Forward Transfer) quantifies the average influence of prior knowledge on the learning new tasks, while BWT (Backward Transfer) measures the average influence of learning tasks the performance of previously learned K 1 tasks.
K
FWT 1
Σ o. -g))
BWT
aK. - a3,5)
(10)
Here, (k.j denotes the accuracy on task j after training on the k-th dataset, and āz represents the accuracy of a random initialized model trained only on dataset DD.
17


Reviews:
[{"review_id": "d0oJ0vAkY3", "reviewer": "Reviewer_see4", "summary": "The paper explores the challenge of catastrophic forgetting in continual learning under a gradient ban setting, where gradients information is unavailable. To address this, the authors investigate zero-order optimization methods, which rely only on forward passes without requiring backpropagation. Their key finding is that zero-order optimization can mitigate catastrophic forgetting while improving computational efficiency compared to first-order optimization. The paper proposes three enhancements to further improve zero-order optimization on mitigating catastrophic forgetting: hybrid first- and zero-order optimization, integrating historical gradient to stabilize optimization, random sparsity in gradient estimation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Justification of Datasets**\n\nWe follow the typical dataset setup to perform all evaluation [1,2]. In general, any overlap has been properly accounted for and domain gap is further considered (e.g. ImageNet-A and OmniBench are acknowledged to have large domain gap with ImageNet, please refer to [1,2]- Datasets Section). Thanks for your nice suggestion, we've stated it in revision for clarity.\n\n[1] Continual Learning with Pre-trained Models: A Survey, IJCAI-24\n\n[2] Class-Incremental Learning: A Survey, TPAMI-24\n\n**Q2: Concern about the Query Budget**\n\nActually, ZeroFlow has comprehensively addressed this concern. In details, (i) Consistent Query Budget: ZeroFlow consistently operates under a query budget of 1 across all evaluations, which can be seen in Tables 1/2, Figures 1/2/4 etc. (ii) Enhanced Training Efficiency: The proposed enhancement methods retain the same query budget of 1 (see Figures 7/8, Tables 3/4.) while further accelerating training (see Enhancement 3 in our response 3 to Review HREV). (i) and (ii) strongly support our claim--Overcoming Forgetting is Easier. (iii) As you mentioned, the overhead analyses partly address this concern. Moreover, we provide extra insights on query budgets dynamics in Figure 5 to ensure a full understanding. We expect such an analysis to inspire subsequent work to extend the scope of ZeroFlow. Overall, this concern has been addressed in the original manuscripts. \nYour expertise in raising this issue is much appreciated.\n\n**Q3: Performance Discrepancy**\n\nAlthough both EASE and APER are prototype-based PTM CL models, EASE incorporates adapter training for each incremental task, wherein the training process builds upon previously trained adapters. In contrast, APER adapts the PTM solely during the initial training stage and subsequently maintains frozen for all following tasks. Consequently, the performance of later tasks in EASE is highly contingent on prior task training, whereas in APER, tasks remain independent of one another. Therefore, if a performance discrepancy arises between ZO and FO optimizers, EASE tends to amplify this gap, whereas APER is only marginally affected.\n\n**Q4: Open Discussion about Black-box LLM**\n\nFirst, our ZeroFlow naturally extends to LLM-as-a-service scenarios. Our tests on the VLM in the rebuttal (the results see Q6) have demonstrated this potential. Second, the forward-pass method we evaluate could be adapted to optimize prompts or lightweight external adapters through sequential API calls. In short, our findings on memory efficiency and forgetting mitigation potentially address key constraints of LLM deployment. Future work could establish benchmarks for black-box LLM continual learning.\n\n**Q5: Extra Eval. on Memory Replay Method**\n\n We further offer the performance of ZeroFlow on a typical replay-based method (MEMO [3], replay buffer=2000) to broaden applicability. As shown below, ZeroFlow remains stable in overcoming forgetting. Below, CFR and INA denote CIFAR-100 and ImageNet-A, respectively.\n\n|Optimizer|Strategy|CFR-Avg|CFR-Last|INA-Avg|INA-Last|\n|---------|--------|------|-------|------|-------|\n|SGD|FO|87.43|79.66|53.15|38.97|\n| |ZO|**85.92**|79.00|52.87|35.81|\n| |Sign|85.72|79.10|53.31|38.18|\n| |Conserve|85.86|**79.20**|49.20|36.51|\n|Adam|FO|86.45|76.17|54.06|41.54|\n| |ZO|85.86|**78.59**|52.70|39.01|\n| |Sign|**86.16**|76.38|53.10|39.82|\n| |Conserve|85.89|77.71|53.20|39.57|\n|-|Forward|84.63|76.32|**53.59**|**40.64**|\n\n\n[3] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning, ICLR-23 (Spotlight)\n\n**Q6: Potential Tests on VLM-CL**\n\nWe evaluated ZeroFlow on continual learning of vision-language model (using MoE4Adapter [4], all training protocols follow [4]). As below, the general utility of ZeroFlow is reconfirmed. Furthermore, we are preparing to open up a leaderboard to the community that will cover more tasks, including PTM-CL, VLM-CL etc.\n\n\n| Method | Strategy | CFR |  |\n|--------------|----------|----------------|----------------|\n| |  | **Avg** | **Last** |\n| | FO | 84.32 | 76.89 |\n| | ZO | 84.27 | **76.91** |\n| MoE4Adapter | Sign | **84.38** | 76.73 |\n| | Conserve | 84.26 | 76.75 |\n| | Forward | 83.96 | 76.53 |\n\n[4] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR-24.\n\n**Q7: Memory/Time Advantages on Larger Transformers**\n\nYes, we had employed two larger transformers (ViT-L/16 and ViT-H/14) to evaluate the efficiency of ZeroFlow, as shown below. All of them offer memory advantages, and ZO with ZO-Sign still run faster than FO.\n\n|Opt|Base-Mem|Base-Speed|Large-Mem|Large-Speed|Huge-Mem|Huge-Speed|\n|---|-------|-------|--------|--------|-------|--------|\n|FO|12.08GB|59.3s|33.27GB|65.0s|78.09GB|190.1s|\n|ZO(q=1)|2.41GB|32.4s|3.77GB|47.0s|6.45GB|118.7s|\n|ZO(q=4)|2.41GB|111.7s|3.77GB|178.3s|6.45GB|442.6s|\n|Sign|2.41GB|32.4s|3.77GB|48.7s|6.45GB|119.3s|\n|Conserve|2.41GB|70.1s|3.77GB|108.9s|6.45GB|222.3s|\n|Forward|3.94GB|45.9s|5.82GB|142.0s|9.85GB|372.5s|", "claims_and_evidence": "Yes, the paper’s empirical coverage of ZO variants (sign-based, conservative, etc.) and ablations on query budgets (q=1,2,4…) reinforce the authors main claims claims.", "methods_and_evaluation_criteria": "Yes. This paper is a benchmark paper, and they authors use metrics including average accuracy, final accuracy, and forgetting.", "theoretical_claims": "The paper primarily focuses on benchmarking and algorithmic proposals rather than formal proofs. The authors reference known theoretical properties of ZO, such as the expected convergence behavior in high-dimensional spaces, but do not introduce new formal theorems.", "experimental_designs_or_analyses": "Yes. \nFor evaluation on EASE and APER models, because the models are based on ViT-B/16 models pre-trained on the ImageNet-21k dataset, it is important to provide justification that the test datasets (CIFAR-100, CUB-200, ImageNet-A, and OmniBenchmark) do not overlap with the pre-training data, or that any overlap has been properly accounted for. This ensures that the reported results are not affected by potential data leakage.", "relation_to_broader_scientific_literature": "The paper draws on two lines of work: (1) zeroth-order optimization, especially in black-box or gradient-banned contexts (e.g., black-box LLM APIs, non-differentiable modules), and (2) catastrophic forgetting in continual learning. The authors cite standard CL approaches (e.g., EWC, replay-based methods), plus references on ZO from both classical (SPSA, random gradient-free) and modern contexts (MeZO for large language models). The synergy of these areas is interesting: while gradient-based solutions dominate CL, “ZeroFlow” demonstrates that forward-only methods can provide surprisingly strong performance plus memory savings.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- Interesting demonstration that catastrophic forgetting can be addressed without gradient signals, which is valuable for real-world “gradient ban” scenarios.\n- Comprehensive coverage of ZO methods (SPSA-based, sign-based, conservative, etc.) and an ablation on query budgets.\n- Detailed analyses with standard CL metrics (accuracy, forgetting), plus resource usage.\n\nWeaknesses\n- The experimental scope, while broad, focuses mostly on standard classification tasks. Future expansions into more diverse data modalities or multi-domain tasks would strengthen generalizability claims.\n- ZO-based training can still be slow if the query budget or the model scale is large (though they partly address this with memory overhead analyses).", "other_comments_or_suggestions": "none", "questions_for_authors": "1. In Table 2, the performance difference between FO and ZO optimization methods on APER is quite marginal, whereas the difference is more pronounced for EASE. Could you provide a possible explanation for this discrepancy? \n2. Investigating black-box LLM usage would be an interesting extension, especially given the paper’s references to LLM-as-a-service scenarios. Can the authors comment on this?\n3. More examples of how ZO training interacts with memory replay or generative replay strategies might broaden applicability.\n4. Potential tests on domain-incremental or cross-modal tasks (e.g., image → text) would help confirm the approach’s general utility.\n5. Have the authors tested ZeroFlow on extremely large-scale models (like full ImageNet training or large pretrained transformers), and do the memory/time advantages still hold there?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethics concerns."}}, {"review_id": "GFnMm85FMk", "reviewer": "Reviewer_HREV", "summary": "Claimed contributions:\n - Contrib 1: benchmark (called ZeroFlow) of continual learning using two previously published strategies: EASE and APER, but only using zero order estimation of descent directions, on vision tasks\n - Contrib 2: insights into the role of forward pass in managing task conflict, and trade-offs between forgetting and memory efficiency\n - Contrib 3: 3 tricks to improve zero order continual learning:\n    - Contrib 3A: Hybrid zero order\n    - Contrib 3B: Leverage historical gradients\n    - Contrib 3C: Sparse update directions: randomly set some directions to zero", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Extensions to Contrib 1**\n\nWe extended the experimental scope to enhance Contrib 1. In detail, we evaluated ZeroFlow on extra strategies: memory replay CL and VLM-CL (see **Q5/7 of Reviewer see4**).\n\n**Q2: Explanation of Contrib 2**\n\nIn Section 3.2, ZeroFlow examines how ZO optimization helps mitigate catastrophic forgetting by comparing its optimization landscape to that of FO optimization. Figures 3 and 6 show the optimization paths of both methods as they balance preserving prior knowledge and adapting to new tasks. The axes represent the two most influential feature vectors, with the blue and red \"X\" markers indicating the optima for old and new tasks, respectively. The black dot represents the learned parameters, initially biased toward the old task, while the black star marks the optimal balance between both tasks. We will explain it in the revision.\n\nAlso, **Reviewer Js2L** also provided a clear explanation of Contrib 2 (**Strengths** and **Designs Or Analyses Section**) for your consideration.\n\n**Q3: More Discussion of Enhancements**\n\n**Enhancement 1: Hybrid Optimization** begins by leveraging gradients for fast adaptation to new tasks. Once the parameters are sufficiently close to the optimal solution, ZO is employed to fine-tune the solution, addressing forgetting.} i) FO ensures rapid convergence and efficient adaptation to new tasks, while ZO refines the solution in regions where gradients are unreliable or sparse. ii) The inherent randomness of ZO helps avoid sharp minima, akin to the principles of SAM, fostering more stable and generalizable solutions.\n\n**Enhancement 2: Historical Utilization** employs an online Exponential Moving Average (EMA) to retain the past update information of old task gradients, adjusting them to minimize deviations from the historical direction.} By weighting the historical gradients, it reduces the impact of fluctuations induced by new tasks, effectively alleviating forgetting. Moreover, it enhances the stability of ZO optimization, ensuring smoother convergence and preserving knowledge from old and new tasks.\n\n**Enhancement 3: Sparse Perturbation** introduces sparsity into the ZO by setting a fraction of perturbation dimensions to 0, thereby reducing the number of perturbed parameters.} i) This mitigates the instability inherent in ZO by lowering variance in gradient estimation, leading to more consistent updates. ii) Sparsity reduces overhead, making ZO method more practical for high-dimensional CL settings.\n\nOverall, we will include above into the revision for clarity.\n\n**Q4: Effects of the Hyperparameters from CL**\n\nWe provided the results on varying projection dimension $r$ and the trade-off parameter $\\alpha$ of EASE. As below, ZeroFlow remain solid, which means that we provided reliable conclusions. We'll provide more analysis in version to ensure reliable conclusions. \"‘Cons’ below refers to ZO-Conserve.\n\n| r            | FO_64    | FO_32    | FO_16    | ZO_64    | ZO_32    | ZO_16    | Sign_64  | Sign_32  | Sign_16  | Cons_64  | Cons_32  | Cons_16 |\n|--------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n| $\\alpha$ 0.3 | 91.11    | 91.02    | 91.37    | 78.25    | 78.14    | 78.97    | 82.77    | 83.16    | 83.84    | 82.77    | 82.15    | 82.19    |\n| $\\alpha$ 0.1 | 91.23    | 91.30    | 91.47    | 78.62    | 78.81    | 79.21    | 83.21    | 83.90    | 83.58    | 82.22    | 82.25    | 82.46    |\n| $\\alpha$ 0.05| 91.37    | 91.39    | 91.54    | 78.45    | 78.82    | 78.94    | 83.12    | 83.25    | 83.15    | 82.46    | 82.18    | 82.41    |\n\n**Q5: More Precise Statements**\n\nWe've carefully reviewed the manuscript, clarifying unclear phrases, as follows,\n\n- Gradient ban: We clarify that we've defined this concept in the original manuscript, see Lines 16, 54 and 40. And we've described its scenario to make the concept easier to understand, see Lines 23 to Lines 40.\n- Forward pass method: this concept refer to gradient-free optimization methods that rely on forward pass, specifically referring to ZO and Forward-Grad method. We'll define it.\n- Forgetting measure: The forgetting measure we used is a common metric in CL [5]. We will define it again for clarity.\n- Function value: It represents the optimization objective in Figure 7, where values approaching zero indicate proximity to the global optimum.\n- $x^{-1}$: denotes the element-wise inversion of the perturbation vector, which is necessary to ensure that the expected value of the gradient estimate aligns with the true gradient when $x$ follows a broader asymmetric distribution [6].\n- Other misc: Small errors are fixed, e.g., the subtitle of section 3.2 corrected to ZO optimization for overcoming forgetting.\n\n [5] A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI-24.\n\n [6] Multivariate Stochastic Approximation using a Simultaneous Perturbation Gradient Approximation, TAC 1992.", "claims_and_evidence": "Contrib 1 is as far as I can tell a new contribution, but it seems rather limited to only compare 2 strategies: EASE and APER. \n\nI was not able to fully appreciate contrib 2: I assume that it refers to figures 3 and 6 and section 3.2, but the figures are not adequately explained (e.g. what do the axes represent ? what is the setup studied here ?).\n\nI was not able either to appreciate the proposed methods to improve zero order continual learning (contrib 3): the techniques are barely described in section 6 with not enough details to be self contained, whereas their should be discussed in more details with ablation studies in order to benefit the community.", "methods_and_evaluation_criteria": "The benchmarked datasets seem reasonable, as they are commonly used in the evaluation of continual learning methods. A limitation is that there are only vision tasks in the benchmark.", "theoretical_claims": "Not applicable: there is no theoretical claim in the paper.", "experimental_designs_or_analyses": "From experience, the performance of continual learning techniques at mitigating forgetting is greatly dependent on specific values of hyperparameters. An extensive discussion of the strategy to choose these hyperparameters is currently missing so I don't think the current state of the benchmark provides any reliable conclusion", "relation_to_broader_scientific_literature": "Previous literature in continual learning and in zero order optimization is adequately mentioned as far as I can tell.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "Many statements are rather imprecise, or even do not make sense, and would benefit from proofreading (examples in the next field below). Some definitions are missing and new terms seem to be introduced without much discussion.", "other_comments_or_suggestions": "- \"gradient ban\" is, as far as I can tell, a phrase coined in the paper.\n- please define a \"forward pass method\"\n- please define \"forgetting measure\" in the benchmark\n- figure 8: what is the \"function value\" ?\n- \"genetic\" (section 3) should it be \"generic\" ?\n- discussion regarding the ImageNet input dimension in section 3 which is irrelevant for defining the number of parameters in conv nets (convolution kernels do not depend on the input size)\n- in eq. 1, what does it mean to compute the inverse of the vector xi ?\n- BP-free and BP-based in section 4.2. This probably refers to backpropagation, but it is never defined, neither used elsewhere in the paper.\n\nImprecise statements:\n - section 3.2 \"ZO optimization for catastrophic forgetting\" sounds like forgetting is a desirable property that the method tries to amplify", "questions_for_authors": "I would suggest some revisions to make the paper more self-contained and easy to read: limit the number of repeated statements and instead carefully define the material needed to understand the benchmarked methods as well as the proposed improvements.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WT2J0uslyn", "reviewer": "Reviewer_Js2L", "summary": "This submission presents a novel benchmark, ZeroFlow, for evaluating overcoming catastrophic forgetting under a gradient ban. The key insight is that forward pass optimization alone can also mitigate forgetting, which challenges the conventional reliance on backpropagation-based optimization. The study evaluates seven forward-pass optimization algorithms across various forgetting scenarios, datasets, and evaluation metrics. The key findings are, (i) Insights into the trade-offs between forgetting and learning; (ii) Effectiveness of forward pass methods for forgetting; (iii) Efficiency and memory management, more practical for CL scenarios; Next, the submission further proposes three new enhancement techniques: hybrid optimization, historical information, and sparsity-induced estimation method, to improve the efficiency and stability of forward-pass.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More Discussion of Enhancements**\n\nIndeed, as you note, Enhancement 3 has a potential acceleration advantage, which benefits from the reduction in average queries. More analysis see Enhancement 3 in our response 3 to Review HREV. Thank you for your suggestion, which effectively improves our proposed enhancement technique. \n\n**Q2: Visualization of Trajectory**\n\nWe fully acknowledge that the small legend hindered the clarity of key insights. We will enhance figure readability by (i) enlarging all legends (old and new task), and (ii) zooming in on optimization endpoints (black, red and blue mark) and joint optimization centers (star mark). These revisions will ensure the visualized insights about task-specific optimization behaviors are more intuitively conveyed.\n\n**Q3: Concise Overview of ZO Series**\n\nZeroth-order optimization aims to minimize/maximize an objective function $f: \\mathbb{R}^n \\to \\mathbb{R}$ without derivative information. The core problem is formulated as $ \\min_{\\theta \\in \\mathbb{R}^n} f(\\theta) $, where $\\theta$ denotes the optimization variable. To enable gradient-based updates, Simultaneous Perturbation Stochastic Approximation (SPSA) is a commonly used technique to approximate gradients by perturbing the input variables. Specifically, the gradient $ \\nabla f(\\theta) $ at point $ \\theta $ is estimated as:\n\n$\\nabla L(\\theta, \\xi; B) = \\frac{f(\\theta + \\epsilon \\xi; B) - f(\\theta - \\epsilon \\xi; B)}{2 \\epsilon} \\cdot \\xi^{-1},$\n\nwhere $ \\xi \\sim \\mathcal{N}(\\mathcal{0}, \\mathcal{I}) $ is a random perturbation vector, and $ \\epsilon > 0 $ is a small perturbation step size (typically adjusted during optimization).\n\n**ZO-SGD:** Using the gradient estimator $\\nabla L(\\theta, \\xi; B)$, zeroth-order algorithms, such as ZO-SGD, follow the iterative update rule:\n\n$\\theta_{t+1} = \\theta_t - \\alpha_t \\nabla L(\\theta_t, \\xi_t; B),$\n\nwhere $\\alpha_t$ is the learning rate at step $ t $. ZO-SGD bypasses explicit gradient computation through local function evaluations, making it suitable for high-dimensional, non-convex optimization problems.\n\n**ZO-SGD-Sign:** A variant of ZO-SGD, known as ZO-SGD-Sign, improves upon the original approach by approximating the gradient direction using the sign of the gradient estimate. The update rule becomes:\n\n$\\theta_{t+1} = \\theta_t - \\alpha_t \\, \\text{sign}(\\nabla L(\\theta_t, \\xi_t; B)),$\n\nwhere $ \\text{sign}(\\cdot) $ denotes the element-wise sign function. This approach often leads to faster convergence in some problems where the magnitude of the gradient is not as important as its direction.\n\n**ZO-SGD-Conserve:** Another variant is ZO-SGD-Conserve, which conserves the gradient information over multiple iterations. The update rule for this method is:\n\n$\\theta_{t+1} = \\theta_t - \\alpha_t \\cdot \\frac{1}{k} \\sum_{i=0}^{k-1} \\nabla L(\\theta_{t-i}, \\xi_{t-i}; B),$\n\nwhere $ k $ is the number of past iterations used to average the gradient estimates. This method is beneficial when the gradient updates are noisy, and averaging helps stabilize the optimization process.\n\n**Forward Gradient:** The forward gradient is an approximation technique where the gradient is computed by evaluating the function at a perturbed point and using a linear approximation. Specifically, the gradient $\\nabla f(\\theta)$ at point $\\theta$ is estimated as:\n\n$\\nabla f(\\theta) \\approx \\frac{f(\\theta + \\epsilon \\xi) - f(\\theta)}{\\epsilon} \\cdot \\xi^{-1},$\n\nwhere $\\xi \\sim \\mathcal{N}(\\mathcal{0}, \\mathcal{I})$ is a random perturbation vector, and $\\epsilon > 0$ is a small perturbation step size. The forward gradient is generally used when the optimization problem requires accurate directional updates without relying on an explicit derivative.\n\n**Q4: Extra Memory of Forward-Grad**\n\nForward Gradient requires more memory than ZO-SGD because it involves computing gradients through the Jacobian-vector product, which necessitates storing all intermediate activations during the forward pass. In models like ViT, this includes storing large attention matrices and other intermediate results. In contrast, ZO-SGD only requires two forward passes with perturbed inputs, without needing to store these intermediate activations, resulting in lower memory usage.", "claims_and_evidence": "The claims made in the submission are indeed well-supported by clear and convincing evidence, as follows,    \n* The visualization of optimization trajectories (Figure 3/6) supports the claim that forward pass methods can effectively balance learning and forgetting.\n* The comparable or better performance supports the claim that effectively overcomes forgetting (Table 1/2).\n* The proposed enhancements are well-supported by experimental results that demonstrate improved performance in overcoming forgetting (Tables 3/4, and Figure 7).", "methods_and_evaluation_criteria": "The manuscript employs reasonable methods and evaluation criteria, including common datasets in forgetting scenarios (CIFAR-100, CUB, ImageNet-A, and OmniBenchmark), well-recognized metrics (accuracy, last-task accuracy, and forgetting measures, and memory, query, and runtime efficiency).", "theoretical_claims": "The manuscript includes reasonable theoretical claims, such as the process of gradient estimation (Equation 1, Algorithm 1) and the definition of zeroth-order optimization in CL scenarios (Section 3.2). Moreover, the authors provide empirical evidence to support their theoretical claims (Figure 3).", "experimental_designs_or_analyses": "The soundness and validity of the experimental designs and analyses in this submission are well-constructed, such as the comprehensive benchmark experiment (Table 1, Figure 2/4), efficiency analysis (Table 2), and reasonable enhancement designs (Figure 7, Table 3/4). It is commendable that the manuscript provides ample illustrative observations, such as Figures 2/5/6/8/10, which add to its value.", "relation_to_broader_scientific_literature": "The submission provides a thorough discussion of the relevant literature. Prior to this study, substantial efforts relied on gradient information to overcome forgetting, they tries a new technical pathway to do this. And their key contribution is overcoming forgetting using the forward pass method, which is related to catastrophic forgetting, and optimization in CL. The authors discuss literature related to these concepts, including CL, FO method, ZO method, and others.", "essential_references_not_discussed": "The literature discussed by the authors appears to be sufficiently comprehensive and closely related to the topic. The cited works cover the necessary background and prior findings, providing a thorough context for their contributions.", "other_strengths_and_weaknesses": "**Strengths,**   \n* The manuscript proposes an interesting and practical scenario, that is overcoming catastrophic forgetting under a gradient ban. The concept of the gradient ban is an intriguing idea. In short, this manuscript introduces a new topic to the community that could potentially inspire some upcoming work.   \n* The authors propose a benchmark to explore the optimization principles of overcoming forgetting under gradient bans. They evaluated a series of methods that enable learning new tasks and retaining old ones using only forward passes. More valuable, the authors provide insightful observations and build their motivation on these findings.   \n* The authors provide insights into the evolution of gradient trajectories for new and old tasks during the optimization, such as managing task conflicts. These observations are inspiring for understanding how just forward pass balance learning new tasks and retaining old knowledge, and would likely inspire further research in CL.     \n* The manuscript is logically structured and engagingly written, with clear presentations. Notably, the authors acknowledge the potential risks of this new topic and discuss them thoroughly. Moreover, they also fully explore the advantages of this benchmark, such as reduced memory usage, faster training, and less forgetting.   \n* Building on their empirical observations, the authors propose three enhancement techniques that achieve promising performance.   \n\n**Weaknesses,**   \n* Compared to the significant contribution of introducing a new technological pathway for the community, the contributions of the three enhancements are somewhat weak but still represent valuable attempts. As I understand it, the mechanisms of Enhancements 1, 2, and 3 seem to also offer shorter training times or less memory usage. I speculate that this efficiency might stem from a reduction in the average queries. Unfortunately, the authors did not conduct such an analysis. I believe completing this would be highly valuable, as this would well-align their observations and the advantages of these optimization principles.   \n* The different behavior of the optimization principles on new and old tasks are insightful for researchers looking to follow this topic. However, the legends in the figures (Figures 3, 7, and 8) are too small, including the optimization endpoints and the joint optimization center for new and old tasks, this makes it difficult to fully appreciate the insights conveyed by these visualizations.   \n* The authors clearly explain the motivation for selecting these forward-pass methods. Given my experience, I can easily follow them. However, for those unfamiliar with these optimization algorithms, there may be a learning curve. Providing brief explanations would be beneficial for improving the accessibility of the manuscript.", "other_comments_or_suggestions": "The key claim of the manuscript is that forward passes alone are sufficient to mitigate forgetting, which challenges the traditional reliance on BP-based optimization idea. Based on my experience, this work offers a new technological pathway for the CL community. And, the insights provided are quite valuable for understanding this topic.    Overall, I tend to be positive about this work. Given some concerns and weaknesses (Please see Weaknesses and Questions), I'm willing to discuss them in the rebuttal.", "questions_for_authors": "Here are a few questions that interest me,   \n* Why does Forward-Grad consume slightly more memory than other forward-pass methods?   \n* Could the authors provide the brief explanations of the optimization principles behind the used forward pass methods in benchmark? (Please see Weakness 3)    \n* Could the mechanisms of Enhancements 1, 2, and 3 provide shorter total training times or reduced total memory usage? Or do only specific enhancements offer benefits? (Please see Weakness 1)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EWwBc5c3l", "reviewer": "Reviewer_WA47", "summary": "This paper investigates continual learning for deep neural network when the gradient is not accessible -- instead of using backpropagation methods (first order methods (FO)), the gradient is approximated by forward pass methods (zeroth-order optimization (ZO)). The paper present ZeroFlow benchmark, where different zeroth-order approaches are proposed (combination of different apprximation method, direction computation and parameter update). The benchmark use ViT-B/16 pre-trained model on IN21K and perform 10 task class incremental learning on multiple datestes (Cifar-100, CUB, ImageNet-A, OmniBenchmark). Based on experimental results, this work proposes 3 different enhancement that can be used to further improve ZO approaches for continual learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Caption of Figure 4** \n\nThanks for pointing out! We've corrected it.\n\n**Q2: Missing Quotes and Descriptions** \n\nWe've included quotes (EASE, CVPR-24 and APER, IJCV-24) and use one sentence to describe them.\n\n**Q3: Longer task sequences** \n\nAs shown below, we evaluated the results on a task sequence of 20. ZO method still provides comparable performance across learning metrics. Besides, we will add evaluations on three other datasets for further comparison in the appendix.\n\n|  |  |  |  CIFAR-100 (20)  |  | |\n|--------|--------|---|----|---|----|\n|Optimizer|Strategy|Avg|Last|FWT|BWT|\n|**SGD**|FO|87.32|80.20|-6.89|-6.79|\n| |ZO|82.65|75.98|-8.33|-7.71|\n| |Sign|83.47|76.13|-8.01|-7.22|\n| |Conserve|82.20|75.94|-8.64|-7.93|\n|**Adam**|FO|86.67|78.19|-7.17|-6.80|\n| |ZO|84.07|76.89|-7.92|-7.19|\n| |Sign|84.16|76.90|-7.95|-7.20|\n| |Conserve|83.82|76.76|-8.04|-7.07|\n|-|Forward|82.84|76.32|-8.25|-7.84|\n\n**Q4: Knowledge Transfer**\n\nFollowing [7], we provide the results of BWT and FWT of EASE with 20-split CIFAR-100 datasets above. We'll add these results and analysis into the Appendix of the main paper as an interesting plot. Thanks for your great suggestions to further enhance the quality of our manuscripts!\n\n[7]  A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI 2024.\n\n**Q5: Upper-bound for ZeroFlow**\n\nThanks for your nice advice. We agree that the upper bound is significantly valuable for ZeroFlow. We added a linear layer for classification after the pretrained ViT-B/16-IN21K backbone and then fine-tune it with several  ZO methods as the upper bound, as shown below. The updated results will be included in Table 1 of the revised manuscript. Moreover, we're opening up a leaderboard to the community, and will offer more upper bound results on various datasets and CL method. Thank you again!\n\n| Joint-training | FO   | ZO   | Sign | Conserve | Forward  |\n|----------------|------|------|------|----------|----------|\n| SGD            | 92.01| 90.66| 91.71| 91.89    | 91.65    |\n| Adam           | 91.97| 91.46| 91.63| 91.50    | 91.65   |\n\n**Q6: Enhancement together**\n\nThe results shown below indicate that the proposed enhancements do not conflict with each other and can be integrated. Specifically, FO training benefits from fine-tuning using hybrid ZO optimization, as illustrated in Figure 7. Additionally, it can be further enhanced by utilizing historical gradients and sparsity perturbation to reverse forgetting and achieve stable optimization, as analyzed in R2Q3. Thanks for your great suggestions.\n\n|                | Hybrid | Historical | Sparsity | Avg   | Last   |\n|----------------|--------|------------|----------|-------|--------|\n| FO-SGD         | -      | -          | -        | 61.24 | 51.02  |\n| ZO-SGD         | -      | -          | -        | 57.87 | 48.32  |\n|                | ✔      |            |          | 61.40 | 51.34  |\n|                |        | ✔          |          | 58.90 | 49.04  |\n|                |        |            | ✔        | 59.47 | 49.24  |\n|                | ✔      | ✔          | ✔        | 62.07 | 51.94  |\n\n\n**Q7: Small Typoes**\n\nWe've corrected the typo in Conclusion and added explanations for the bold terms in Table 1.", "claims_and_evidence": "The claims are clearly supported by the appropriate experimental results. Mostly are easy to follow by the reader, however, some description can be improved for a better readability, e.g. Figure 4 - presenting last-task accuracy, however the bolded description \"Results for Forgetting\".", "methods_and_evaluation_criteria": "The proposed methods seem like a comprehensive combination of multiple ZO forward-pass methods. But not an expert here if the selection is good. \n\nFor the dataset and evaluation metrics -- this work uses standard CL performance evaluation metrics. However, here, maybe interesting would be to see forward/backward transfer and values for the particular tasks (in appendix). That would be interesting and make the benchmark more informative.", "theoretical_claims": "Most of the claims are supported by empirical analysis. No theoretical claims with proofs.", "experimental_designs_or_analyses": "Experimental designs follow the standard class-incremental strategy of spiting datasets into multiple tasks. In this case, it's always ten.", "relation_to_broader_scientific_literature": "The relation to forward-pass only optimization and appropriate methods are well discussed and introduced in Sec.3. \n\nCL literature is well-covered and referenced.", "essential_references_not_discussed": "In Sec. 4.1. when EASE and APER appears, there are no references. One sentence explanation for each method would be also helpful for the reader (or a longer one in different place -- Sec 3?). Currently, the reader just receives the acronyms.", "other_strengths_and_weaknesses": "Strengths\n1. Quite comprehensive benchmark with multiple methods, datasets, and ViT-based backbone.\n2. Interesting insights and proposal of further enhancements.  \n\nWeaknesses: \n1. Not presenting longer sequences than 10 tasks.", "other_comments_or_suggestions": "1. Presenting joint training for all the methods - as an upper-bound. \n2. Would be interesting to see if there's a different knowledge transfer between the tasks in ZO vs FO methods. Maybe that can be added to the appendix and one the most interesting plot to the main paper? \n\n\nSmall ones: \n1. A mistake in Conclusion in the last sentence, 3 enhancements were proposed, not 2. \n2. Formatting of Table 1: what bolding means? Some sections are missing bolding at all - it is confusing for the reader in the current form.", "questions_for_authors": "Enhancements are presented alone, separately posing improvements. It's not clear if they can be combined together and what will be the results. Maybe the authors already tried this and can share the results in the appendix?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews, three accepts (including strong)and one reject. The paper studies continual earning under a gradient ban (where there is no access to gradient information of the model). The paper evaluates a variety of forward methods. The benchmark shows that zero/order optimization can perform continual learning at lower memory requirements. The paper also proposes several techniques to furter improve results based on several insights on ZO methods. Three reviewers appreciate these results, and even consider that this can open new directions for CL research. The negative reviewer found explanations unclear and statements impresice but has low confidence.

The AC has also read the paper, and agrees that explanations can be improved on many occasions. However, the paper presents an insightful evaluations of ZO methods for CL and contains several interesting results. Therefore the AC recommends acceptance of the paper. The authors should incorporate the rebuttal into the final amera ready version.
Decision: Accept (poster)