Adaptive Update Direction Rectification for Unsupervised Continual Learning
Authors: Yizhao Gao
Nanyi Fei
Zhiwu Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual learning
unsupervised learning
representation learning

Abstract: Recent works on continual learning have shown that unsupervised continual learning (UCL) methods rival or even beat supervised continual learning methods. However, most UCL methods typically adopt fixed learning strategies with pre-defined objectives and ignore the influence of the constant shift of data distributions on the newer training process. This non-adaptive paradigm tends to achieve sub-optimal performance, since the optimal update direction (to ensure the trade-off between old and new tasks) keeps changing during training over sequential tasks. In this work, we thus propose a novel UCL framework termed AUDR to adaptively rectify the update direction by a policy network (i.e., the Actor) at each training step based on the reward predicted by a value network (i.e., the Critic). Concretely, different from existing Actor-Critic based reinforcement learning works, there are three vital designs that make our AUDR applicable to the UCL setting: (1) A reward function to measure the score/value of the currently selected action, which provides the ground-truth reward to guide the Critic's predictions; (2) An action space for the Actor to select actions (i.e., update directions) according to the reward predicted by the Critic; (3) A multinomial sampling strategy with a lower-bound on the sampling probability of each action, which is designed to improve the variance of the Actor's selected actions for more diversified exploration. Extensive experiments show that our AUDR achieves state-of-the-art results under both the in-dataset and cross-dataset UCL settings. Importantly, our AUDR also shows superior performance when combined with other UCL methods, which suggests that our AUDR is highly extensible and versatile.

Under review as a conference paper at ICLR 2023
ADAPTIVE UPDATE DIRECTION RECTIFICATION FOR UNSUPERVISED CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works on continual learning have shown that unsupervised continual learn- ing (UCL) methods rival or even beat supervised continual learning methods. However, most UCL methods typically adopt fixed learning strategies with pre- defined objectives and ignore the influence of the constant shift of data distribu- tions on the newer training process. This non-adaptive paradigm tends to achieve sub-optimal performance, since the optimal update direction (to ensure the trade- off between old and new tasks) keeps changing during training over sequential tasks. In this work, we thus propose a novel UCL framework termed AUDR adaptively rectify the update direction by a policy network (i.e., the Actor) at each training step based on the reward predicted by a value network (i.e., the Critic). Concretely, different from existing Actor-Critic based reinforcement learning works, there are three vital designs that make our AUDR applicable to the UCL setting: (1) A reward function to measure the score/value of the currently selected action, which provides the ground-trui reward to guide the Critic's 's pre- dictions; (2) An action space for the Actor to select actions (i.e., update directions) according to the reward predicted by the Critic; (3) A multinomial sampling strat- egy with a lower-bound on the sampling probability of each action, which is de- signed to improve the variance of the Actor's ss selected actions for more diversified exploration. Extensive experiments show that our AUDR achieves state-of-the-art results under both the in-dataset and cross-dataset UCL settings. Importantly, our AUDR also shows superior performance when combined with other UCL meth- ods, which suggests that our AUDR is highly extensible and versatile.
INTRODUCTION
Continual learning has recently drawn great attention, for it can be applied to learning on sequence of tasks without full access to the historical data (Rusu et al., 2016; Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Fernando et al., 2017; Kirkpatrick al., 2017; Zenke et al., 201 Most of existing methods focus on supervised continual learning (SCL), and only a few (Rao et al., 2019; Madaan al., 2021; Fini al., 2022) pay attention unsupervised continual learning (UCL). UCL is an important yet more challenging task which requires a model to avoid forgetting previous knowledge after being trained on sequence of tasks without labeled data.
Recent UCL methods (Rao et al., 2019; Madaan et al., 2021; Fini et al., 2022) have achieved promis- ing results, and even outperform the SCL methods. However, these UCL methods are still limited by the fixed learning strategies with pre-defined objectives. For instance, LUMP (Madaan et al., 2021) proposed a fixed lifelong mixup strategy that integrates current and memory data in a random ratio sampled from a Beta distribution regardless of the shift in data distributions. This non-adaptive paradigm not ideal for UCL., since the optimal update direction of achieving the best performance on all learned tasks is continuously changing during training. Therefore, new adaptive paradigm for UCL to model the process of selecting the optimal update direction is needed.
In this work, we thus devise new UCL framework termed AUDR that can adaptively rectify the update direction (see Figure 1), where a policy network (i.e., the Actor) is proposed to select the best action for current data batch and value network (i.e., the Critic) is designed for predicting the
action's 's latent value. The Actor is trained to maximize the Critic's 's prediction reward, and the Critic is trained to more precisely predict the reward for the Actor's 's selected action. Different from the

---

Under review as a conference paper at ICLR 2023
Reward Function
2 A cdicc space
Actor-Crit
o Sampling strategy
Fastumes
Rostiiication
Curent
Mixed Features
Commen
Mixed Features
Fiod mixup stmlogy
Adautive mixup stmlogy
UCL Loss
UCL Loss
Memory
Memory
(a) Traditional UCL method (LUMP)
(b) AUDR (Ours)
Figure I: Illustration of the traditional UCL method LUMP and our AUDR. Their main difference lies in that our AUDR adopts an ector-Critic architecture to rectify the update direction (i.e., adap- tive mixup strategy) with three core designs while LUMP has a fixed mixup strategy.
short-sighted approaches (e.g., directly using a learnable parameter) that can only adjust the update direction based on current batch of data/loss, our AUDR predicts the total future rewards which is more and more precise/reliable during training. This is inspired by the Actor-Criti learning, a combination of policy-based and value-based methods, in the reinforcement learning field (Sutton et al., 1999; Haarnoja et al., 2018; Yarats et al., 2020; Mu et al., 2022). Actor-Critic learning enables the policy network to be updated at each training step (instead of after completing each task) with sampled transitions (i.e., from one state to next state), and thus it is possible to be transferred to Cctor-Critic methods under the UCL setting, because: (1) there is no environment (or reward function) that could give feedback rewards to all input states; (2) the action space for the Actor is not explicitly defined; (3) how to alleviate the problem of the trade-off between the old and new tasks remains unclear.
UCL. However, we still have difficulty in directly deploying existing
To address these problems, our AUDR thus has three core designs: (1) A reward function defining the ground-truth reward to guide the Critic's training. It is based two UCL losses of the next model (after one-step gradient descent) by respectively using the current and memory data. Thus, this reward can represent the changes of model performance on old and new tasks with the selected action, which is then utilized to conduct a continual TD-error loss to train the Critic. (2) An action space containing different actions (i.e., different update directions) for the Actor select. To be more specific, an action with larger value (e.g., 0.99) represents that the memory data accounts for higher percentage when mixed with current data, and thus the update direction is more oriented towards improving the model performance on old tasks. (3) A multinomial sampling strategy to sample the action based on the action probability distributions predicted by the Actor Concretely, for each input feature, the Actor outputs a probability distribution where each action is associated with a probability holding a lower-bound above zero to improve the variance of the Actor's selected actions. We then multinomially sample one action per feature and all samples vote for the final action. This strategy is designed to explore more diverse actions to avoid the model falling into a local optimal update direction. Note that the Actor-Critic module of our AUDR is only employed for training and we only use the backbone network for testing as in LUMP (Madaan et al., 202 l).
Furthermore, we combine the proposed adaptive paradigm with another representative method DER (Buzzega et al., 2020) for UCL to verify the extensibility of our AUDR. Specifically, we use different coefficients of the penalty loss as the new action space replace the action space men- tioned above, which is key factor that affects the update direction in DER. Other settings remain the same as in our original AUDR. We find that our AUDR+DER outperforms DER for UCL by a large margin. This demonstrates that our AUDR is highly generalizablleveesattie. We believe that our work could bring some inspirations to the continual learning community.
Our main contributions are four-fold: (1) We are the first to deploy an adaptive learning paradigm for UCL, i.e., we propose a novel UCL framework AUDR with an Actor-Critic module. (2) We devise three core designs in our AUDR to ensure that the Actor-Critic architecture is seamlessly transferred to UCL, including a reward function, an action space, and a multinomial sampling strategy. (3) Extensive experiments three benchmarks demonstrate that our AUDR achieves new state-of-the- art results on UCL. (4) Further analysis on combining our proposed adaptive paradigm with another UCL method shows that our AUDR is highly generalizable and has great potential in UCL.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Continual Learning. Existing continual learning methods can be mainly categorized into three groups: (1) Expansion-basedd methods have dynamic architectures which add extra extended net- works for in-coming new tasks (Rusu et al., 2016; Fernando et al., 2017; Alet et al., 2018; Chang methods deploy either regularization con- straints (Li & Hoiem, 2017: Aljundi et al., 2017: Hou et al., 2018; Rannen et al., 2017; Hou et al., 2019) or penalty losses (Kirkpatrick et al., 2017: Zenke et al., 2017: Farajtabar
al., 2018; Li al., 2019). (2) Regularization-busee
al., 2020; Saha
et al., 2021) to align the old and new models. (3) Rehearsal-busee methods adopt a memory buffer to restore the memory data of previous tasks (Rebuffi et al., 2017; Lopez-Paz & Ranzato, 2017; Riemer et al., 2018; Chaudhry et al., 2019; Buzzega et al., 2020; Aljundi et al., 2019; Chaudhry al., 2020; Cha et al., 2021). In contrast to these supervised continual learning works, (Rao et al., 2019; Madaan et al., 2021; Fini et al., 2022: Davari et al., 2022) have started to study the unsupervised continual learning (UCL) setting in a fixed learning paradigm by combining continual learning methods with unsupervised methods (Chen et al., 2020: Chen & He, 2021; Zbontar et al., 2021). Our AUDR is basically a rehearsal-
method for UCL but with an adaptive training paradigm which rectifies the update direction at each training step. Note that the adaptive paradigm of our AUDR is different from those methods that utilize a learnable parameter to adjust the update direction or even learn to prompt (Wang et al., 2022). since our AUDR is long-sighted (predicting future rewards) and the adaptively rectified action is only used for training (the Actor- module is dropped at the test phase). We provide more discussions about their differences in Appendix B.
Actor-Critic Learning. Actor-Critic is widely-used architecture in recent reinforcement learn- ing (RL) works. (Peters & Schaal, 2008) builds the Actor-Critie algorithm on standard policy gradient formulation to update the Actor; (Schulman et al., 2017; Mnih et al., 2016; Gruslys et al., 2017; Haarnoja et al., 2018) choose to maximize or regularize the entropy of the policy; CURL (Laskin et al., 2020) combines the unsupervised learning with Actor-Critic reinforcement learning; DrQ (Yarats et al., 2020) designs a data-regularized Q to improve the Actor-Critin method; CtrlFormer (Mu et al., 2022) proposes control transformer to tackle the forgetting problem in visual control. Our AUDR is the first work to apply Actor-Critie learning to the UCL setting. It consists of a similar Actor-Critic module as in CURL, DrQ, and CtrlFormer, but also has vital differences: a reward function for UCL to replace the environment of RL, new action space designed for the Actor, and multinomial sampling strategy ensure the diversity of actions.
METHODOLOGY
3.1 UNSUPERVISED CONTINUAL LEARNING
Unsupervised continual learning (UCL) requires the model to be trained on a sequence of tasks without labeled data. We follow the learning protocol proposed in LUMP (Madaan al., 2021) to conduct our study on UCL. Concretely, let D = [Di, D2,- ...... Dx) denotes dataset with N tasks. For each task f, it has D.- = (ar,, y/.,, 19. ) with n, samples, where zt,i is the input image and yt, is the ground-truth label (which is utilized only during the validation and test phases). For simplicity, we omit & and only use F; Ys in the following subsections. For each input sample Ty, we first randomly augment it into two views x/. .². We then consider se with parameters 0 as the backbone (encoder) and employ it to obtain d-dimensional feature representations (fo(z]), fo(x;) E Rdd Formally, based on widely-used contrastive learning framework SimSiam (Chen & He, 2021), the main training loss for UCL can be defined as:
LSimSram (fo fo()))) = D(MLP((f( fo(z)) (
1 ((MLP()(())))666)))))) (DM 2
+
(1)
where D(p, 2) = cos(p,, stop_ (ooo__grrdee(())))) = = Iplla lella and MLP(-) denotes a multi-layer
p
perception. After training, the model is then evaluated by a K-ncarest neighbor (KNN) classifier (Wu et al., 2018) following the setup in (Chen et al., 2020; Chen & He, 2021; Madaan et al., 2021).
Directly applying the SimSiam- learning loss mentioned above is difficult to obtain a well- performed model due to the catastrophic forgetting problem that the model performance on previous tasks drops significantly during sequential training. To tackle this forgetting problem, rehearsal- based methods with memory buffer to store limited previous data most popular solutions in the
3

---

Under review as a conference paper at ICLR 2023
Opest Xmer +
Obest) Yeur brue
Encoder
The Best Action
fa.
Critie
Action Score
(bactor J
Current Data
Momenitm
Actor
Update
Memory Data
Encoder
Action Score
Target Critic
2s+1
f8,
Continual TD-error
Rge leur +
Jymam
current data
memory catnn
02.0 predicted rewards
Figure 2: Schematic illustration of our AUDR. At each training step s, the Actor selects the best action Abest and the Critic predicts the reward of taking this action. We train the Actor with the predicted reward Q,- A reward function Ryt with current loss Lear and memory loss Lmmm of the next-step encoder fo.-1 is proposed to train the Critic (along with the Target Critic).
past few years. For instance, LUMP (Madaan et al., 2021) proposes a life-long mixup strategy to integrate the memory data with current data as follows:
t A mm, + A)
2 X
x.. + +(1 (1- -
B
Imizup
Lsan Stam
(3)
B 1=1
where B is the mini-batch size, x m5 .m.. denote two augmentations of the memory data zm, and X Beta(a, ar), (0, o0). This strategy is fixed across all tasks and thus it easily leads to sub-optimal performance. In addition. randomly changing the update direction at each training step not intuitive. In our opinion, whether to use more memory data (i.e., larger A) or more current data (i.e., smaller A) should be measured by appropriate pattern. Therefore, in the following, we propose adaptively rectify the update direction by an Actor-Critic architecture.
3.2 ACTION PREDICTION WITH THE ACTOR
We present the schematic illustration of our AUDR in Figure 2. In contrast to randomly sampling the mixup ratio from fixed Beta distribution, we utilize a policy network (i.e., the Actor) to predict the score distribution of all actions. Concretely, we first define an action space A {a,lī 1,2, Nact) where each action a, denotes different values of and Naet denotes the number of actions. Further, we adopt an Actor F&. Rd + IR Nact to predict and select a best action abest based on the representations of the current data z4;
best
Garg max (sx.ffofttmaatn
(4)
where (r:(), denotes the j-th element of n (x;). For mini-batch with B data samples, we first predict the best action of each sample by Eq. (4) and then vote for the action that appears the most frequently as Abest- We take the selected best action Abest as the learning strategy of the current training step s to compute the UCL loss:
Abest Fmmi + (1 Obest) D
T Gbest 2m (1 abest) 22
(5)
B
Luct
B Lsam (fe. (5) (x,)),
(6)
where 0, denotes the parameters of the encoder f at the current training step Note that the Abest detached (i.e., without gradient) in Luct. Therefore, this UCL loss is only used for training the encoder fe. In this paper, follow recent Actor-Critie learning works (Laskin al., 2020; Yarats et al., 2020; Mu et al., 2022) to train the Actor for better predicting the actions, whose learning objective is to maximize the reward predicted by value network (i.e., the Critic) Q+ WRd+NNct
4

---

Under review as a conference paper at ICLR 2023
Formally, for each training step s, we define the actor loss as:
Luctor 1 Σ B O..
1 E (c)oncattfo..a (ofe
where concat(-,-) is to concatenate the input vectors as shown in Figure 2 and Qs denotes the predicted reward at the current training step s. Training the Actor with Lactor requires the Critic's prediction be precise and reliable. However, there is existing environment which provides the ground truut rewards of input representations. Therefore, we are supposed to design a dedicated environment (or reward function) for our AUDR.
3.3 REWARD FUNCTION THE CRITIC
The widely-used concept "environment" in recent Actor-Critic learning works (Laskin et al., 2020; Yarats et al., 2020; Mu et al., 2022) refers to pre-defined interactive system, which can provide the ground-truth reward to any input state for the agent. Without the environment, the basis for measuring the values of actions is missing. Therefore, in our AUDR, we desvise reward function Ryt compensate for the lack of the environment in UCL. Rgt measures the model performance on both current data (i.c., Leur) and memory data (i.c., Limem after taking the action Obesti
Lear SimSiam (Jo, i fo 1 (x))),
(8)
Lmmm LSiSSamm fr (=mm fo. f0 (1 ())
(9)
Re (Tear + Temem);
(10)
where fo.11 denotes the updated model. Note that the training data still from the same batch. In other words, Rge evaluates how well the model can perform on the same data after updating. Further, we maintain a target Critic Q.p. to help train the Critic 2.. It is first proposed by TD3 (Fujimoto et al., 2018 8) to stablize the training process and has been adopted in many recent works (Laskin et al., 2020; Yarats et al., 2020; Mu et al., 2022). Concretely, Q.p, is initialized by Q.p and then updated by momentum update strategy (i.e., slowly updated by small part of the parameters of Q.p,, which is also known as exponential moving average (EMA) (Holt, 2004):
Pr = m & + (1-m)- .P:.
(11)
where is the momentum coefficient. Similar to the Critic, the target Critic can predict the reward of input features and actions. Differently, the Critic predicts the reward Qa,i for encoder fa, while the target Critic predicts the reward Qa+1,a for encoder fe..:
Q++1,5 = 2p, (concat( (
(R+), To (Jo (I:)))).
(12)
As directly predicting the whole future reward difficult, we could measure the difference between two training steps by temporal difference learning (i.e., TD-error) (Watkins, 1989), which is an update rule based on Bellman equation (Bellman, 1966). Since we have re-designed the ground- truth reward for UCL, we thus call the learning objective of the Critic as "continual TD-error". Formally, we define the continual TD-error as follows:
Leritic = (Qai (1S9) + 27-0))14))
(13)
1=1
where y denotes the discounted factor. During training, the predicted reward will be closer the ground-truth reward and thus the selected action becomes more reliable.
3.4 RESTRICTIONS ON SAMPLING PROBABILITY
Although we can predict the best action each training step by our AUDR framework introduced above, it still can not guarantee that the predictions are completely accurate, nor that taking other actions will definitely lead to worse performance. fact, the predicted optimal model updating direction is possibly not global optimal. Therefore, completely relying on the predictions and ignor- ing other choices will hinder the model's exploration of a better update direction. To address this problem, propose sample each action according to the predicted action score (each score has
5

---

Under review as a conference paper at ICLR 2023
range through clamping) by multinomial sampling. Formally, for each sample 2; and each training step s, we adjust Eq. (4) and predict the best action as follows:
# (rt) (rampssoftmmxxxT((t (fo, (44))),Pmins Penin Pmna));
(14)
Obest (MASs
(15)
where clamp(', Pmin+ Pmaz) denotes clamping each element of the input vector into the interval [pmin; Pennar, and MS (- 1) denotes a multinomial sampling strategy that samples one action accord- ing to the input score 7o (2:) and returns the subscript of the selected action. To be more specific, each element of softmax (fe (r:))) represents an action's 's score which will be set to pmin if it smaller than Pmin and will be set to Pmoz if it is larger than Pmaz- Overall, we can obtain the
lower-bound sampling probability Piow for each action as follows:
exp(Pmsn)
Plow exp(Pmin) + (N 1) exp(Pmaaa
(16)
which means that each action has at least the probability of Piow to be sampled. is calculated by assuming the most extreme case, where only one action's 's weight (i.e., one element of (z(2))) is Pmin and the others are Pmorn after clamping.
3.5 FULL ALGORITHM
There are three main modules in our AUDR framework: the encoder, the Actor, and the Critic. These modules are updated separately at the training phase and promote each other. Concretely, the encoder is learned by minimizing the SimSiam-based UCL. loss with an adaptively adjusted mixup strategy. The Actor aims to maximize the prediction score of the Critic. The Critic improves its prediction preciseness through a continual TD-error. For easier and clearer understanding, we summarize the full training process of our AUDR as follows: (1) The Actor selects the best action from the action space by Eq. (14) and Eq. (15); (2) The encoder is updated with the selected action by Eq. (6); (3) The Actor is updated with the predicted reward of the Critic by Eq. (7); (4) The Critic is updated with the continual TD-error by Eq. (13). Note that the Actor-Critic architecture is removed the test phase and only the encoder is evaluated with KNN classifier as in other UCL methods. We present the pseudocode of the full algorithm of our AUDR in Appendix A.
Furthermore, it is worth noting that defining the update direction (or the action space) in the mixup style as we did above is not the only way to construct our AUDR. In fact, this adaptive paradigm could be integrated into many other UCL works. For instance, DER (Buzzega al., 2020) is a representative work of regularization baaubarrzzaion methods, which maintains penalty function to align the representations of old and new models. Let the weight of the penalty function the final loss be the action, we could define new action space for our AUDR. We conduct experiments and provide more details in Sec. 4.4 to show that our AUDR+DER obtains great performance improvements over DER, which demonstrates that our AUDR is indeed extensible and versatile.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We train and evaluate our model on three benchmarks for in-dataset UCL: (1) Split CIFAR-10 (Krizhevsky, 2009) contains 5 tasks, with 2 classes (randomly sampled from 10 classes) per task. (2) Split CIF (Krizhevsky, 2009) consists of 20 tasks, with 5 classes (randomly sampled from 100 classes) per task. (3) Split Tiny-ImageNet is subset of ImageNet (Deng al., 2009) which has 20 tasks with 5 classes per task. We also evaluate our models (pre-trained on Split CIFAR-10 or Split CIFAR- 100) on three benchmarks for cross dataset UCL: MNIST (LeCun, 1998), Pashion-MNIST (FMNIST) (Xiao et al., 2017), and SVHN (Netzer et al., 2011).
Evaluation Protocol. We follow recent works (Mirzadeh et al., 2020; Madaan et al., 2021) to establish the evaluation protocol of UCL. Formally, let Ar, denote the test accuracy of the model on task é after trained on task f, where the total number of tasks is N. We define two metrics: (1) Average Accuracy denotes the average classification accuracy of the model on all learned tasks after sequential training: Accuracy = 1/4 Σ AN.I-

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison results with state-of-the-art methods under the in-dataset UCL setting on three benchmarks: Split CIFAR- 10, Split CIFAR- 100 and Split ImageNe.. "Accuracy" denotes aver- age accuracy and "Forgetting" denotes average forgetting. "Multi Muulti-Takk" is the upper-bound method
which is based on multi-task learning. Standard deviation results are shown in brackets.
Split CIFAR-10
Split CIFAR-100
Split Tiny- ImageNet
Method
Accuracy (1) Forgetting (L) Accuracy (1) Forgetting (1) Accuracy (1) Forgetting ())
Multi- Task
95.76 (2000)
86.31 (20.38)
82.89 (20,9))
Finetune
90.11 (20.12)
5.42 (80.08)
75.42 (a0.7)
10.19 (2037)
71.07 (20.20)
9,48 (30.5)
PNN (Russ 2016)
90.93
66.58
62.15 (+1.28)
SI (Zenke et 2017))
92.75 (e0.06)
1.81 1 (20.21)
80.08 (21
5.54 (at 30)
72.34 (20.42)
8.26 (49.60
DER (Bureega et al., 2020)
91.22 (00030)
4.63 (60.26)
77.27 (al/30)
9.31 (2009))
71.90 (=7.44)
8,36 (42.00
LUMP (Madaan et al., 2021)
91.00 (20.40)
2.92 (80.5))
82.30 35)
4.71 tel 52)
76.66 (=2.29)
3.54 (+1.04)
Ours
93.29 (18.11)
1.72 (10.12)
84.04 (on.11)
3.35 (10.16)
77.67
3.48
Table 2: Comparison results with the state-of-the-art methods under the cross-dataset UCL setting. All models are pre-trained on Split CIFAR- (or Split CIFAR-100), and then directly evaluated on MNIST, FMNIST, SVHN and Split CIFAR-100 (or Split CIFAR-10).
Split CIFAR-10
Split CIFAR-100
Method
MNIST
FMNIST
SVHN
CIFAR-100
MNIST
FMNIST
SVHN
CIFAR-10 10
Multi-Task
90.69 (1.17)) 80.65
0342 47.67
(a44)
39.55
(u)
90.35
u026 81.11
(1:5)) 52.20
anso 70.19 u015)
Finetune
89.23 (40.9) 80.05
2834) 49.66 (m0)))
34.52 12)
85.99 uss 76.90 (a.11) 50.09 (114)) 57.15 (=099)
SI (Zenke et 2017)
93.72 (40.5)) 82.50
57.88 ca10
36.21 mm
91.50
(1:26) 80.57 (400)) 54.07 (2:79) 60.55 (:330)
DER (Buzzega al., 2021)
88.35 (40,3)) 79.33 uses)
48.83 (,3)))
30.68
87.96 u200: 76.21 (0a)) 47.70 an9))
56.26
LUMP (Madaan al., 2021)
91.03 (ed.22) 80.78
45.18 (..57)
31.17 (3)
91.76
a1.77 81.61 (2045) 50.13 (2071) 63.00 (20.3)
Ours
93.98 (e0.3)) 83.78 (.8.3)) 55.95 (=1.76) 39.77 (40,53) 94.34 (4:31) 83.09 (40.3)) 55.29 (48.80) 69.33 (4842))
(2) Average Forgetting is the average performance decrease of the model on each task between its 1 E (AA), - AN,s). maximum accuracy and the final accuracy: Forgetting = N-T
Implementation Details. All baseline methods and our AUDR are implemented based on Sim- Siam (Chen & He, 2021) with R8 (He et al., 2016) as the backbone encoder. The Actor and the Critic are both MLP-based structures. Concretely, the Actor has 4 linear layers and the Critic has similar architecture with one more 3-layer MLP head which is adopted for clipped dou- ble Q-learning (Van Hasselt et al., 2016 6; Fujimoto et al., 2018). We provide more details for the Actor-Critic architecture in Appendix B. To obtain our main results and make fair comparison to recent competitors, we follow LUMP (Madaan et al., 2021) to average the evaluation results over three independent runs with different random seeds. More details are given as follows: (1) At the training phase, all images are randomly augmented into two views by horizontal-flip. color-jitter, gaussian- and gcay-scale. (2) We train our model for 200 epochs per task (the same for all baseline methods). (3) The learning rate is set to 0.03 and the memory buffer size is set to 256 (as in all baseline methods). Our action space has 10 actions which are uniformly sampled from [0, 1] while the minimum action score Pmin is set to 0.08. The source code will be released soon.
4.2 COMPARISON TO STATE-OF METHODS
In-Dataset UCL. Table I shows the accuracy and forgetting results of our AUDR under the in-dataset UCL setting on three datasets: Split CIFAR-10, Split CIFAR-100, and Split Tiny- ImageNet. We compare our AUDR model with recent representative methods including expansion- based method PNN (Rusu et al., 2016), regularization- based methods SI (Zenke et al., 2017) and DER (Buzzega et al., 2020), and rehearsal-based method LUMP (Madaan et al., 2021). Note that Finetune denotes the lower-bound of UCL, which means directly finetuning the model across all the tasks without any continual learning strategies. The upper- of UCL is Multi-Task, which is to simultaneously train the model on all tasks and thus it has no forgetting results. All the results for the baseline methods are directly copied from LUMP (Madaan et al., 2021). It can be clearly seen that our AUDR achieves new state-of tftthe-rrt results on all three datasets. Particularly, our AUDR outperforms the second-best method LUMP by 1.68% on accuracy and 0.87% forgetting (com-
7

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study results for our AUDR. Three groups of experiments are constructed to show the impact of different (a) reward functions, (b) action spaces, and (c) samping strategies.
(a) Different reward functions.
(b) Different number of actions.
(c) Different sampling strategics.
Split CIFAR-10
Split CIFAR-I0
Split CIFAR-10 10
Method
Accuracy Forgetting
Method
Accuracy Forgetting
Method
Accuracy Forgetting
None
91.00
2.92
Naet =
91.00
2.92
random
91.11
3,92
Lewr
91.17
2.59
Nast =
92.92
2.32
learnable
91.30
3.14
Lenewn
89.83
0.55
Naet 10 93.29
1.72
w/o MS
91.94
2.97
Leur+ Limm
93.29
1.72
Nact 20
92.62
2.94
w/ MS
93.29
1.72
paring average results on all three datasets). Note that the performance of our AUDR is remarkable since it is quite close to the upper-bound (e.g., 84.04 vs. 86.31 on Splic CIFAR-100),
Cross-Dataset UCL. Table 2 shows the accuracy and forgetting results of our AUDR under the cross-dataset UCL setting. We first train our model on the training set of Split CIFAR-10 (or Split CIFAR- 100) and then directly evaluate on MNIST, FMNIST, SVHN and Split CIFAR- (or Split CIFAR-10). We can observe that: (1) Our AUDR beats the second-best method SI (Zenke et al., 2017) in out of 8 cases and outperforms it by 2.32% in average, which demonstrates that our AUDR has stronger generalization ability. (2) The results of multi-task learning (i.e., Multi-Task) are not the upper-bound under cross-dataset UCL. This suggests that the model's generalization ability can be better enhanced by UCL (e.g., AUDR and SI) than by multi-task learning.
4.3 ABLATION STUDIES
There are three core designs in our AUDR to help transfer the Actor-Criti Critic architecture to the UCL setting: (1) reward function, (2) an action space, and (3) the multinomial sampling strategy. We thus separately conduct experiments to analyze the contributions of them in Table 3.
Impact of Different Reward Functions. Table 3a shows the results of our AUDR with different reward functions. Note that we have defined the original reward function in Eq. (10), which has two components: Leur and Linen We can observe that the results obtained by AUDR with Leur+ Lonem are better than those obtained by AUDR with none reward function (i.e., 1st row) or with only one of Lear and Lmmm (i.e.. 2nd and 3rd rows). Therefore, the reward function should balance the model performance on both old and new tasks for UCL.. The reason why the forgetting results of Tumem is smaller than Tecur + T.mem (0.55 vs. 1.72) lies in the much lower accuracy results of T.mem. which means that it doesn't 't learn the old knowledge well and thus it has little knowledge to forget.
Impact of Different Action Spaces. We present the results of our AUDR with different action spaces (i.e., different number of actions) in Table 3b. We compare models with Nact E {0, 5, 10, 20} and find that: (1) Learning to select actions with our AUDR is better than randomly sampling them from fixed distribution (i.e., 1st row vs. others). (2) More actions do not always lead to better performance (i.e., 3rd row vs. 4th row). In our opinion, learning with more actions means having a more difficult training process for the Actor-Critic module, especially with limited training iterations (i.e., 200 epochs per task for fair comparison with baseline methods).
Impact of Different Sampling Strategies. Table 3c shows the results of our AUDR with different sampling strategies. We compare our multinomial sampling (MS) strategy with three other alterna- tives: random sampling (Ist row), learnable parameter to directly predict the action's 's value (2nd row), and without our MS strategy (3rd row). We have the following observations: (1) The Actor- Critic module achieves less satisfactory performance without multinomial sampling (MS) strategy (i.c.. 3rd row vs. 4th row), which demonstrates the effectiveness of MS. (2) Models with discrete action space are easier to be trained than those with continuous action space (i.e., 2nd row vs. 3rd row). Note that the 2nd row of Table 3c contains the results of our AUDR with learnable parameter to represent the action. In other words, has a continuous action space [o, 1]. which can be regarded as a space with infinite actions. Training the parameter in this way, its value will have little change between two training steps (e.g., 0.96 and 0.90). However, the optimal actions between these two steps may have large margin (e.g.. 0.96 and 0.16). This hypothetical situation is possible (and common) due to the random sampling and the distribution shift in UCL. Therefore, a discrete action space is more suitable than continuous action space for AUDR.
8

---

Under review as a conference paper at ICLR 2023
tosk
tzek5
Accurary
Forgetting
Probahiiity
6.396
92
Fongeting 5
0.09
0.07
0.05
88
1
3
5
6
8
9
10
Pantune
DER
AUDR+DDER
Actions
Mithods
Probability distributions different actions.
(b) Results for our AUDR+DER on Split SpFAR 10.
Figure 3: Further analysis for AUDR. (a) The probability distributions of sampled actions for two different tasks (task and task 5) on Split CIFAR-10. (b) Accuracy and forgetting results for re-dggieeed AUDR with DER (i.e., AUDR+DER) on Split CIFAR-10.
4.4 FURTHER ANALYSIS
Action Distributions of Different Tasks. To validate the effectiveness of the Actor-Critic module in our AUDR, draw sampled action distributions of different tasks from Split CIFAR-10. Con- cretely, we present the results for the first task (task 1) and the last task (task 5) in Figure 3a. Note that the larger the action serial number is, the closer the action value to I (i.c., the coefficient of memory data is higher in Eq. (5)). We can observe that: (1) When training on the last task, our AUDR tends to choose more-higher-value actions than training on the first task (e.g., 8% more ac- tions of [8, 9, 10j and 6.5% less actions of [1, 2, 3, 4J). This demonstrates that the model needs to pay more attention to reviewing the old knowledge in the later tasks. (2) Every action on all tasks has chance to be sampled by AUDR (more than 0.05 in Figure 3a), which is mainly due to multinomial sampling strategy that holds a lower- bound probability for each action.
Re-design AUDR with DER. To demonstrate that AUDR is extensible in UCL, change the mixup strategy to another UCL method DER (Buzzega et al., 2020). Note that DER was first proposed for SCL and then re-implemented for UCL in (Madaan et al., 2021). Formally, the UCL loss for DER is defined as follows:
LDER
+ (Zm..) Fm,ll22,
(17)
where or is fixed coefficient and Fm, denotes the stored features of memory data. The second term of this formula is penalty function to align the outputs of new and old models but with a fixed ratio a. We thus replace with our predicted action Abest to formulate our AUDR+DER:
LAUDR-DER = 1 5 B
(LsimSion (a fe(x?)) + abest (If - Fl.)
(18)
1=1
Figure 3b shows the results for our AUDR+DER on Split CIFAR-10. More results on Split CIFAR- 100 and Split Tiny-Imagell
also presented in Appendix C. It can be seen that our AUDR+DER outperforms DER by a large margin on both accuracy and forgetting. Such success of AUDR+DER demonstrates that proposed adaptive paradigm (i.e., AUDR) highly extensible/versatiee and has great potential in the continual learning field.
CONCLUSION
In this paper, we propose an Actor-Critic framework with adaptive update direction rectification (AUDR) for unsupervised continual learning. We devise three novel designs for our AUDR: (1) reward function considering the model performance on both current and memory data, which provides reliable ground-truti reward for training the Critic; (2) An action space with discrete actions for the Actor to select: (3) A multinomial sampling strategy to ensure the variance of sampled actions. We conduct extensive experiments on three benchmarks to show that AUDR achieves new state-of-te of-te--f-th-arr the-art results for both in-dataset and cross- cross-dataset UCL. Further analysis on action distribution and AUDR+DER demonstrate that AUDR indeed effective and extensible.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. In CoRI., pp- 856-868, 2018.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, pp. 3366-3375, 2017.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurlPS, 12:11817-11826, 2019.
Richard Bellman. Dynamic programming. Science, 153(3731):34 1966.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. NeurIPS, 33:15920-15930, 2020.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV, pp. 2021.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas Griffiths. Automatically composing representation transformations as a means for generalization. arXiv preprint (cXiv:1807.046400 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 2019.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hind- sight to anchor past knowledge in continual learning. arXiv preprint arXiv:2002.08765, 3, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp- 15750-15758, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019.
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Prob- ing representation forgetting in supervised and unsupervised continual learning. In CVPR, pp. 16712-16721, 2022.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pp. 248-255, 2009.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- 2020.
ual learning. In AISTATS, pp.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 1701.0170.00770 2017.
Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In CVPR, pp. 9621-9630. 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In ICML, pp. 1587-1596, 2018.
Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G Bellemare, and Remi Munos. The reactor:
A sample-efficient actor-critic architecture. arXiv preprint arXiv: 04451, 5, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- 2018
cations. arXiv preprint arXiv:
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting, 20(1):5-10, 2004.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In ECCV, pp. 437-452, 2018.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rehalancing. In CVPR, pp. 831-839, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. PNAS, (14(13)355213522 2017.
et al. Overcom-
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, In ICML, pp. 5639-5650, 2020.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In ICML, pp. 3925-3934, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):2935-2947, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. NeurIPS, 30:6470-6479, 2017.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient based hyperparameter optimiza- tion through reversible learning. In ICMI., pp. 21 2113-2122, 2015.
Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In ICLR, 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. NeurIPS, 33:7308-777 -3200, 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, pp. 1928-1937, 2016.
Yao Mu, Shoufa Chen, Mingyu Ding. Jianyu Chen, Runjian Chen, and Ping Luo. Ctrlformer: Learning transferable state representation for visual control via transformer. arXiv preprint arXiv:2206.0880 08888, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682 697, 2008
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In ICCV, pp. 1320-1328, 2017.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. NeurIPS, 32:7647-7657, 2019.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pp. 2001-2010, 2017.

---

Under review as a conference paper at ICLR 2023
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint a7X11:1810.11910, 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 14Xx::6606.0661,, 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning- arXiv preprint arXiv:2103.09762, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NeurIPS, 12:1057-1063, 1999.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. In AAAI, pp. 2094-2100, 2016.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139-149 2022.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge, 1989.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, pp. 3733-3742, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Mushion-mmiste a novel image dataset for benchmark- 2017, 2017.
ing machine learning algorithms. arXiv preprint
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing
deep reinforcement learning from pixels. In ICLR, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.00645,, 2021.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pp. 1-120-1 2021.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995, 2017.
12

---

Under review as a conference paper at ICLR 2023
A FULL ALGORITHM OF AUDR
We provide the pseudocode of the full algorithm for our AUDR in Algorithm 1.
Algorithm Pseudocode of AUDR.
Input:
Encoder fe with parameters 0;
Actor T6 with parameters Φ;
Critic Q₂ with parameters yp;
Target Critic 2,p with parameters ypri
An action space A = {ayli 112, 1,2, Nact) of Nact actions;
A dataset D [D1, D2. : . of N tasks.
Output: The learned fü
1: Initialize the Target Critic 2pp. = -2pi
2: for all task 1, 2, N do
3: for all iteration = 1.2. MMaaaMaeaaiin do
4: Sample mini-batch with B images (mi))=i
5:
Obtain the best action Gbest by the Actor with Eqs. (14-15); Obtain UCL loss Luct with Eqs. (5-6);
6:
7:
Update fe using SGD and obtain fe., ferti
8:
Obtain the predicted reward R..; with Eq. (7);
9:
Update the Actor as using SGD;
10:
Obtain ground-truth reward Rg with Eqs. (8-10);
11:
Obtain the target reward R5-1,; with Eq. (12);
Obtain the continual TD-error Leritie with Eq. (13);
13:
Update the Critic Qu using SGD;
Update the target Critic 2p, using EMA with Eq. (11);
15:
end for
end for
17:
return the found best fö.
B MORE IMPLEMENTATION DETAILS AND DISCUSSIONS
Details of Actor and Critic. The structures of our Actor and Critic networks are the same in DrQv2 (Yarats et al., 2021), where the Actor has I-layer trunk network (Lynear+Layemmomm= Tanh) with one 3-layer MLP head and the Critic has 1-layer trunk network with two 3-layer MLP heads. The Critic trained by clipped double Q-learning (Van Hasselt et al., 2016; Fujimoto et al., 2018) to alleviate the over-estimation problem, where two MIP heads represent two Q-functions Q Qup which separately predict the rewards R1,, R2 Then the final predicted reward Rs,, of Eq. (7) is obtained by: Rg,, min RA,, R2
Discussions on Comparing AUDR with Other Possible Methods. The core idea of our AUDR is to adaptively rectify the update direction during training. In this work, the instantiated "update direction" based on the mixup ratio (of AUDR) or the penalty loss weight (of AUDR+DER), which is a pre-defined byper-parameter of the original method (LUMP DER). In addition to AUDR, there are three possible approaches to adjusting the update direction during training: (1) Directly using learnable parameter (through MLP layer) represent the update directions. The main drawback of this method lies in the slight change of the action value between two steps, which has already been discussed in Sec. 4.3.
(2) Finding the best hyper- parameters by meta-learning (Maclaurin et al., 2015) or reinforcement learning (Cubuk et al., 2019). Different from their objectives of finding the best hyper-paranen combination by training the models several times (each has a whole training process), our AUDR an online method to adaptively rectify the hyper-parameter (i.c., the update direction) and thus the syper-paramete is continuously changing during training instead of fixed. Concretely, those syper-paramee search methods focus on finding an optimal policy for a neural network to solve specific task (i.e., the best parammee has fixed value once found). When transferring them to the UCL setting, they would search the action space to find the best (but fixed) ratio. Since each exploration step requires whole training process, the computation cost of their schema is enormous
13

---

Under review as a conference paper at ICLR 2023
Accuracy
Forgetting
Accuracy
Forgetting
Acsuiny
Foreeling
Acxinay
71
69
Plneuzee
DER
AUDR+DER
Dinetuze
DER
AUDR+DIR
Method
Results our ADDR on Split CIFAR-100.
Results AUDREDER on Split ImageNet.
Figure 4: Results for our AUDR+DER on the other two datasets.
Lactor
Split CIFAR-10
500
Loss
0-
-500
20000
30000
40000
400
Split CIFAR-100
Loss
-200
-400
5000
10000
15000
20000
25000
Split Imageet
500
Loss
-500
10000
15000
20000
25000
35000
40000
training iterations
Figure 5: Visualization of Actor Loss during training.
(e.g., 10 trials will lead to 10 times training complexity). The online schema of our AUDR with the Actor-Critic architecture instead only in need of one whole training process to adjust the mixup ratio at each step, which is significantly more efficient.
(3) Learning task-specific prompts (Wang et al., 2022) to be used in the test phase. Differently, the extra Actor-Crite Critic module of our AUDR is removed during testing which is more applicable and resource-saving in real-world application scenarios (e.g., large-scale pre-training).
C MORE EXPERIMENTAL RESULTS
We provide re-designed version our AUDR our main paper and present the experimental results Split CIFAR- 10 (see Figure 3b). For more comprehensive study, we provide the results of our AUDR+DER on Split CIFAR- 100 and Split Tiny-ImageNet in Figure 4, We can observe that AUDR+DER achieves better performance than the competitors in all cases.
14

---

Under review as a conference paper at ICLR 2023
Table More results for our AUDR over Split-CIFAR-00. deeotees that our model is trained with 300 epochs per task (otherwise 200 epochs by default).
(a) Different Actor-Critic layers.
(b) Longer epochs for 20 actions.
(c) Training with BarlowTwins.
Split CIFAR-10
Split CIFAR-
Split CIFAR-I0
Method
Accuracy Forgetting
Method
Accuracy
Forgetting
Method
Accuracy
Forgetting
None 91.00
2.92
5
92.92
2.32
Finetune
87.72
4.08
2-layer 91.47
3.55
Na
93.29
1.72
DER
88.67
2.41
4-layer 93.29
1.72
No
92.62
2.94
LUMP
90.31
1.13
6-layer 93.69
1.61
Nac==20
93.60
2.61
AUDR-BT
91.53
1.98
Table 5: Detailed ablation studies. "Max Accuracy" denotes the maximum accuracy on each task over Spilt-CIFAR-10 during training.
Max Accuracy
Overall
Method
Task
Task 2
Task 3
Task 4
Task5
Accuracy
Forgetting
LImem
92.90
83.10
88.45 92.81
92.45 95.60
94.50 96.08
89.83
0.55
Leur
Lmem
97.25
91.60
93.29
1.72
D VISUALIZATION OF ACTOR LOsS
To verify the training stability of the Actor-Critic architecture applied in our AUDR, we present the plot of the training loss of the Actor Figure 5. can be seen that minimizing the actor loss Lactor is generally stable during training and finally converges on all datasets. Note that there are relatively large fluctuations during training, which normal since the distribution of data is constantly changing across sequential tasks.
E FURTHER EVALUATION
Different Actor-Critic Architectures. We present more results for different Actor-Critic architec- tures in Table 4a. The MI.P-based modules of both Actor and Critic are set to have the same number of layers (0, 2, 4, or 6). We can observe that larger MIP used for Actor/Critic indeed has stronger learning ability and thus leads to better results.
Training AUDR with More Epochs. As shown in Table 3b of our main paper, when the number of epochs is limited (e.g., 200 epochs per task), training our AUDR with a larger action space becomes more difficult and thus suffers from performance degradation. To make further verification, train our model with 20 actions for more (i.e., 300) epochs and present the new results Table 4b. We find that it leads to better performance, which confirms our assertion.
Training AUDR with Barlow-Twine To show the generalization ability of our AUDR, we change the SimSiam loss with another unsupervised learning loss Barlow-Twins (Zbontar et al., 2021), and then denote our method as AUDR-BT. Experimental results are shown in Table 4c. The results of the competitors are directly copied from LUMP (Madaan al., 2021). We can observe that our AUDR-BT still achieves the best accuracy on Split CIFAR-10.
More Detailed Ablation Study. We provide more detailed results about the ablation study for training our AUDR with only Lmem in Table 5. It achieves much lower overall forgetting rate due the lower maximum accuracy (i.e., Max Accuracy) on each task. However, performs significantly worse than training our AUDR with Lear + Tinem in terms of overall accuracy.
15


Reviews:
[{"review_id": "syaxJrfWLL", "reviewer": "Reviewer_kqaK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the recently proposed setting of unsupervised continual learning (UCL), which I understand as generally studying the extent that unlabelled information can be exploited in a continual learning setting leveraging self-supervised learning. The authors build a new approach, AUDR, extending the recent paper LUMP that uses an actor-critic paradigm to guide learning. The authors note that the three core design features of their algorithm include the proposal of a reward function, action space, and multinomial sampling strategy that lead to improvements in this setting. AUDR is validated with respect to relevant baselines and ablations in both within-dataset and cross-dataset transfer settings. ", "summary_of_the_review": "As highlighted above, I think that there are many strong elements to this paper. I think it explores an important topic and has a pretty solid set of empirical settings and results. I also generally like the idea of using RL to improve continual supervised learning. My main concerns that keep me on the edge about this paper are that the contributions are quite incremental and that the design decisions are not really formally justified within some theoretical framework or conception of the problem. A nagging concern of mine is also that the results may be quite sensitive to hyperparameters given all of the moving parts. \n\nUpdate After Feedback: \n\nI appreciate the note by the authors about the code and the experiment applying learning for more epochs. This positioning makes more sense to me than in the original case. However, the comment about the main innovations is consistent with my previous understanding. Moreover, I am unsure about the action space design contribution as while it may improve computational efficiency (as the action space size is tied to sample efficiency in RL theory), it is unclear how this balances with the cost of tuning this hyperparameter in general. ", "strength_and_weaknesses": "Strengths: \n- Leveraging unlabelled information during continual learning could be very impactful with widespread use if done correctly. \n- AUDR is shown to outperform the baselines UCL method LUMP in both in-dataset and cross-dataset settings. \n- AUDR is shown to be extensible and able to build off other existing continual learning methods like DER.\n- The reward function used is shown to add value when trained with a td style critic. I wonder if a simple sum of these two values would more generally be better stated as a weighted combination. However, what the authors have proposed is simple and requires less tuning. \n- The authors have highlighted the value of the multinomial sampling strategy in ablations.\n- I quite like the idea of using RL approaches to improve supervised continual learning! It reminds me of the discourse on this topic in Khetarpal et al., \"Towards Continual Reinforcement Learning: A Review and Perspectives.\" 2020. \n\nWeaknesses: \n- The main innovations of the paper are quite incremental on top of previous work in slightly different domains. \n- The paper is mainly driven by high-level intuitions and most design decisions are not really formally justified. \n- I am not sure if I read the result in Table 3b as positive. The authors try to highlight this as a contribution, but it seems to me more like highlighting that the approach is brittle to a particular hyperparameter that is not so clear how to set apriori. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_6uInwmeTt", "reviewer": "Reviewer_6y1n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper devises a strategy for unsupervised continual learning by formulating gradient updates as actions taken by an actor whose loss is determined by a critic approximating the future SimSiam loss value. Specifically, the predictions of the actor determine how much memory and current data are weighted in the encoder SimSiam loss. To increase the diversity of the actor predictions, they clamp the lower bound of the probability of any one action. They evaluate their algorithm on CIFAR-10, CIFAR-100, Tiny-ImageNet, and on a cross dataset evaluation. Their method outperforms the prior art on accuracy and they also show that their method maintains reasonable performance over a couple of different choices of reward functions and action spaces.", "summary_of_the_review": "Overall I think there are promising results in the paper, but I would appreciate if the authors could address my clarification questions and demonstrate that the method is reliable across different unsupervised learning losses.", "strength_and_weaknesses": "The greatest strength of the paper is its results: the method attains reasonable performance improvements over baselines, although many are within a standard deviation of other methods. It is also the first work to cast unsupervised continual learning as reinforcement learning to my knowledge. The paper would be improved by demonstrating that the method is applicable across different loss types (e.g., aside from just simsiam). The authors could also compare against P-MNIST, R-MNISt, and MNIST-360 as DER does. I would be comfortable accepting the paper if the method was reliable across different unsupervised learning variants and if the authors address some of the clarification points below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nth9HLHtLU", "reviewer": "Reviewer_GyrZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs an Actor-Critic to learn the optimal mix-up strategy for performing Unsupervised Continual Learning (UCL). The corresponding framework, named AUDR, enables adaptively rectifying the update direction in UCL according to its performance under different mixtures of memory data and incoming data. AUDR includes three major designs including (1) A reward function for measuring the immediate performance after applying a mixture strategy; (2) An action space for representing the mixture strategy; (3) A multinomial sampling strategy based on the softmax function and min-max clamping. The experiments, including several ablation studies under different datasets, empirically demonstrate the performance of the proposed model.", "summary_of_the_review": "The paper introduces an intriguing framework that enables the application of RL into UCL. The main ideas are well explained. The main claims are correct (to the best of my knowledge). The claims are supported by empirical results. Since the reviewer's background is mainly about RL, the comments are mostly based on the actor-critic design in this work. Although some notations need to be revised, the general quality is satisfying. I am hoping the authors respond to my worries. All in all, I vote for a weak acceptance. ", "strength_and_weaknesses": "\n### Strength\n\n1. The paper is well-structured, and the main ideas are easy to follow.\n2. The experiments validate the performance of AUDR from multiple perspectives, and the results are generally promising.\n3. The application of actor-critic to solve UCL problems is new. To the best of my knowledge. AUDR or similar framework has not been studied by previous works.\n\n### Weaknesses\n\n1. **Running Complexity.** Training an RL agent often requires multiple runs of explorations and exploitation to update the policy and the value function (e.g., Actor and the Critic networks). Compared to the traditional life-long mixup strategy, AUDR has a larger time and space complexity for building and updating the RL models. The increase in complexity is more significant in a continual learning environment, which often includes multiple tasks (e.g., N tasks in the paper). The planning horizon is very long, compared to a single-task environment. \n\n2. **Continual Reinforcement Learning** I believe the paper has not strictly followed the Continual Reinforcement Learning setting [1], where interventions are not allowed. For example, the game won't end, and movements cannot be reversed. If an action $a$ is performed in a state $s$, we won't know the outcome of other actions, so the vanilla exploration used by traditional RL works will be impossible. The agents must continuously adapt to the environment. If this work does follow this setting, please clarify. \n\n[1] Khimya Khetarpal, Matthew Riemer, Irina Rish, Doina Precup. Towards Continual Reinforcement Learning: A Review and Perspectives. \n\n3. **Some Notations have been misused.**\n\n- $R_{s,i}$ in formula (13) does not indicate reward. In fact, $R_{s,i}$ is the expectation of discounted cumulative rewards by following the current policy (represented by the actor). Please denote it by $Q$ instead of $R$.\n\n- Formula (4) puts a softmax function on top of a policy function. This is abnormal since a policy commonly predicts either the probabilities for discrete actions or the expectation values (assuming the risk-neural policy) for continuous actions. In this work, the action space is discrete, so $\\pi$ defines the probabilities for each of the candidate actions. In this case, why a softmax layer is necessary?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L7InTv-a3a", "reviewer": "Reviewer_1R3g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on Unsupervised continual learning (UCL) and proposes Adaptive Update Direction Rectification (AUDR), an adaptive learning paradigm for the UCL setting. Mainly, the paper uses an Actor-critical approach, where the actor selects the best action and is updated with the predicted reward by the Critic. It proposes a reward function based on the current task and replay-buffer performance to guide the Critic’s training, which is updated using the continual TD error. During the evaluation, the paper compares the learned encoder with prior methods, showing superior performance on multiple benchmark datasets.", "summary_of_the_review": "The paper proposes a novel and interesting approach to continual learning; however, the paper can be strengthened further. I am happy to increase my score if the authors address the above concerns. Notably, It lacks a comparison to prior hyper-parameter search approaches and is limited to two choices of hyper-parameters.", "strength_and_weaknesses": "I am well-familiar with the literature and read the full paper in detail. Accordingly, I'll describe the strengths and weaknesses of the paper in the order of originality + quality, clarity, and reproducibility. \n\n## Originality and Quality\n### Strengths \n* The choice of hyper-parameters plays a vital role in most rehearsal-based methods. The paper proposes an interesting approach for modeling the hyper-parameter search for UCL as an actor-critic framework and would interest the CL community.\n* The proposed AUDR framework is flexible and applicable to various prior CL methods, as also demonstrated in the empirical evaluation of the paper.\n\n### Weaknesses\n* The paper's objective is primarily a hyper-parameter search for UCL, focusing on the mixup ratio or penalty loss weight. The paper includes a discussion of various hyper-parameter search methods in the appendix. Still, the efficiency of the proposed method over the prior methods needs to be clarified in the current form. The paper claims that it is an \"online\" method and more efficient than other approaches; however, in this case, it should show a comparison with these methods and highlight the efficiency of the proposed method.\n* The proposed framework has also been restricted to two hyper-parameters and can be further strengthened by incorporating other hyper-parameters into AUDR.\n* AUDR also requires additional MLP-based architectures for both Actor and Critics. The paper should include ablation and a discussion on the choice of these architectures in the UCL setting. \n\n---\n\n## Clarity\nThe paper was well-written and easy to follow. I have a few suggestions and clarifying questions:\n* While the paper focuses on LUMP, I suggest updating the notations and figures to the proposed AUDR as a general framework applicable to adapt the buffer hyper-parameters of prior CL methods to strengthen the proposed method.\n* The paper highlights that more actions only sometimes lead to better performance, possibly due to limited training iterations. I recommend increasing the training epochs for each task to check if that makes the training process more efficient and improves performance.\n* The paper should also compare the generated mixup examples using AUDR and LUMP. Additionally, it would be beneficial to include a discussion on the selected actions with high rewards for both LUMP and DER compared to the hyper-parameters used in prior works.\n* The paragraph before subsection 3.2 - In addition, it is not … -> The sentence is incomplete.\n* Section 4.2, last line of paragraph 1, on on -> on\n* The paper mentions that forgetting for $L_{mem}$ is smaller than the combination of losses because it did not learn the old knowledge well. I suggest supporting this statement with an accuracy comparison of prior tasks during training in the appendix.\n* The references are inconsistent, NIPS and NeurIPS are randomly used interchangeably, and few articles don't use the conference bibliographies.\n\n---\n\n## Reproducibility\n The code is not provided with the submission. Since the paper is empirical, it is necessary to provide the code to aid the reproducibility of future works.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject