Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on Unsupervised continual learning (UCL) and proposes Adaptive Update Direction Rectification (AUDR), an adaptive learning paradigm for the UCL setting. Mainly, the paper uses an Actor-critical approach, where the actor selects the best action and is updated with the predicted reward by the Critic. It proposes a reward function based on the current task and replay-buffer performance to guide the Critic’s training, which is updated using the continual TD error. During the evaluation, the paper compares the learned encoder with prior methods, showing superior performance on multiple benchmark datasets.
strength_and_weaknesses: I am well-familiar with the literature and read the full paper in detail. Accordingly, I'll describe the strengths and weaknesses of the paper in the order of originality + quality, clarity, and reproducibility. 

## Originality and Quality
### Strengths 
* The choice of hyper-parameters plays a vital role in most rehearsal-based methods. The paper proposes an interesting approach for modeling the hyper-parameter search for UCL as an actor-critic framework and would interest the CL community.
* The proposed AUDR framework is flexible and applicable to various prior CL methods, as also demonstrated in the empirical evaluation of the paper.

### Weaknesses
* The paper's objective is primarily a hyper-parameter search for UCL, focusing on the mixup ratio or penalty loss weight. The paper includes a discussion of various hyper-parameter search methods in the appendix. Still, the efficiency of the proposed method over the prior methods needs to be clarified in the current form. The paper claims that it is an "online" method and more efficient than other approaches; however, in this case, it should show a comparison with these methods and highlight the efficiency of the proposed method.
* The proposed framework has also been restricted to two hyper-parameters and can be further strengthened by incorporating other hyper-parameters into AUDR.
* AUDR also requires additional MLP-based architectures for both Actor and Critics. The paper should include ablation and a discussion on the choice of these architectures in the UCL setting. 

---

## Clarity
The paper was well-written and easy to follow. I have a few suggestions and clarifying questions:
* While the paper focuses on LUMP, I suggest updating the notations and figures to the proposed AUDR as a general framework applicable to adapt the buffer hyper-parameters of prior CL methods to strengthen the proposed method.
* The paper highlights that more actions only sometimes lead to better performance, possibly due to limited training iterations. I recommend increasing the training epochs for each task to check if that makes the training process more efficient and improves performance.
* The paper should also compare the generated mixup examples using AUDR and LUMP. Additionally, it would be beneficial to include a discussion on the selected actions with high rewards for both LUMP and DER compared to the hyper-parameters used in prior works.
* The paragraph before subsection 3.2 - In addition, it is not … -> The sentence is incomplete.
* Section 4.2, last line of paragraph 1, on on -> on
* The paper mentions that forgetting for $L_{mem}$ is smaller than the combination of losses because it did not learn the old knowledge well. I suggest supporting this statement with an accuracy comparison of prior tasks during training in the appendix.
* The references are inconsistent, NIPS and NeurIPS are randomly used interchangeably, and few articles don't use the conference bibliographies.

---

## Reproducibility
 The code is not provided with the submission. Since the paper is empirical, it is necessary to provide the code to aid the reproducibility of future works.

clarity,_quality,_novelty_and_reproducibility: I elaborate on all these aspects in the above section.
summary_of_the_review: The paper proposes a novel and interesting approach to continual learning; however, the paper can be strengthened further. I am happy to increase my score if the authors address the above concerns. Notably, It lacks a comparison to prior hyper-parameter search approaches and is limited to two choices of hyper-parameters.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider the recently proposed setting of unsupervised continual learning (UCL), which I understand as generally studying the extent that unlabelled information can be exploited in a continual learning setting leveraging self-supervised learning. The authors build a new approach, AUDR, extending the recent paper LUMP that uses an actor-critic paradigm to guide learning. The authors note that the three core design features of their algorithm include the proposal of a reward function, action space, and multinomial sampling strategy that lead to improvements in this setting. AUDR is validated with respect to relevant baselines and ablations in both within-dataset and cross-dataset transfer settings. 
strength_and_weaknesses: Strengths: 
- Leveraging unlabelled information during continual learning could be very impactful with widespread use if done correctly. 
- AUDR is shown to outperform the baselines UCL method LUMP in both in-dataset and cross-dataset settings. 
- AUDR is shown to be extensible and able to build off other existing continual learning methods like DER.
- The reward function used is shown to add value when trained with a td style critic. I wonder if a simple sum of these two values would more generally be better stated as a weighted combination. However, what the authors have proposed is simple and requires less tuning. 
- The authors have highlighted the value of the multinomial sampling strategy in ablations.
- I quite like the idea of using RL approaches to improve supervised continual learning! It reminds me of the discourse on this topic in Khetarpal et al., "Towards Continual Reinforcement Learning: A Review and Perspectives." 2020. 

Weaknesses: 
- The main innovations of the paper are quite incremental on top of previous work in slightly different domains. 
- The paper is mainly driven by high-level intuitions and most design decisions are not really formally justified. 
- I am not sure if I read the result in Table 3b as positive. The authors try to highlight this as a contribution, but it seems to me more like highlighting that the approach is brittle to a particular hyperparameter that is not so clear how to set apriori. 
clarity,_quality,_novelty_and_reproducibility: The writing quality of this paper is pretty good. The quality of the experiments is definitely sufficient for work in this area, considering a number of settings and important ablations. The paper contributes to an important emerging area, pushing the boundaries of what can be accomplished without labels. However, the techniques considered are of incremental novelty in light of the previous literature that has considered all of the main ideas before, albeit in slightly different settings. The authors were not able to release their code in time for publication, but promised to make it publicly available shortly. 
summary_of_the_review: As highlighted above, I think that there are many strong elements to this paper. I think it explores an important topic and has a pretty solid set of empirical settings and results. I also generally like the idea of using RL to improve continual supervised learning. My main concerns that keep me on the edge about this paper are that the contributions are quite incremental and that the design decisions are not really formally justified within some theoretical framework or conception of the problem. A nagging concern of mine is also that the results may be quite sensitive to hyperparameters given all of the moving parts. 

Update After Feedback: 

I appreciate the note by the authors about the code and the experiment applying learning for more epochs. This positioning makes more sense to me than in the original case. However, the comment about the main innovations is consistent with my previous understanding. Moreover, I am unsure about the action space design contribution as while it may improve computational efficiency (as the action space size is tied to sample efficiency in RL theory), it is unclear how this balances with the cost of tuning this hyperparameter in general.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper devises a strategy for unsupervised continual learning by formulating gradient updates as actions taken by an actor whose loss is determined by a critic approximating the future SimSiam loss value. Specifically, the predictions of the actor determine how much memory and current data are weighted in the encoder SimSiam loss. To increase the diversity of the actor predictions, they clamp the lower bound of the probability of any one action. They evaluate their algorithm on CIFAR-10, CIFAR-100, Tiny-ImageNet, and on a cross dataset evaluation. Their method outperforms the prior art on accuracy and they also show that their method maintains reasonable performance over a couple of different choices of reward functions and action spaces.
strength_and_weaknesses: The greatest strength of the paper is its results: the method attains reasonable performance improvements over baselines, although many are within a standard deviation of other methods. It is also the first work to cast unsupervised continual learning as reinforcement learning to my knowledge. The paper would be improved by demonstrating that the method is applicable across different loss types (e.g., aside from just simsiam). The authors could also compare against P-MNIST, R-MNISt, and MNIST-360 as DER does. I would be comfortable accepting the paper if the method was reliable across different unsupervised learning variants and if the authors address some of the clarification points below.
clarity,_quality,_novelty_and_reproducibility: The description of the formulation of the actor critic objective could be more concise. I believe most RL researchers expect reward design to be a challenge specific to each new setting, so instead of explaining why this is challenging in the last paragraph of the introduction and in the first paragraph of section 3.3, I would devote more time to describing what reward function was picked and why. Similarly, instead of describing the target critic update procedure, the authors could reference which actor-critic algorithm they use and refer readers to the corresponding paper.

Miscellaneous points to clarify:
- Is this setting non-episodic? If so, how is the critic inferring the expected cumulative reward (as is typical in RL)? Under the current description, it looks like the critic passes the policy the next step reward to prevent the policy from taking a bad gradient step.
- Should the critic reward be negated? Else the policy is taking actions to maximize the simsiam loss predicted by the critic.
- The DER paper presents DER and DER++ which attains slightly higher performance on some tasks. Which variant is compared against in this work?
summary_of_the_review: Overall I think there are promising results in the paper, but I would appreciate if the authors could address my clarification questions and demonstrate that the method is reliable across different unsupervised learning losses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper designs an Actor-Critic to learn the optimal mix-up strategy for performing Unsupervised Continual Learning (UCL). The corresponding framework, named AUDR, enables adaptively rectifying the update direction in UCL according to its performance under different mixtures of memory data and incoming data. AUDR includes three major designs including (1) A reward function for measuring the immediate performance after applying a mixture strategy; (2) An action space for representing the mixture strategy; (3) A multinomial sampling strategy based on the softmax function and min-max clamping. The experiments, including several ablation studies under different datasets, empirically demonstrate the performance of the proposed model.
strength_and_weaknesses: 
### Strength

1. The paper is well-structured, and the main ideas are easy to follow.
2. The experiments validate the performance of AUDR from multiple perspectives, and the results are generally promising.
3. The application of actor-critic to solve UCL problems is new. To the best of my knowledge. AUDR or similar framework has not been studied by previous works.

### Weaknesses

1. **Running Complexity.** Training an RL agent often requires multiple runs of explorations and exploitation to update the policy and the value function (e.g., Actor and the Critic networks). Compared to the traditional life-long mixup strategy, AUDR has a larger time and space complexity for building and updating the RL models. The increase in complexity is more significant in a continual learning environment, which often includes multiple tasks (e.g., N tasks in the paper). The planning horizon is very long, compared to a single-task environment. 

2. **Continual Reinforcement Learning** I believe the paper has not strictly followed the Continual Reinforcement Learning setting [1], where interventions are not allowed. For example, the game won't end, and movements cannot be reversed. If an action $a$ is performed in a state $s$, we won't know the outcome of other actions, so the vanilla exploration used by traditional RL works will be impossible. The agents must continuously adapt to the environment. If this work does follow this setting, please clarify. 

[1] Khimya Khetarpal, Matthew Riemer, Irina Rish, Doina Precup. Towards Continual Reinforcement Learning: A Review and Perspectives. 

3. **Some Notations have been misused.**

- $R_{s,i}$ in formula (13) does not indicate reward. In fact, $R_{s,i}$ is the expectation of discounted cumulative rewards by following the current policy (represented by the actor). Please denote it by $Q$ instead of $R$.

- Formula (4) puts a softmax function on top of a policy function. This is abnormal since a policy commonly predicts either the probabilities for discrete actions or the expectation values (assuming the risk-neural policy) for continuous actions. In this work, the action space is discrete, so $\pi$ defines the probabilities for each of the candidate actions. In this case, why a softmax layer is necessary?


clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality. 
The paper is well-structured, and the main ideas are easy to follow.
### Novelty.  
The application of actor-critic to solve URL problems is new. To the best of my knowledge. AUDR or similar framework has not been studied by previous works.
### Reproducibility. 
The code has not been uploaded. It is unclear whether the source code will be released. The hyper-parameters for training are unknown. The training complexity requires more study.
summary_of_the_review: The paper introduces an intriguing framework that enables the application of RL into UCL. The main ideas are well explained. The main claims are correct (to the best of my knowledge). The claims are supported by empirical results. Since the reviewer's background is mainly about RL, the comments are mostly based on the actor-critic design in this work. Although some notations need to be revised, the general quality is satisfying. I am hoping the authors respond to my worries. All in all, I vote for a weak acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the problem of unsupervised continual learning (UCL) and proposes a new algorithm (named AUDR). The algorithm extends LUMP, an existing algorithm which mixes the views of the current samples with old samples retrieved from memory to prevent forgetting in continual learning. To extend LUMP which samples the mixing coefficient from a fixed distribution, AUDR adjusts the mixing coefficient during training: small values correspond to using memory samples (rehearsal) while large values correspond to learning new tasks. The authors propose a mechanism for automatically adjusting the coefficient using an actor-critic algorithm.

During the discussion, the following points were raised by the reviewers:
- the limited scope of the algorithm (tuning only two hyperparameters and a small number of UCL methods)
- the mixed results obtained by varying some hyperparameters (the number of actions and the number of epochs)
- the lack of a fundamental framework supporting the proposed algorithm.

The authors' responses partially resolved these concerns but a significant amount of uncertainty still remained. Due to the raised concerns, especially the lack of a fundamental framework supporting the proposed algorithm, I recommend rejection.
justification_for_why_not_higher_score: Due to the raised concerns, especially the lack of a fundamental framework supporting the proposed algorithm.
justification_for_why_not_lower_score: The experimental results look good.
Decision: Reject