Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposes a method to correct a recency bias in replay-based task-free continual learning, but separately optimising the final connected layer  of network from the rest of the network. Focuses on continual learning in vision with evaluations in the area.
strength_and_weaknesses: Strengths
- A very simple but highly effective method for bias correction in task-free continual learning
- Ablates the effect of various components (e.g. utility of surrogate classifier).
- Well written paper with decent set of evaluations in the vision space.
- Good comparison to baselines and shows particularly good performance with respect to the bias metric proposed, and another metric proposed by Cai et al (2021).

Weaknesses
- Lack of understanding into why various components work despite ablations. This makes it difficult to find as much value in the paper, given the focus on toy datasets. Concretely, in section 4.5 it details why a surrogate classifier is necessary. However, there is no exposition on why surrogate classifier is necessary. There is a reference to gradient flow and figure 1 but nothing beyond that.

Minor Comments & Questions
- Provide a name for the bias metric, and then use that name in the tables
- Highlight performance of OBC in tables.
clarity,_quality,_novelty_and_reproducibility: Clear and should be easily easy to reproduce given the simplicity.
summary_of_the_review: A simple method to mitigate the effect of the recency bias in task-free continual learning. Well written paper. Accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on online bias correction during task-free continual learning. The paper first shows, both theoretically and empirically, why simple experience replay biases on the results of the recent stream observation. Second, the paper introduces the metric to quantify prediction biases. Using the observation from this, the paper proposes a simple approach called Online Bias Correction which appropriately modifies the final layer of the network to correct for the biases online. Finally, the paper concludes with extensive experimentation of the OBC showing significant improvement on a number of task-free continual learning methods on multiple datasets.
strength_and_weaknesses: Strengths:
1. Online task-free continual learning has recently drawn more attention because it focuses on solving practical challenges. The direction of this paper is promising as it focuses on correcting the bias incurred during training, addressing a major problem in this area.
2. The claim of correcting the prediction bias by changing the way the classifier is trained is well supported by extensive experimentation of various methods on a good amount of datasets.
3. The proposed method, Online Bias Correction, is simple but very effective. Also, it acts as a wrapper around other task-free continual learning methods. The main advantage of this is that it can be applied to existing methods without any major change in existing architecture and training procedure.
4. The paper also includes the ablation study on surrogate classifier which was introduced by the proposed method. Its effectiveness is well supported by extensive experimentation.

Weaknesses:
1. Even though the paper has used a variety of datasets, many recent papers in this area also use CORe50 [1]. I believe that it would strengthen the experiments and would make it easier to compare against some other work.
2. In section 3.2, the explanation on data sampling bias is unclear. First, they state that past and future observations have contributed equally. Then it is changed to not contributing equally, making it confusing. I believe that a clear mathematical definition would likely remove the confusion.
3. Main concern of the method is that it is a wrapper over the existing methods which makes it dependent on them and also might not guarantee effectiveness on the methods outside the ones used in the experiments.
4. Table 1 includes a column called “Bias”. Is this the result of the Jensen-Shannon divergence? 

[1] Lomonaco, Vincenzo, and Davide Maltoni. "Core50: a new dataset and benchmark for continuous object recognition." Conference on Robot Learning. PMLR, 2017.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and follow. The ideas in the paper are clear and presented in a coherent manner. The contribution seems to be novel. The authors have pledged to make the source code public upon acceptance for reproducibility.
summary_of_the_review: The paper focuses on a problem that is underexplored. The ideas presented are novel. The contribution is significant in the area of Bias correction, especially in online settings of task-free continuous learning. Even though the proposed method is a  wrapper around the existing methods, I think it is a significant and valuable contribution in this area as it is also evident from the rich set of results included.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of recency bias in the task-free continual learning setup. It proposes a new metric to quantify prediction bias and a method to mitigate it through adjusting parameters of the final model layer post training. The authors then propose a online procedure for training and unbiasing the model over the data stream via an experience replay mechanism. They test the performance of the algorithm over multiple datasets comparing favourable to several baselines.
strength_and_weaknesses: Strength:
The paper looks into the very important problem of continual learning. It is very readable and easy to follow.

Weaknesses:
The paper lacks on clarity and quality at several major points. These are concretely listed in the next section. It is also of limited novelty - also clarified in the following section.
clarity,_quality,_novelty_and_reproducibility: There are many points, where the paper is not sufficiently clear.
- It is not clear, how the task-free setup is instantiated in the theoretical discussion and in the experiments. It argues that in the task-free setup, no strong assumptions are made about the non-stationarity over the stream (neither task nor class incremental). What is the source of the unseen data here? Some future data coming after the last observation in the current stream? Or are these some examples generated in parallel with the whole stream (and therefore tracing the entire history of the distribution shifts)? Are the classes assumed to be stable or can they possibly change as well? Can there be more/less classes, Can these change in nature? Similarly, how is the experimental data organized for the continual setup with distribution shifts? Are these organized into binary or 10-wise tasks (as common in the task and or class incremental setup) without indicating this to the model? Or is the data randomly mixed? Where, how is the distribution shift happening? How is the test set constructed?
- In section 2.2 you claim that the larger the stream, the larger the memory will be. Why "will" it be? Or do you mean to say it "shall" be larger? You further claim to investigate algorithms with computation cost O(n) independent of m. Isn't the memory itself a function of n? Why would the cost be ever O(mn)?
- In the experimental section, accuracy, bias and information retention are reported. Is accuracy and bias calculated over test set? How is the test set constructed with respect to the shifting distribution in the stream? Is the av IR equivalent to accuracy over the train set (= the whole stream)? If not, how is it different?

I believe the ideas are also not particularly novel
- the hierarchical view of NNs as a feature extractor with a follow up classifier has been introduced previously for example in the works of Naftali Thisby and his group.
- So was the need to weight unequally the current and replay observations to compensate for the oversampling of current distribution (e.g. Ramapuram et al., Lifelong generative modeling, 2020)

There are some further clarity or quality issues
- in section 3.3. the prediction bias quantification is outlined. The prior class probabilities are defined as expectations over the one-hot class encodings. The expectation is taken with respect to what distribution. The notation suggests that the output p is again a class-long vector, is that so? The same goes for q as expectation of the classification outputs. What is the distribution used in this expectation?
- JS is suggested as symmetric divergence. Why the need for JS? What would be wrong with KL and using the prior for the weighting?
- section 4.2 you claim "we can essentially compare the quality of the representations that their feature extractor learns". How can you compare these?
- section 4.4. "the computational graph in the left part of Figure 1 flows from the feature extractor to the classifier, but the classifier is not updated with the resulting gradients". What do you mean? That in this setup you do not train the classifier? Or did you mean to say the feature extractor? 

Reproducibility is limited:
- Authors do not indicate, if the code is anywhere available
- How are the expectations in 3.3 calculated?

More general question - you seem to be training the classifier and the surrogate classifier in a completely disconnected manner (no sharing of parameters or parameter passing, etc.). Isn't this eventually going to lead into a complete divergence between the two classifier models with which the feature extractor is unlikely to be able to cope?

summary_of_the_review: The topic of the paper - continual learning - is an important one and novel methods in this area would be of great interest to the community. However, I find the current paper lacking significantly in clarity, quality and novelty to be recommendable for publication at ICLR. I therefore recommend rejection from the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for correcting bias of continual learning method using experience replay. The proposed method uses a memory from past examples to train the last layer of a neural network trained by an experience-replay-based method. This prevents the model from being biased to recently observed samples. By fixing layers other than the last, it also prevents overfitting to the chosen samples from the memory. The authors show that the proposed method indeed reduces the bias and improves the overall performance.
strength_and_weaknesses: # Strengths
- The paper points out an important weakness of experience replay that seems relevant to the research of continual learning.
- The proposed method is simple and widely applicable without big implementation or computational overhead.
- The experiments consistently show its effectiveness.

# Weaknesses
- The problem setup on the mathematical level is not clear to me. The paper says that it does not assume strong assumptions, but without any assumptions on how the distribution can change, I don't think we can say much about the learnability. One possible direction may be considering some form of cumulative regret as in online learning, but the paper does not seem to be considering such a setup either.
- The setups of the experiments are also not clear to me. How the samples are drawn in the stream from the dataset (e.g., gradually including new classes) seems important, but I could not find any part mentioning that.
- I could not understand how the proposed method makes the right amount of bias correction. If the effect is too strong, it can be biased in the opposite way, to the past experience side.
- The measure of the "bias" is misleading because it does not quantify the bias in the sense of the standard statistics. (The expectation is not taken over all possible draws of the training samples.) It is also a quite weak measure because it averages over the samples.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, but the technical part can be improved. At least, I would like to see the problem formally defined especially around how the data are generated and how the learned model is evaluated. As I mentioned, the setups of the experiments are not clear to me, and it may be difficult to reproduce them.
summary_of_the_review: Although the work is well-motivated, the paper is easy to read, and the experiments show the effectiveness of the proposed method, some important details are not clear, and the theoretical part might have a weakness. Also, there is no evidence that the proposed method really corrects the bias theoretically or even empirically because the measure of the bias does not seem to capture the bias that the authors are trying to mitigate. I vote for weak reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method to correct recency bias in replay-based task-free continual learning by optimizing the final connected layer of network from the rest of the network and demonstrates its effectiveness on a number of vision benchmark tasks.

The reviewers agreed that the paper points out an important problem in experience replay relevant to continual learning. The reviewers appreciated the simplicity of the method and agreed that the empirical results are strong. 

The reviewers thought the paper could benefit greatly from more motivation, intuition, and conceptual justification for the design choices of the proposed method, and further clarification of various definitions such as what "bias" means in this context. 
justification_for_why_not_higher_score: Lack of conceptual justification or intuitive motivation for why this method should work.
justification_for_why_not_lower_score: Paper identifies and tackles an important problem in continual learning w/ experience replay and proposes an effective method.
Decision: Accept: poster