Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work targets the problem of Continual Learning (CL) and In particular, the problem of populating the buffer memory in the Experience Replay (ER)-based paradigm. In ER, we are allowed to store a small subset of the incoming data stream, therefore deciding on a good policy to populate the memory buffer is important. The paper proposes MEMORABLE INFORMATION CRITERION (MIC) which is a combination of the surprisal and learnability of an incoming example and further motivates this with a  bayesian analysis. Using MIC, it introduces two algorithms, namely, InfoRS and infoGS for buffer update. The paper includes experiments on standard CL datasets with a focus on General Continual Learning (GCL) guideline, which focuses on not having dependence on task boundaries. Finally, it reports the proposed method's performance on imbalanced data streams and shows that infoRS outperformed reservoir sampling, GSS, and some other baselines. 

main_review: Strengths:

1. I think that the paper was easy to follow overall, and the MIC criterion is intuitive. The tSNE plots (of the selected example for the memory of size 200) showing the ablation study was also quite interesting. 
2. The proposed algorithm outperformed RS and other mentioned baselines, without too much computational overhead over RS. 
3. Authors do study over other possible metrics than MIC, including Entropy Reduction, therefore empirically justifying the criterion. 

Weakness: This subsection is a collection of doubts, suggestions about some more experiments (including other baselines), and how this submission can be improved overall. 

Baselines and Performance metrics:
1.  I suggest including the paper [1] and online k-medoid clustering. [1] “Coresets via Bilevel Optimization for Continual Learning and Streaming” NeurIPS’20. 
2. In my opinion, it is also important to report the backward transfer or forgetting metric, similar to the DER paper.
3. While the manuscript contains final testing results for different imbalance levels, I think for CL papers a plot of accuracy after each task provides more information ( For example, see Fig 3 in the A-GEM paper -  EFFICIENT LIFELONG LEARNING WITH A-GEM).

Technical Questions:

1. How sensitive is the performance with the thresholding hyperparameter?
2. Fig (5) right, similar to infoRS, I think for RS one also needs the confidence bars. 
3. In the section about robustness to imbalance, does a different seed change the task ID in which we add more epochs? Moreover (more of a sanity check), but do authors keep the task-iD same across different “R” values?
4. Algorithm (2): What is the need to compute A_M^-1 and b_M when the memory is not full? Cannot one just do that once we fill in the memory, and use features based on the new parameters?  
5. Algorithm (2): Rather a technical question - why are we only choosing one example from each batch? There can be a case that we have two examples with similar (but not the same) and high MIC values and are learnable (two different classes, right when we begin a new task). 
6. Regarding GSS: Are authors using the GSS-IQP, or the GSS-Greedy? Moreover, one easy trick one can do to accelerate GSS is to consider gradient with respect to the last layer only, the way it is done in the paper [1] (See [2] as well). [1] Coresets for Data-efficient Training of Machine Learning Models, ICML’20. [2] Not all samples are created equal: Deep learning with importance sampling, ICML’18. 
Regarding GSS: Rather a sanity check, but do authors use DER++ loss in this case for the fair comparison?
7. In Section E appendix (Proof), equation (19), where the KL is expanded, in my opinion, the numerator seems notationally odd. That is, why is x_{\ast} mentioned as a conditioned variable and as a parameter at the same time. Wouldn’t it suffice to just use it as a conditioned variable? 

Technical suggestions:
1. Fig (2), please show how does the memory look like in the tSNE plot for the cases with small sizes such as 10/25 for RS and infoRS (w/ and w/o learnability)  
2. What is mentioned here as DER is actually DER++ in the original paper.
3. In the related works section, which talks about dataset distillation, there is another paper [1] (ICLR’21) that also talks about dataset distillation with applications in continual learning. [1] Dataset Condensation with Gradient Matching.

Writing Suggestions:

1. In equation (1), I think it will be good to not use \theta for both h and g. 
2. Equation (3), during the overall update step, we update memory as well as the network parameters. Therefore, I suggest using notations accordingly.
3. In the algorithm block, I would recommend adding line numbers, which improve readability and cross-referencing in a discussion. 
4. Section 2.1, last paragraph: Add spacing after the period right before the “Furthermore”.

summary_of_the_review: It is an interesting work, however, lacks discussion about some baselines (not mentioned in the paper) and other performance metrics (about forgetting, BWT). There are some questions (mentioned in the detailed review) that need to be answered in order to make it a strong submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers task-free continual learning in the context of online memory selection. The author consider the use of information-theoretic principles and propose the surprise and learnability criteria to select informative points and avoid outliers. The authors also propose InfoRS to sample among selective points with high information. The methods are validated on continual learning benchmark datasets, and show efficiency and improvements over existing approaches. 
main_review: Strengths:

+ The proposed methods seem fairly natural. In particular, the notion of surprise and learnability seem well-suited to the current problem at hand. The computation of the Bayesian posteriors is also natural and the Gaussian assumption, together with conjugacy, results in efficient updates, which is nice. 

+ The InfoRS algorithm also seems natural for sampling. 

Weaknesses:

- It would be preferable for the authors to consider the effects of each of their proposed modules separately -- surprise, learnability as well as InfoRS. Currently from the experimental section, it is unclear which module contributes to which aspect of the improvement. An ablation study would be useful. 

- I am wondering whether the authors have examined the effect of the many hyperparameters in their study such as \eta, \alpha and \beta. 

- The datasets used seem rather simplistic, just MNIST and CIFAR10. 
summary_of_the_review: Generally, the paper has some novel contributions but also some minor weaknesses. If the authors can address the weaknesses/queries I raised in the main review, I'm happy to increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers an online memory selection task for continual learning applications. The paper proposes a new (information-theoretic) criterion to pick informative points and void outliers. This criterion is a combination of the surprise and learnability score. In addition, the paper presents a stochastic information-theoretic reservoir sampler (InfoRS) to sample among selective points with high information.

The paper empirically demonstrates the proposed criteria encourage to select representative memories for learning the underlying function. They consider continual learning benchmark as well as ablation studies of data imbalance. 

main_review: Objective of the work:
* The paper extends the existing research direction in general continual learning. The key contribution is proposing the criteria to select and replace the existing points in the memory buffer.

Strong points:
* A new method for online memory selection. The surprise captures how unexpected a new point is given the memory, and allows us to include new information in the memory. The learnability captures how much of this new information can be absorbed without interference, allowing us to avoid outliers.
* The proposed Memorable Information Criterion (MIC) is a weighted combination of the proposed learnability and surprise scores. The authors also connect that MIC is related to Information Gain. In the appendix, the paper has compared these criteria (MIC, IG, ER) thoroughly.

Weak points:
* The surprise score is related to the uncertainty score in literature which has been widely used to select unseen points. While the learnability makes sense to distinguish the unfamiliar and outlier points. Having said that, the reviewer thinks that the learnability score can make the algorithm too preservative to learn new information.

* The performance may critically depend on balancing how much we want to “surprise” versus “learn” through $\eta$. It is highly recommended to have the ablation study with respect to $\eta$.

* Bayesian linear regression requires matrix inversion of a matrix $d \times d$ where $d$ is the feature dimension. The Sherman-Morrison Formula, Eq. (11), can be used to efficiently update the inverse matrix. However, while the reviewer acknowledges the trick works well in practice, the reviewer doesn't see this is a great technical contribution of the paper.


Presentation: 
* The paper is in general well written and easy to follow.

Reproducibility:
* The results appear to be reproducible though I would encourage the authors to release their code to enable better validation of their claims.


Minor points:
* The author has demonstrated the effectiveness of the proposed criteria using a greedy algorithm to replace the existing points with the new points if certain condition is met. It could be interesting to consider either replacing (as the proposed framework) versus expanding without replacing. The reviewer thinks that we may need to enlarge the memory buffer overtime to learn more tasks. However, this would be out of the scope of the paper.

* In Figure 2, it is not clear to compare the middle figure and right figure to see the effect of learnability.

* The paper presents a scalable Bayesian model that can compute surprise and learnability with a small computational footprint by exploiting rank-one matrix structures. Instead of using a deep network and Bayesian linear model, I am wondering if it is better to directly train a Bayesian neural network for estimating the uncertainty?

* a footnote must be placed at the end of a clause or sentence, such as adding the number after the comma or full stop.


summary_of_the_review: The key contribution is proposing the criteria to select and replace the existing points in the memory buffer. However, the technical novelty is limited. The paper includes several techniques which have been widely used elsewhere. The surprise (uncertainty estimation) has been widely used in literature. The Bayesian linear regression and the rank-one update trick are also well studied.

Empirically, the performance may critically depend on balancing how much we want to “surprise” vs “learn”. This should be well studied in the experiments. Perhaps, the paper can propose a technique to adaptively learn $\eta$ over time.

The reviewer thinks the paper is currently at the borderline.
review_title: null

Meta-review: One way of avoiding catastrophic forgetting in continual learning is through keeping a memory buffer for experience replay.  This paper addresses the problem of online selection of representative samples for populating such memory buffer for experience replay.  The paper proposes novel information-theoretic criteria that selects samples that captures surprise (samples that are most informative) and learnability (to avoid outliers).  They utilize a Bayesian formulation to quantify informativeness. They provide two algorithms: a greedy approach, and an approach that takes timing (when to) update memory into account based on reservoir sampling to mitigate possible issues with class imbalance.

Pros:  The paper is well written and organized.  It was easy to follow. The formulation is novel and technically sound. The idea of taking learnability into account is novel and interesting.  It provides a nice way of avoiding outliers and balancing surprising information. The authors presented the motivation for each part of the framework well.  

Cons: To understand the contribution of each component of the formulation and competing criteria, an ablation study is needed.
Reviewers had several detailed suggestions and questions, including sensitivity to hyperparameters, additional citations, additional data sets beyond MNIST and CIFAR10, etc.  

In the rebuttal, the authors have addressed several of these concerns.  Please make sure to include and incorporate reviewer suggestions in the final revised version.
Decision: Accept (Poster)