Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problems of classification and object detection from natural image datasets in a continual learning setting, whereby the different classes/objects to be learned are not observed together but sequentially. It is assumed that a memory buffer of a certain size is available to store data in. The strategy used by the paper is to fill this memory buffer with compressed data samples (with JPEG) rather than the original images. Such compression introduces a quantity-quality trade-off, which this paper empirically analyses. Additionally, the paper proposes an automated way to select the amount of data compression based on determinantal point processes.
main_review: Strengths:
-	The empirical comparisons appear to be performed to good standards.
-	The results are convincing, even if not very surprising.
-	Practical benefit of the proposed memory replay with compression approach is demonstrated on a large scale object detection task.

Weaknesses / suggestions:
-	The claim that memory replay with compression is neglected is somewhat too strong. In particular, I think there are two existing lines of work that would be good to discuss:
* Firstly, there is highly related previous work addressing the problem of online continual compression: http://proceedings.mlr.press/v119/caccia20a.html
* Secondly, one motivation for generative replay (rather than replaying stored samples) is that learning a generative model can result in a compressed representation of the original data (even if in practice this is not always the case).
-	Section 3 is not completely clear and somewhat confusing. Based on the description in this section, I would think that this paper deals with domain-incremental learning (https://arxiv.org/abs/1904.07734) or the new instances setting (http://proceedings.mlr.press/v78/lomonaco17a.html), but from the rest of the paper I understand it deals with class-incremental learning / the new classes setting.
-	The paper generously uses the term state-of-the-art (SOTA). In almost all cases, I don’t think the use is fully justified. I also don’t think the use of this term is relevant or necessary for this paper, and I would recommend removing most if not all mentions of SOTA in the paper.

summary_of_the_review: I think this is a solid paper. It is generally clear, well-written and insightful, even if its contributions are not surprising or technically challenging. I support acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose memory replay with data compression, which is both an important yet neglected baseline and a promising direction for continual learning. Using a naive technique of data compression with a properly selected quality, the proposed method can achieve the SOTA performance in a time-efficient and plug-and-play way. Since the compression quality is highly nontrivial for the efficacy of memory replay, the authors provide a novel method based on determinantal point processes (DPPs) to determine it efficiently, and validate their method in both class-incremental learning and semi-supervised continual learning of object detection.
main_review: Strengths :

1. This proposed method demonstrates its advantages in realistic continual learning scene.
2. The experiment of this paper is sufficient.
3. The method is easy to follow, and the performance is improved to a certain extent.
4.The paper is well written, and the supplementary material is abundant.
Weaknesses :
1. The novelty of this paper is incremental. 
2. Only the average accuracy is reported. There is a lack of some other common used metrics like average forgetting.

summary_of_the_review:  The novelty of this paper is incremental but the experiment of this paper is sufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the problem of catastrophic forgetting for continual learning is explored. The core idea is to benefit from data compression to reduce the space to store data samples and then replay the reconstructed data points that are built using the compressed versions. JPEC method has been explored empirically to determine an optimal compression rate through solving an optimization problem. Experiments on three benchmark datasets are performed to demonstrate that the method is effective.
main_review: Strengths 
1. The problem of catastrophic forgetting has gained considerable attention.

2. The paper is written well and easy to follow.

3. Experiments support the claims.

Weaknesses:

1. The idea is not novel.

2. Unnecessary complexities in the implementation of the idea.

3. Analytical and ablative experiments are limited.
summary_of_the_review: 1. The proposed idea is not novel. It is somewhat trivial that whenever we have storage limitations, data compression is going to be helpful. This on its own is not an idea at the level of a top-tier venue.


2. I find subsection 4 to be an unnecessary section to make the idea look non-trivial but I think it only is complicating the method unnecessarily.  A naive grid search can do the job. The justification that the authors have provided is that grid search is of "huge computational cost". But first of all, it is not clear how large this computational cost is. Second, even if it is large, it is not going to be a huge burden for continual learning. Because you can determine q one time before starting model execution. This is what we do in most cases when we want to tune a hyperparameter. There is no theoretical contribution in that section either and along with the appendices, only known results are rewritten. As a result, the idea contribution of this manuscript is highly limited.


3. In experiments, it is mentioned that 20 samples per class are stored. However, this is going to lead to a memory buffer that is going to grow as more classes are used. The appropriate way to do this is to consider a memory buffer with a fixed size and then discard samples as new tasks are learned to replace a portion of old samples with new samples.

4. In the results, standard deviation should be added to make the comparison more informative.

5. It is not very clear how much advantageous data compression would be for continual learning. An additional analytic experiment can be to show that how much memory can be saved by storing less compressed samples while getting no considerable performance degradation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a data compression method for memory-replay based continual learning algorithms. With the data compression method,  more old training samples can be stored in the memory to better capture old data distribution. However, there is a trade-off between the quality and quantity of compressed data, the authors propose to use determinantal point processes to determine the quality of the data compression.  Extensive experiments show that with the proposed method, a naive compression method can achieve the SOTA on several continual learning benchmarks. 
main_review: Strengths:

1. The paper is well motivated. The idea is simple but makes sense in the context of memory based continual learning. 

2. The experiment to show the tradeoff between the quality and quantity of compressed data is also interesting. It might be expected but the authors conduct a detailed analysis.

3. The experiments are also extensive which show the benefits of the proposed approach.

Weaknesses:

1. The motivation of the proposed method is not that clear. Why maximize the conditional likelihood is a good way to select q? 

2. In continual learning, we would like the samples in the memory to represent the old data distribution,  why the proposed objective based on DPP can achieve this?


More questions:


1. Also, Figure 3 is a little confusing, which one is dark dot and which is light dot?

2. Why  keeping the feature volume is a good criteria for selecting q?

3. Is it possible to store examples with mixed quality?
summary_of_the_review: In this paper, the authors propose to compress the data for memory-based continual learning. Since there is a tradeoff between quantity and quality of the compressed data, the authors propose a novel method to decide the quality of the data. The idea is well-motivated and novel for continual learning.
review_title: null

Meta-review: This works considers limitations of rehearsal-based methods in the context of continual learning (classification and object detection). Rehearsal-based methods provide a strong baseline, but a loss in predictive performance arises when the memory is limited in size. The authors propose to leverage compression (JPEG) to increase the number of data (images) stored in the memory. The approach is evaluated in the context of an autonomous driving application.

The additional experiments conducted by the authors were highly appreciated and helped clarify open questions (e.g., class-incremental learning set-up, DPP objective to determine size of the memory, quantity vs quality of compressed data, etc.). The authors addressed the issues raised by three out of four reviewers, who did not have further comments. The remaining reviewer found that the methodological contributions of this paper, namely of using compression in the context of CL, was pretty straightforward. However, the authors addressed the concerns raised by the reviewer regarding the selection of the compression quality q as far as I am concerned and conducted additional experiments to further demonstrate the usefulness of the approach. I would encourage the authors to include this discussion in the final version of the paper. I would also encourage them to include the additional experiments they conducted with fixed memory size and amount of memory that can be saved.
Decision: Accept (Poster)