Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper:  In this paper, the other proposes a novel selection approach based on
Principal Component analysis and median sampling. This approach avoids the
pitfalls due to outliers and is both simple to implement and use across various
incremental machine learning models. It also has independent usage as a sampling
algorithm.

strength_and_weaknesses: Strength：
In general, the article has a complete structure and clear thinking.
The experimental setting is reasonable, experiments verified the proposed algorithms on Sports  datasets.

Weaknesses：
The description of the proposed method is not detailed enough, and the innovation point is not outstanding enough.
This paper mainly focuses on the effect of the strategy, and it is suggested to add more theories to support the strategy, so that the method can be more explainable.
The paper points out that the median is more suitable than the mean. It is suggested to carry out relevant experimental demonstration in the experimental part.

clarity,_quality,_novelty_and_reproducibility: In general, the paper has a complete structure, clear thinking and some innovation.
summary_of_the_review: Overall, the article has a complete structure and clear thinking. But the content needs to be further improved, including experiment and theory.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Training machine learning on streaming data sets has been an important problem with the widespread use of online systems. This often requires selecting specific examples to update your model. The paper proposes a PCA-based exemplar sampling algorithm. 

strength_and_weaknesses: The proposed technique includes a simple PCA-based dimension reduction and mapping the data along these dimensions and sample data with enough diversity. With this technique, outliers are not included in the training data. There has been a decent amount of experimentation and details provided.  I have seen some drawbacks in the paper, though.

1. How big of a problem is this? What is the real-life use case and significance of the problem, and the innovation aspect (scope/impact) is not clear? 
2. The proposed solution is simple, but I would be surprised if something along these lines has never been tried (I did not do a thorough literature survey, but it seems more like a simple use case of PCA). With this regard, I think the paper lacks novelty. 
3. There can be more depth and rigor in the experimentation. For example, PCA provides good strength in eliminating outliers. It sort of smooths out the training data—like a filter. However, it comes up with removing more edge/nuance cases in a data set as well. Usually, these samples are less in population and lost in the PCA analysis—often considered as noise. However, they are very valuable with respect to providing unique information for the training. It is not clear what the trade-off is here in this case and how we can address selecting such under-represented groups in this framework.  
clarity,_quality,_novelty_and_reproducibility: The Paper is written mostly well, with a bit of unnecessary verbosity in the introduction. The concepts explained are basic  It could have benefited from more visuals and be more compact.

summary_of_the_review: I don't think that the paper is strong enough to be at ICLR. From experimentation to justifying the importance of the problem, the novelty in the proposed solution, there are many gaps.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of continual learning with a novel exemplar selecting algorithm.
The authors claim that selecting the exemplars based on PCA-based direction well addresses the catastrophic forgetting problem of the datasets with high intra-class variance.
The experimental results on various benchmarks show the effectiveness of the proposed framework.
strength_and_weaknesses: - Strengths
    - The proposed framework demonstrates better performance than other baselines for the newly conducted benchmarks, Sports 73 and Sports 100.
    - The idea of sampling the most unaffected data along the principal axis seems novel and interesting.
    
- Weaknesses
    - The writing quality of the paper is far behind the bar of top-tier conferences like ICLR.
        - For instance, what is the point of the first paragraph of the introduction section? The paragraph consists of intro to CF, data distribution and class imbalance, the goal of this research, and etc. The paragraph should be split into multiple paragraphs to represent the crucial points for the authors' claim.
        - Formatting issues
            - Missing numbering for some subsections (e.g., Evaluation Metrics) and sections (e.g., Conclusion).
            - Inconsistency of the format for the references.
    - Datasets
        - Is there any quantitative evidence that sports datasets have higher variance than other common datasets, such as CIFAR-100 and ImageNet, which are widely used for continual learning benchmarks? For me, the claim is not that convincing since many sports-based video datasets, such as UCF-101, or Fine-gym, do not have large intra-class variance. While I’m not an expert on the Sports 73 or Sports 100 datasets, I’m not sure why such datasets have high variance and are expected to have many outliers. Actually, when I schemed the Sport 100 datasets on the Kaggle page, it seems the datasets have less variance than CIFAR or ImageNet, since the viewpoint of camera and poses of actors seem pretty similar to each data in the same class. The claim that “Sports images were taken indoors and outdoors in diverse lighting conditions” in the Section 5.1 also seems inappropriate since the augmenting contrast is one of the widely used augmentation strategies, and thus the diverse lighting will be considered when we train datasets like ImageNet with strong data augmentation.
    - Possible discussion in the Methods section
        - Selection of exemplars
            - As far as I understand, the sampled exemplars by $i$th principal direction and $j$th (where $i < j$) direction are not related to each other much. Then, the median points sampled by $j$th direction could be the outliers for $i$th direction, since two different principal axes are orthogonal. While it might be the extreme case, I think it might happen quite often when $m$ increases since the data embeddings used for sampling lie in the high-dimensional space. The claim of the paper could be improved if the authors provide any mathematical proof or intuitive explanation for this.
            - In the Abstract section, there is a sentence "herding, which is a way to select a random looking sample by a deterministic algorithm". However, the proposed PCA-based framework is also almost deterministic given the dataset except for the case that some directions share the same singular values. Moreover, what do the authors mean that herding selects a random looking sample, while it selects the samples closest to the mean class vector?
        - Generating balanced dataset using data augmentation
            - What is the reason for generating data only for $r_k^i = \|c_j^i\| - \|c_k^i\|$? It might be better to generate $\|c_j^i\|$ data for each class using KeepAugment and replace $\|c_k^i\|$, rather than just generating $\|r_k^i\|$ to boost the regularization effect.
    - Experimental results are not sufficient to support the effectiveness of the proposed framework.
        - The combination of other sampling algorithms and the diverse data augmentation strategy must be studied more deeply.
            - According to Table 2, it seems KeepAugment highly affects the performance. Then, it is worth checking the performance of other baselines (RM and GDumb) equipped with KeepAugment. Moreover, the good results on TinyImageNet in Figure 5 in Appendix B also might be due to the effect of KeepAugment, since the difference is marginal. Therefore, the effectiveness of the proposed algorithm for the balanced datasets is not convincing. It might be better to show the results on CIFAR-100 datasets, which is a widely used benchmark for class incremental learning. Training details for other baselines on Sports 73 and Sports 100 datasets are also missing.
            - What if the model employs different data augmentation strategies (e.g., CutMix, etc., ), rather than KeepAugment?
        - It is unclear whether improvement in performance is driven by class-balancing or augmented data.
            - What is the reason for generating data only for $r_k^i = \|c_j^i\| - \|c_k^i\|$? It might be better to generate $\|c_j^i\|$ data for each class using KeepAugment and replace $\|c_k^i\|$, rather than just generating $\|r_k^i\|$ to boost the regularization effect.
    - Minor
        - Need consistency for the term “Principal Component **a**nalysis” and “Principal Component **A**nalysis”
        - It seems the columns of Table 2 should be exchanged with each other, according to Table 1.
clarity,_quality,_novelty_and_reproducibility: - Clarity : The paper has some ambiguous parts for experimental details, as mentioned in the weakness section.
- Quality : The presentation of the paper is not that neat.
- Novelty : The idea of using PCA to sample exemplars seems novel.
- Reproducibility : While the authors do not provide their source code, the implementation details for the main framework seems sufficient. However, the implementation details for the baseline algorithms should be added, since the authors conducted the experiments for Sports 73 and Sports 100 for the first time.
summary_of_the_review: Despite the paper studies some novel concepts with respect to PCA-based exemplar sampling, I hardly agree that the current version of the submission meets the bar of ICLR, since there are some missing experiments as stated in the weakness section and the writing quality should be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a PCA-based exemplar sampling method in continual/incremental learning. All three knowledgeable reviewers agreed that this paper is not yet ready for publication due to several shortcomings. In particular, many reviewers pointed out that writing and presentation should be improved, and the experimental results are not convincing. The verification using only sports-based video data without using standard benchmark datasets such as CIFAR or ImageNet also raised doubts about the superiority of the proposed method. In addition, some additional concerns on the design choice of the method were also raised by one reviewer, but unfortunately, the authors did not respond, so the proposed method was not sufficiently justified despite some novelty. This AC would like to encourage the authors to resolve all the issues raised this time and submit it to other venue.
justification_for_why_not_higher_score: writing/experiments should be improved.
justification_for_why_not_lower_score: n/a
Decision: Reject