Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a new problem called Learning with Evolving Class Ontology (LECO) where the objective is to learn finer classes over time. The authors have shown that labeling new data with fine-grained annotation is more valuable and proposed to use the techniques from semi-supervised learning and learning with partial labels literature to utilize the old coarsely-grained data. The authors have proposed a benchmarking protocol for LECO on top of two image classification datasets: CIFAR100 and iNaturalist and shown promising results.
strengths_and_weaknesses: Strengths:

- Tackles a somewhat new and interesting problem

- The proposed solution is simple and intuitive

- Experiments are extensive and systematic

Weaknesses:

- The technical novelty of the proposed solution is low. This is not a bad thing in general. However, in this particular case, since the proposed problem (LECO) is somewhat similar to the existing continual learning problems and the proposed solution is an ensemble of multiple existing ideas I find this to be a major weakness.

- I find the current problem setup and solution a bit restrictive. It overlooks a lot of practical concerns. (a) assumes a negligible cost for data acquisition, if new data is not available one can only relabel old data. (b) only expects finer labels in subsequent time periods. What happens if a new category is introduced which doesn't have a parent. The pseudo-labeling approach will mistakenly assign previous coarse-grained classes to this newly introduced category. Filtering the pseudo-labels based on class ontology might work in this case but if the data is long-tailed (the new class being a head class) this might remove a lot of old data from training.   

- The findings are not very surprising. If data acquisition cost is ignored and a large portion of data is labeled (which is the case in the current experimental setup) it is expected that a recent SOTA semi-supervised method will be able to reach a performance closed to supervised upper-bound accuracy trained with all data.

questions: - How were the levels selected for the iNaturalist-LECO experimental setup?

- The proposed method uses a large validation set. Is the final performance sensitive to the validation set size? Using such a large validation set to search hyperparameters is not very realistic. Were the upper-bound hyperparameters searched in a similar manner? 

- What happens if the fine-grained splitting of a corase-grained class is long-tailed. Most of the pseudo-labels for the coarse-grained old class will be assigned to the head fine-grained child class.  

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces the problem of Learning with Evolving Class Ontology (LECO), a general case of Class Incremental Learning, where instead of introducing new classes at each time step, we refine existing class labels into more granular ones. The authors ask whether, given incoming data and a fixed annotation budget, it is better to annotate the new data with the new labels or re-annotate the old data. In other words, whether it is better to train a classifier on a smaller but homogenous dataset or a larger one with a mix of old (coarse) and new (fine) labels. Based on experiments on CIFAR and iNaturalist, the authors conclude that by using semi-supervised learning and incorporating the hierarchical taxonomy, it is possible to achieve excellent classification accuracy with a heterogeneous dataset. In particular, by using pseudo-labelling and Learning-with-Partial-Labels, it is possible to get close to the performance of a classifier trained on a homogenous dataset of the same size.
strengths_and_weaknesses: The main strength of the paper is clarity. The structure is good, the writing is easy to understand, and the results are presented in a logical, incremental order. The authors precisely state the research questions and contributions and briefly summarise the conclusions in the introduction, which helps to guide the reader. The paper includes relevant details to reproduce the experiments.

I have no reservations about the quality of the submission. The experimental setup is sound, and the proposed models seem appropriate to tackle the presented problem. The authors evaluate their methods on two different datasets, modified to match the evolving ontology scenario. As mentioned below, the evaluation could be improved by including related work.

The biggest weakness of the submitted work is originality. While the LECO setting appears novel and could be an interesting type of class-incremental learning, limiting it to two time steps and giving the model access to all the data turns it into fine classification with coarse supervision, a problem tackled by other omitted work, such as:

- A Weakly Supervised Fine Label Classifier Enhanced by Coarse Supervision, Taherkhani et al. 2019
- Weakly Supervised Image Classification with Coarse and Fine Labels, Lei et al. 2017
- A Pseudo-Label Method for Coarse-to-Fine Multi-Label Learning with Limited Supervision, Hsieh et al. 2019
- From Categories to Subcategories: Large-scale Image Classification with Partial Class Label Refinement

The authors combine existing methods in a novel way, which could be a valuable contribution. However, the lack of comparison to previous approaches makes it hard to judge whether their work advances state of the art and therefore undermines the significance of their findings. My suggestion would be to either extend the work to multiple TPs or focus on the problem of fine classification with coarse supervision: remove the LECO formulation and compare their method with existing work. I think either would make the submission much stronger.
questions: In the "Technical Insights" section of the introduction, we read that "Under the LECO protocol of the iNaturalist benchmark (Section 3), (1) labelling new data outperforms relabeling the old: 73.3% vs 70.9% in accuracy;". I couldn't find these numbers in the result tables, so it is unclear how the label mismatch is handled in this base case. Is the strategy to learn a feature extractor on the old data and fine-tune it on the new data?

In the "Benchmarks" section of Chapter 3, the authors claim that "using all the labelled data is realistic because, for example, safety-critical systems (e.g., autonomous vehicles) must not trade off recognition accuracy with parsimonious memory buffer." While the buffer size is indeed arbitrary, for any practical continual learning system, it has to be *finite*. If we extend the LECO setting to multiple TPs, the buffer size will eventually become an important consideration.

One of the strategies in Chapter 4 is to "train an entire network from scratch with random weights." The phrasing is unclear here. Does it mean "train a randomly initialised neural network"?

Another strategy in Chapter 4 is to train a classifier on top of a randomly initialised feature extractor. What is the purpose of this? Is it to provide a lower bound on the classifier performance?

In Section 6.2., the first strategy is described as follows: "we filter out pseudo-labels that do not align with the ground-truth coarse labels." What does filter out mean in this context? Is the data point rejected? Is the coarse label used instead?
limitations: The authors correctly identified an essential caveat in their research question: obtaining new data is generally equally or more costly than annotating it. Re-annotation is also usually quicker than annotating from scratch because the annotator can use existing labels to constrain the task to a handful of classes.

One significant problem mentioned but not explored enough is the assumption that the findings would generalise to multiple TPs. In particular, it would be interesting to see the performance of Learning-with-Partial-Labels with multiple levels of granularity.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new continual learning problem setup, where class vocabulary becomes more fine grained in a continual fashion. Different than classic continual learning this setup allows access to the historical examples. Thus it is not prone to catastrophic forgetting.

The paper explores several research questions, like when the vocabulary evolves whether to annotate new data or relabel the old data (without collecting new data), how to leverage coarse label (of old data), and whether to finetune the model trained on old data, or train from scratch.

The paper show that a semi supervised approach, that only requires labelling the new data without relabeling the old data, is almost equivalent to relabeling all the data (both old and new).

The approach uses the new data to provide pseudo labels for the old data, and use the old data coarse labels to resolve reconcile conflicts between the pseudo fine labels and the true coarse labels.

strengths_and_weaknesses: 
### Strengths

* The paper is well written, very easy to read, and provides decent baselines and upper bounds.

* The problem setup is important from a practical standpoint, and novel to the best of my knowledge (I am not an expert in continual learning, or hierarchical learning).

* The suggested approach saves relabeling efforts, which can become quadratic with the number of "time periods".


### Weaknesses

* Although the problem setup is general, and allowing multiple "time periods" (TP), in practice the experiment are only with a single TP. I think this is the major weakness of the paper, because it is not clear how well the approach generalizes, as the labels become more fine grained.

* The approach was only evaluated on two datasets. It would be useful to provide experiments for a different data domain (not vision). 

* The problem setup ignores the fact that aside from storage, retraining on the old samples may be very costly from a compute standpoint.  E.g. retraining a model with a scale of Giga samples like a in autonomous driving may cost millions of dollars. There are recent works like [1,2,3] that try to alleviate this problem. I suggest to discuss it in the related work.

* This setup may introduce biases. There may be a distribution shift when collecting more fine grained labels. It would have been beneficial to add this type of bias to the benchmark, and demonstrate how sensitive is the approach to such a shift.

* It is unclear what are the open questions and the next steps. If the approach reaches the upper bound, does it mean that this problem is solved?

* Experiments: There is a better upper bound. Which is utilizing both the old and new labels for the old data 


### References
[1] Houlsby et al. Parameter-Efficient Transfer Learning for NLP, ICML 2019

[2] Li et al, Cross-domain Few-shot Learning with Task-specific Adapters, CVPR 2022

[3] Cohen et al, "This is my unicorn, Fluffy": Personalizing frozen vision-language representations, ECCV 2022

questions: 
1. It is not clear in what the new problem setup is different than LPL


2. The approach is somewhat similar to [4], which was applied to a different problem setup. I suggest citing it in the related work.

###References
[4] Goel et al, Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation, CVPR 2022
limitations: Ok,

See my feedback in the weakness section
review_title: null

Meta-review: metareview: The setting of evolving and refining classes over time is certainly a practical one in domains such as text classification. This paper offers some insights on questions like whether the entire data should be relabeled, or can one achieve near optimal performance by labeling only the new chunk. The paper concludes that joint training on old and new data, even if inconsistent, in conjunction with semi-supervised learning can be fairly effective. 
confidence: Certain
Decision: Accept