Continual Learning with Evolving Class Ontologies
Authors: Zhiqiu Lin
Deepak Pathak
Yu-Xiong Wang
Deva Ramanan
Shu Kong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
Machine Learning
Lifelong Learning
Visual Recognition
Continual Learning
Semi-Supervised Learning

Abstract: Lifelong learners must recognize concept vocabularies that evolve over time. A common yet underexplored scenario is learning with class labels that continually refine/expand old classes. For example, humans learn to recognize ${\tt dog}$ before dog breeds. In practical settings, dataset ${\it versioning}$ often introduces refinement to ontologies, such as autonomous vehicle benchmarks that refine a previous ${\tt vehicle}$ class into ${\tt school-bus}$ as autonomous operations expand to new cities. This paper formalizes a protocol for studying the problem of ${\it Learning with Evolving Class Ontology}$ (LECO). LECO requires learning classifiers in distinct time periods (TPs); each TP introduces a new ontology of "fine" labels that refines old ontologies of  "coarse" labels (e.g., dog breeds that refine the previous ${\tt dog}$). LECO explores such questions as whether to annotate new data or relabel the old, how to exploit coarse labels, and whether to finetune the previous TP's model or train from scratch. To answer these questions, we leverage insights from related problems such as  class-incremental learning. We validate them under the LECO protocol through the lens of image classification (on CIFAR and iNaturalist) and semantic segmentation (on Mapillary). Extensive experiments lead to some surprising conclusions; while the current status quo in the field is to relabel existing datasets with new class ontologies (such as COCO-to-LVIS or Mapillary1.2-to-2.0), LECO demonstrates that a far better strategy is to annotate ${\it new}$ data with the new ontology. However, this produces an aggregate dataset with inconsistent old-vs-new labels, complicating learning. To address this challenge, we adopt methods from semi-supervised and partial-label learning. We demonstrate that such strategies can surprisingly be made near-optimal, in the sense of approaching an "oracle" that learns on the aggregate dataset exhaustively labeled with the newest ontology. 

Continual Learning with Evolving Class Ontologies
Zhiqiu Lin' 'CMU
Deepak Pathak'
Yu-Xiong Wang?
Deva Ramanan 1.3 Shu Kong"*
UIUC
3 Argo AI
*Texas A&M University
Open-source code in webpage
Abstract
Lifelong learners must recognize concept vocabularies that evolve over time. A common yet underexplored scenario is learning with class labels that continually refine/expand old classes. For example, humans learn to recognize dog before dog breeds. In practical settings, dataset versioning often introduces refinement to ontologies, such as autonomous vehicle benchmarks that refine previous vehicle class into school-bus as autonomous operations expand to new cities. This paper formalizes protocol for studying the problem of Learning with Evolving Class Ontology (LECO). LECO requires learning classifiers distinct time periods (TPs); each TP introduces new ontology of "fine" labels that refines old ontologies of "coarse" labels (e.g., dog breeds that refine the previous dog). LECO explores such questions as whether to annotate new data or relabel the old, how to exploit coarse labels, and whether to finetune the previous TP's model train from scratch. To answer these questions, we leverage insights from related problems such as class mnnsemnncremntall learning. We validate them under the LECO protocol through the lens of image classification (on CIFAR and iNaturalist) and semantic segmentation (on Mapillary). Extensive experiments lead to some surprising conclusions; while the current status quo in the field is relabel existing datasets with new class ontologies (such as COCO-to-LYY
or Mapillary 1.2-to-2.0), LECO demonstrates that far better strategy is annotate new data with the new ontology. However, this produces an aggregate dataset with inconsistent old-vs-new labels, complicating learning. To address this challenge, we adopt methods from semi-supervised and partial-label learning. We demonstrate that such strategies can surprisingly be made near-optimal, in the sense of approaching an "oracle" that learns on the aggregate dataset exhaustively labeled with the newest ontology.
Introduction
Humans, as lifelong learners, learn to recognize the ontology of concepts which is being refined and expanded over time. For example, learn recognize dog and then dog breeds (refining the class dog). The class ontology often evolves from coarse to fine concepts in the real open world and can sometimes introduce new classes. We call this Learning with Evolving Class Ontology (LECO).
Motivation. LECO common when building machine-learning systems practice. One often trains and maintains machine-leameed models with class labels (or class ontology) that refined over time periods (TPs) (Fig. [ Such ontology evolution can be caused by new requirements in applications, such autonomous vehicles that expand operations to new cities. This is well demonstrated in many contemporary large-scale datasets that release updated versions by refining and expanding classes,
such Mapillary [52 55 Argoverse [2 78., KITTI [4] 23 and iNaturalist (33.3] 69 75.. For
example, Mapillary V1.2 55 defined the road class, and Mapillary V2.0 52 refined it with fine
Authors share senior authorship.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Perizd (TP9
(TP)
Parioc (TP9)
Time Period 1 (TP)
desorony 9ilowally
oló ontology coarse lahils
nety dntalloy fie labels
PG3 image
aidemaik
rainernis Cacullacue
Urio toevia
sopus Asac
srattiossi--parkag teaftie-sigg Front
Lcbchysetes Audociaii
RGP image
tanatolia
Taturazia Gattate
Biblecta Doccinace
Mapilary-LRCO Benahmaric world datased soenario V12TTP VEAITP.
INIPLECOO Benchmark: Long
ovatves from course fine.
Figure 1: Learning with Evolving Class Ontology (LECO) requires training models in time periods (TPs)
with new ontologics that refine/expand the previous ones. Left: The Mapillary dataset, which was constructed
study semantic segmentation for autonomous vehieles, has updated versions from V1.2 (55 V2.0 52), which refined previous labels with fine- ine-grained ones on the same data, c.g., V1.2's s si dewalk 1k is split to V2.0's s sidewakk k and Briveway. The visual demonstration with semantic segmentation ground-truth blacks out unrelated classes for clarity. We explore LECO using this dataset. Right: To study LECO, we also repurpose simulate the coarse-to-fine ontology evolution through the lens of image classification. LECO asks the first basic question: for the new that defines new ontology of classes, should one relabel the old data or label new data? Interestingly, both labeling protocols have been used in the community for large-scale datasets. Our study provides the (perhaps obvious) answer one should always annotate new data with the new ontology rather than re-annotating the old. One reason is that the former produces more labeled data. But this produces an aggregate dataset with inconsistent old-vs-new labels. We make of insights from sepi-supervised learning and learning-win with--iitttt saat--artial--aae annotations, approaching the upper bound of learning from an oracle aggregate dataset with all new labels.
the large-scale iNaturalist dataset 37 69 [5
to learn from such heterogenous
labels including parking-aisi and soad-shoulder; Argoverse V1.0 [2] defined the bus class, and Argoverse V2.0 (78 refined it with fine classes including school-bus
Prior art. LECO requires learning new classes continually in TPs, similar to class-incremental 74 85 They differ in important ways. First, CIL assumes brand-new labels that have no relations with the old [47 49 60), while LECO's new labels are fine-grained ones that expand the old / "coarse" labels (cf. Fig. Interestingly, CIL can be seen
learning (CIL) [18
special case of LECO by making use of a "catch-all" background class 41 (often denoted as void or unlabell led in existing datasets [14 22 49
(55) D. which can be refined over time to include specific classes that were previously ignored. Second, CIL restricts itself to a small memory buffer for storing historical data to focus information retention and catastrophic forgetting [00 47]. However, LECO stores all historical data (since large-scale human annotations typically come at a much higher cost than that of the additional storage) and is concerned with the most-recent classes of interest. Most related to LECO are approaches that explore the relationship of new classes to old ones; Sariyildiz er al. [64] study concept similarity but remove common superclasses in their study, while Abdelsalam al. explore CIL with a goal to discover inter-class relations on data labeled (as opposed to assuming such relationships can be derived from a given taxonomic hierarchy). To explore LECO, we set up its benchmarking protocol (Section
52
33 and study extensive methods modified from those of related
problems (Sections
5 and
Technical insights. LECO studies learning strategies and answers questions as basic as whether one should label new data or relabel the old date") Interestingly, both protocols have been used in the community for large-scale dataset versioning: Argoverse [12 78 annotates new data, while Mapillary [52 (55) relabels its old data. Our experiments provide definitive answer
should always annotate new data with the new ontology rather than reannotating the old data. The simple reason is that the former produces more labeled data. However, this comes cost: the aggregate
We assume the labeling cost is same for either case. One may think re-annotating old data is cheaper, because does not need to the old annotations (c.g., show old vehicle labels and just ask for fine- -ne-igr make-and- and-model label). But in practice, in order to prevent cvrry-over of annotation errors and to simplify the annotation interface, most benchmark curators tend to relabel from scratch, regardless of on old or new data. Examples of relabeling
new data and may be able redesign annotation interface that exploits just
old-data from scratch include Mapillary V1.2 [55 V2.0 [52] and COCO 48
LVIS [27]

---

Benchmark
#TPs
#classes/1P
#train/TP
#tes//PP
Table 1: LECO benchmarks use CIFAR100 42]. iNaturalist [69] (5). and Mapillary [52 [55] We also vary the number of training data in exper- iments; as conclusions hold across settings, we report these results in Table ??, ??, ??.
CIFAR-LECO
2
20/100
10k
10k
(Nat-LECO
2
123/810
50k
4k
Mapillary-LECO
2
66/116
2.5k
2k
(Na-ITP-LECO
123/339/729/810
25k
4k
dataset is now labeled with inconsistent old-vs-new annotations, complicating learning. We show that joint-training [47] on both new and old ontologies, when combined with semi-supervised learning (SSL), can be remarkably effective. Concretely, we generate pseudo labels for the new ontology on the old data. But because pseudo labels are noisy 44 58 68) [9] and potentially biased [77]. we make use of the coarse labels from the old-ontology as "coarse supervision", or the coarse-fing fine label relationships [26 45 61 73), to reconcile conflicts between the pseudo fine labels and the true coarse labels (Section O.. There is another natural question: should one finetune the previous TP's model or train from scratch? One may think the former has no benefit (because it accesses the same training data) and might suffer from local minima. Suprisingly, we find finetuning actually works much better, echoing curriculum learning [5 21 63 Perhaps most suprisingly, we demonstrate that such strategies are near optimal, approaching an "upperbound" that trains on all available data re-annotated with the new ontology.
Salient results. To study LECO, we repurpose three datasets: CIFARIO0, iNaturlist, and Mapillary (Table IT. The latter two large-scale datasets have long-tailed distribution of class labels; particularly, Mapillary's 's new ontology does not have strictly surjective mapping to the old, because it relabeled the same data from scratch with new ontology. Our results on these datasets lead to consistent technical insights summarized above. We preview the results on the iNaturalist-LECO setup (which simulates the large-scas scale long-tailed scenario): (1) finetuning the previous TP's model outperforms training from scratch on data labeled with new ontology: 73.64 73.64% vs. 65. 65.40% in accuracy; (2) jointly training on both old and new data significantly boosts accuracy to 82.98%; 9 (3) taking advantage of the relationship between old and new labels (via -artning-itt-PPrriill--- refinement) further improves to 84.34%, effectively reaching an "upperbound" 84.51%!
and SSL with pseudo-label
Contributions, We make three major contributions. First, we motivate the problem LECO and define its benchmarking protocol. Second, we extensively study approaches to LECO by modifying methods from related problems, including supervisd learning, caaseminrremetall learning, and learning with partial labels. Third, we draw consistent conclusions and technical insights, as described above.
2 Related Work
and LECO both require learning classifiers for new classes over distinct time periods (TPs), but they have important differences. First, the typical CIL setup assumes new labels have no relations with the old 47 49). while LECO's new labels refine or expand the old / "coarse" labels (cf. Fig. [ Second, CIL purposely sets small memory buffer (for storing labeled data) and evaluates accuracy over both old and new classes with the emphasis on information retention or catastrophic forgetting 100 40 47 (2). However, LECO allows all (history) labeled data to be stored and highlight the difficulty of learning new classes which are fine/subclasses of the old ones. Further, we note that many real-wod wal-world applications store history data (and should not limit a buffer size to save data) for privacy-related consideration (e.g., medical data records) and as forensic evidence (e.g., videos from surveillance cameras). Therefore, to approach LECO, we apply CIL methods that do not restrict buffer size, which usually serve as upper bounds in CIL [17]. In this work, we repurpose "upper bound" CIL. methods for LECO including finetuning 3 80 joint training IT 47 51) as well as the "lower bound" methods by freezing the backbone
Class-incremental learning (CIL) 18 47 74 85
20 66).
Semi-supervised learning (SSL) learns over both labeled and unlabeled data. State-of-the-art SSL methods follow self-training paradigm [54 65 79 that uses model trained over labeled data to pseudo-label the unlabeled samples, which are then used together with labeled data for training. For example, Lee et al. 44 use confident predictions as target labels for unlabeled data, and Rizve er al. [62] further incorporate low- low-probability predictions as negative pseudo labels. Some others improve which force predictions to be similar for We leverage insights of SSL to approach I.COO e.g., pseudo labeling the old data. As pseudo labels are often biased [77), we further exploit the old-ontology to reject or improve the inconsistent pscudo labels, yielding better performance.
SSL using self-supervised learning techniques 24 83
different augmentations of the same data 6 7
68
79
84
3

---

Learning with partial labels (LPL) tackles the case when some examples are fully labeled while others are only partially labeled ( [5 labels can be coarse superclasses which annotate some training data [3
26
53 56 87 In the area of fine-grained recognition, partial
In a TP of LECO, the old data from previous TPs can be used as partially-labeles examples as they contain only coarse labels. Therefore, to approach LECO, we explore state-of-the-art methods (70) in this line of work. However, it is important to note that ontology evolution in LECO can be more than splitting old classes, e.g., the new class special suubecube can be a result of merging multiple classes child and poli ce -fliice-ffffooiiieeeeeeeeeeelee- -officer Indeed, we find it happens quite often in the ontology evolution from the Mapillary's 's versioning from V1.2 [55] to V2.0 [52]. In this case, the LPL method does not show better results than the simple joint training approach (31.04 vs. 31.05 on Mapillary in Table 4).
45
61 73
3 Problem Setup of Learning with Evolving Class Ontology (LECO)
Notations. Among T time periods (TPs), TP' has data distributions D' = X x Y', where X is the input and Y is the label set. The evolution of class ontology ideally modeled as a tree structure: the 2th class y Y! is a node that has a unique parent y E Yt-1, which can be split into multiple fine classes in TP'. Therefore, Y Y
D'
TP' concerns classification w.r.t label set Y
Benchmarks. We define LECO benchmarks using three datasets: CIFAR 100 [42 iNaturalist 66 (55], and Mapillary 552 [T CIFARI00 is released under the MIT license, and iNaturalist and Mapillary are publicly available for non- non-commercial research and educational purposes. CIFAR 100 and Mapillary contain classes related to person and potentially have fairness and privacy concerns, hence we cautiously proceed our research and release our code under the MIT License without re-distributing the data. The iNaturalist and Mapillary are large-scale and have long-tailed class distributions (refer to the Table ?? and ?? for class distributions). For each benchmark, data from the corresponding dataset to construct time periods (TPs), e.g., TP" and TP' define their own ontologies of class labels Y and Y respectively. For CIFAR-LECO, we use the two-level hierarchy offered by the original CIFAR 10 dataset 42]: using the 20 superclasses as the ontology in TPo, each of them is split into 5 subclasses as new ontology in TP'. For iNat-LECO, we used the most recent version of Semi Aomi-iNat-225 2021 (71] that comes with rich taxonomic labels at seven-level hierarchy. To construct two TPs, we choose the third level (i.e., "order") with 123 categories for TP", and the seventh ("species")) with 810 categories for TP!. We further construct four TPs with each having ontology out of four distinet taxa levels ["order", "family", "genus", "species"]. For Mapillary-LECO, we use its ontology of V1.2 (55 (2017) in TPo, and that of V2.0 [52) (2021) in TP'. It is worth noting that, as a real-world LECO example, Mapillary includes catch-all background class (aka void) in V1.2, which was split into meaningful fine classes in V2.0, such as temporary caapporary-baarieee barrier, traffic- iis-issnddd and void- ground.
sample
Moreover, in each TP of benchmark CIFAR-LECO and iNat-LECO, we randomly sample 20% data as validation set for hyperparameter tuning and model select. We use their official valsets test-sets for benchmarking. In Mapillary-LECO, we do not use a valset but instead use the default hyperparameters reported in [72), tuned to optimize another related dataset Cityscapes [14]. We find it unreasonably computational demanding to tune on large-scale semantic segmentation datasets. Table summariis our benchmarks' statistics.
Remark: beyond class refining in ontology evolution. In practice, ontology evolution also contains the case of class merging, well as class renaming (which is trivial to address). The Mapillary's ' versioning clearly demonstrates this: 10 of 66 classes of its V1.2 were refined into new classes, resulting into 1 116 classes in total of its V2.0; objects of the same old class were either merged into a new class or assigned with new fine classes. Because of these non-trivial cases under the Mapillary-LECO benchmark, state-of-the-ee ahr methods of learning with partial labels are less effective but our other solutions work well.
Remark: number of TPs and buffer size. For most experiments, we set two TPs. One may think more TPs are required for experiments because CIL literature does so [8 CIL emphasizes catastrophic forgetting issue caused by using limited buffer to save history data, so using more TPs for more classes and limited buffer helps exacerbate this issue. Differently, LECO emphasizes the difficulty of learning new classes that are subclasses of the old ones, and does not
47
74 85
Recall that
'The data distribution may not be necessarily static where LECO can still happen, as discussed in Section Our work sets a static data distribution to simplify the study of LECO.

---

limit buffer to store history data. Therefore, using two TPs are sufficient to show the difficulty. Even so, we have experiments that four TPs, which demosntrate consistent conclusions drawn from those using two TPs. Moreover, even with unlimited buffer, CII. methods still work poorly on LECO benchmarks (cf. "TrainScratch" [59] and "FreezePrev"
Table 2
Metric. Our benchmarks study LECO through the lens of image classification (on CIFAR-LECO and iNal-LECO) and semantic segmentation (on Mapillary-LECOO, Therefore, we evaluate methods using mean accuracy (mAcc) averaged over per-class accuracies, and mean intersection-ovee-uuiioo (mloU) averaged over classes, respectively. The metrics treat all classes equally, avoiding the bias towards common classes due to the natural long-taie tailed class distribution (cf. iNaturalist [69] 75 and Mapillary (55)) (cf. class distributions in the appendix). We do not use specific algorithms to address the long-tailed recognition problem but instead adopt the recent simple technique to improve performance by tuning weight decay (2. We run each method five times using random seeds on CIFAR-LECO and iNat-LECO, and report their averaged mAcc with standard deviations. For Mapillary, due to exoribitant training cost, we perform one run to benchmark methods.
Baseline Approaches to LECO
We first repurpose CIL and continual learning methods as baselines to approach LECO, as both require learning classifiers for new classes in TPs. We start with preliminary backgrounds.
train convolutional neural networks (CNNs) and treat a network as feature extractor f (parametrized by 8,) plus a classifier g (parameterized by 0g). The feature extractor consists of all the layers below the penultimate layer of ResNet [28). and the classifier is the linear classifier followed by softmax normalization. Specifically, in TP', an input image x is fed into f to obtain feature representation = = f(x: Qy). The feature is then fed g(; 0g), w.r.t classes in Y We train CNNs by minimizing
Preliminaries. Following prior art (39
into g for the softmax probabilities, the Cross Entropy (CE) loss using mini-batch SGD. At TP', construct training batch B we randomly sample K examples, i.e., B ((I1,1)), (#K, 2k)) The CE loss on B is
L(B') Σ ((({))(((zx)))))
(BL
where H(p.g) = = P(c)log(((() is the entropy between two probability vectors.
Similary, for semantic segmentation, we use the same CE loss T at pixel level, along with the recent
architecture HRNet with OCR module 772 76 81 (refer to appendix C for details).
Annotation Strategies. Closely related to LECO is the problem of class-incremental learning (CIL) or more generally, continual learning (CL) [47 59 60 82 85). However, unlike typical CL benchmarks, LECO does not artificially limit the buffer size for storing history samples. Indeed, the expense of hard drive buffer is less of problem compared to the high cost of data annotation. Therefore, LECO embraces all the history labeled data, regardless of being annoated using old or new ontologies. This leads to a fundamental question: whether to annotate data re-label the old, given a fixed labeling budget at each TP (cf. Fig. ?
(LabelNew) Sample and annotate N new examples using new ontology, which trivially infers the old-ontology labels. We further have the history data as labeled with old ontology. (RelabelOld) Re-annotate the N history data using new ontology. However, these N examples are all available data although they have both old- and new-ontology labels.
The answer perhaps obvious- - LabelNew because it produces more labeled data. Furthermore, LabelNew allows one to exploit the old data to boost performance (Section(5). With proper techniques, this significantly reduces the performance gap with a model trained over both old and new data assumably annotated with new-ontology, termed as AllFine short for "supervised learning with all fine labels"
Training Strategies. We consider CL baseline methods as LECO baselines. When new tasks (i.e., classes of Y') arrive in TP', it is desirable to transfer model weights trained for y in TP--1 [47 60). To do we initialize weights 0y: of the feature extractor fe with 0p-1 trained in Tp--1 i.e., Oy: = 0 Then we can (FreezePrev) Freeze f to extract features, and learn classifier g' over them 47
5

---

Table 2: Results of bascline methods for LECO (more results in the Table ??). For all our experiments, each method five times and report the averaged mean per-class accurayyintersection perrectioooove--ueee over-union (mAcc/mlol %) with standard deviations when available. For each benchmark, we bold the best mAcc/mloU including ties that fall within its std. FinefunePrev is the best learning strategy that outperforms TrainScratch (cf. 73.64% vs. 65.69% with T LebelNew on iNat-IFCCO). This implies that the coarse-to-fing fine evolved ontology serves as good learning curriculum. FreezePrev underperforms TrainSeratch (cf. 50.82% vs. 65.69% with LabelNew on (Nat-LFCCO), confirming the difficulty of learning new classes that refine the old, LabelNew and RelabelOld achieve similar performance with the former using only new data but not the old altogether. If using both old and new data, we boost performance as shown in 3] The conclusions hold for varied number of training images and generalize the Mapillary experiments, yet all methods fall short the AllFine (which trains all data assumed to be labeled with new ontology).
TP mAcc/mloU
Benchmark
# #images/ TP
TPº mAcehnloU
TP Strategy
LabelNew
RelahclOld
AllFine
FinetunePrev
65.83 0.53
65,30
0.32
71.30 0.41
CIFAR-LECO
77.78 1 0.27
FreezePrev
52.61 0.50
52.60 = 0.52
53.79 + 0.43
10000
TrainScratch
65.40 = 0.53
61.98 = 0.34
71.43 0.24
FinetuncPrev
73.64 = 0.46
71.26 + 0.54
84.51 +: 0.66
iNar-LECO
64.21
1.61
FreezePrev
50.82 0.68
54.60
0.49
54.08 + 0.31
50000
TrainSeratch
65.09 0.46
65.78 + 0.56
73.10 0.80
FinetunePrev
30.39
29.05
31.73
Mapillary-LEoe LECO
2500
37.01
TrainScratch
27.24
27.21
27.73
(FinetunePrev)
Finetune f" and learn classifier g altogether 47
(TrainSeratch) Train fc from random weights [59].
Results. We experiment with different combinations of data annotation and training strategies in T We adopt standard training techniques including SGD with momentum, cosine annealing learning rate schedule, weight decay and RandAAgment [16]. We ensure the maximum training epochs (2000/300/800 on CIFAR/NNa/Mapillar respectively) to be large enough for convergence. See Table caption for salient conclusions.
5 Improving LECO with the Old-Ontology Data
The previous section shows that FinetunePrev is the best training strategy, and LabelNew and RelabelOld achieve similar performance with the former being not exploiting the old data. In this section, we stick to FinetunePrev and LabelNew and study how to exploit the old data to improve LECO. To do so, we repurpose methods from supi-super learning and incremental learning.
Notation: We refer to the new-ontology (fine) labeled samples at TP' as St C X x Y and old-ontology (coarse) labeled samples accumulated from previous TPs Slit-1 St S
5.1 Exploiting the Old-Ontology Data via Semi-Supervised Learning (SSL)
State-of-the-an art SSL methods aim increase the amount of labeled data by eseudo-labeling the unlabeled examples [25] 770. Therefore, natural thought to utilize the old examples is to ignore their old-ontology labels and use SSL methods. Below, we describe several representative SSL methods which are developed in the context of image classification.
Each input sample æ. if not stated otherwise, undergoes strong augmentation (RandAugment [16 for classification and HRNet [72 augmentation for segmentation) denoted as A(z) before fed into
f. For all SSL algorithms. we construct batch B5 of K samples from S; and a batch Bas of M samples from S 11 For brevity, we hereafter ignore the superscript (TP index). On new-ontology labeled samples, we use Eq. ① to obtain L(Bx), and sum it with an extra SSL loss on Byg denoted as LSST. (Bar). We have four SSL losses below.
Following 77), we use "Self-Training" to refer to teacher-student distillation procedure 29 Note that we adopt the FinetunePrev strategy and so both teacher and student models are initialized from the previous TP's weights. We first have two types of Self- Traininn:::::

---

ST-Hard (Self- Training with one-hot labels): We first train a teacher model (fteacher and 9teocher) on the training set St, then use the teacher to "pseudo-label" history data in S Rssentially, for a (Im, ym) E S1it=1, the old-ontology label yo is ignored and one-hot hard label is produced as Qh = arg max Gteacher (Steacher (Leacher(AA(r.)) (A(mm))) Finally, we train a student model (ʃ stuudent and Istudent) with CE loss:
LSSI. (Bm)=
Σ
Hlghs Istudent (f stsuddon (A())))
(fm:))88
ST-Soft (Self- Training with soft labels): The teacher is trained in same way as ST- Hard, but the difference is that pscudo-labels are softmax probability vectors gs 9teocker teecher (A(zm ))) instead. The final loss is the KL divergence [58] 70]:
CSSL(BM)
Σ H(gs: Istudent sttudett (A(rm)))
(3)
(Zm.))EBBr
In contrast to the above SSL methods that train separate teacher and student models, the following two train single model along with the pseudo-labeling technique.
PL (Pseudo-Labeling) [44] pseudo-labels image I' as q = g( (f (A(Tm))) using the current model. also filters out pseudo-labels that have probabilities less than 0.95.
(Bst(BB))
Σ
[max
(4)
(xmr)EBu
FixMatch [68] imposes consisteney regularization into PL by enforcing the model to
produce the same output across weak augmentation (a(#m), usually standard random flipping and cropping) and strong augmentation (A(zm), in case is weak augmentation followed
by RandAugment [16). The pseudo-label produced by weakly augmented version qu = g( (a(())))) The final loss is:
(BsL(BM) Σ
1 [max que > 0.95) ((
(5)
(Emr))BBx
While the above techniques are developed in the context of image classification, they appear to be less effective for semantic segmentation (on the Mapillary dataset), e.g., ST-Soft requires extremely large storage save per-pixel soft labels, FixMatch requires two large models designed for semantic segmentation. Therefore, in Mapillary-LECO, we adopt the ST-Hard which is computationally friendly given our compute resource (Nvidia GTX-3090 Ti with 24GB RAM).
5.2 Exploiting the Old-Ontology Data via Joint Training
The above SSL approaches discard the old-ontology labels, which could be a coarse supervision in training. A simple approach to leverage this coarse supervision is Joint Training, classic incremental learning method [8] 47 that trains jointly on both new and old tasks. Interestingly. Joint Training is usually the upper bound performance in traditional incremental learning benchmarks [47). Therefore, we perform Joint Training to learn shared feature extractor f and independent classifiers w.r.t both new and old ontologies. We define the following loss at TP* using old-ontology data from Tpl to 1
gg') ..'....
(B))
Σ
Σ
Mly 9 (f(A(rm))
MEIL,-. - 1-1) (Em..N))BB
where I the coarse label of Im at TP' and B% is the subset of Bar with label yf available. Importantly, because new-ontology fine labels y! Y' refine old-ontology coarse labels, e.g.. y +-1 E y we can trivially infer the coarse label for any fine label. Therefore, we also apply Lotd for fine- fine-grained samples in Bx, i.e., (B))(Bx). We have the final loss for Joint Training as Joint (Bx)(Bk) (Bra(B
5.3 Results
Table 3 compares the abooe-mentiones SSL and Joint Training methods. In training, we sample the same amount of old data and new data batch, i.e., [Bar| = |Bx| (64 30 / 4 for CIFAR / (iNat / Mapillary). We assign equal weight LESL. Joint and L. See Table Caption for conclusions.
7

---

Table 3: Results of SSL and Joint Training methods. Following the notation of Table 2 we study LabelNew annotation strategy that make use of algorithms for learning from old examples without exploiting the relationship between old-vs-new labels. Recall that AllFine uses & only to train on all the data labeled with new ontology (doubling annotation cost). Both Loint and the best Lsss. (ST-Sofft outperform the baseline which uses loss L only (i.e., LahelNew + FinetunePrev in Table 2 ST-Soft yields the most performance gain. Using all the three losses together consistently improves mAce/ / mloU for all the henchmarks, confirming the benefit of exploiting the old examples, e.g,, on iNor-LFCCO, this boosts the performance to 83.6%, approaching the AllFine (84.5%)
TP' mAcc/mlol
Benchmark
#images TP
SSL Alg.
Lavint +Sst
L only
+SssL
+ +osiit
AllFine
Ha-Hard
67.11 + 0.34
67.97 + 0.37
ST-Soft PL.
69.86 +: 0.30
69.77 = 0.71
CIFAR-LECO
10000
65.83 + 0.27
68.74
71.30 = 0.41
65.86
68.43
FixMatch
67.13 + 0.36
69.03 + 0.44
ST-Hard
75.23 + 0.14
+ 0.38
ST.Son
79.64 + 0.41
83.61 + 0.39
iNar-LECO
50000
73.64 0.46
82.98 + 0.33
84.51 = 0.66
PL.
73.56
82.87 +
FixMatch
71.37 + 0,54
83.0 03.00 + 0.33
MapillaryLEECO
2500
ST-Hard
30.39
30.06
31.05
31.09
31.73
6 Improving LECO with Taxonomic Hierarchy
While the SSL approaches exploit old samples and Joint Training further utilizes their old-ontology labels, none of them fully utilize the given taxonomic relationships between old and new ontologies. The fact that each fine- seeegrained label y E Y refines a coarse label y E Y suggests a chance to improve LECO by exploiting such taxonomic relationship between the old and new classes [58 69]
6.1 Incorporating Taxonomic Hierarchy via Saamiing-itt--Partia----
(LPL)
Joint Training trains separate classifiers for each TP. This is suboptimal because we can simply marginalize fine-classes' probabilities for coarse-classes' probabilities [19]. This technique has been used in other works [32 69] and is commonly called Learningwwith Setpiing-with--rria------ee-e Partial-Labels (LPL) [32]. Concretely, the relationship between classes in Y and Y with t' can be summarized by an edge matrix Ei-r with shape Y x |Y'|. E,- [i. = 1 when an edge yf - v exists in the taxonomy tree, i.e., of is the superclass of w/: otherwise E, (i, j] prediction probability of a coarse-grained label y" for sample we sum up the probabilities in (x)) that correspond to fine labels of y",, which can be efficiently done by a matrix vector multiplication: g' (x)) Er Formally, we define loss over a sampled batch B&r C. CS1ttt1 as: LOILLPI (BM)
Therefore, to obtain
Σ
Hlyiin g (f' (A(Im))) Ex-re)
VE(L,...111) (Rm::)))88
We apply this loss on the fine-labeled data and define the LPL Loss LLPL (otarri(B)) Lo
6.2 Incorporating Taxonomic Hierarchy via Pseudo-label Refinement
Another strategy to incorporate the taxonomic hicrarchy is to exploit coarse labels in the pscudo- labeling process, which has been explored in a different task (semi-supervised few-shot learning) (58) Prior art shows that the accuracy of pseudo labels is positively correlated with SSL performance (88
Therefore, at TP', in pseudo-labeling an old sample Em, we use its coarse label yr (t' < t) to clean up inconsistent pseudo labels. In particular, we consider
1. Filtering: we filter out (reject) pseudo-labels that do not align with the ground-truth coarse labels, i.c., rejecting pseudo-label q = g" (f*(())) E.-re (arg max 9c: v =
2. Conditioning: we clean up the pseudo-label g by zeroing out all scores unaligned with the true coarse label: q(c] == q(c] E,-rr (c, y We then renormalize qlc] = glc//E, q(i].
8

---

Table 4: Results of exploiting taxonomic hierarchy (aceuracy in %). We copy the best results in Table (all obtained by Jaint +LSst) for reference. Adding supervision taxonomic hierarchy effective: (I) simply using Lt.II. already matches the previously best performance (by by +oo +ssLt): (2) adding SSL loss along with pseudo-label refinement via either Filtering or Conditioning further bridges the gap to AllFine. As result, for example, on iNat-LECCO, the best strategy (+LLPL + Listt/Cond with ST-Hard) is only 0.2% lower than the AllFine!
TP' mAce
Benchmark
Wimages/ / TP
SSL Alg
Loont CoRL
+CIPL +CsL//Filtr
+LPP
+LLPL
+Sst//Cond
AllFine
ST-Hard ST-Soff
69.90 0.29
70.42 = 0.23
70.10 0.12
70.02 :: 0.63
CIFAR-LECO
10000
6.77
69.31 + 0.30
=
PL FixMatch
69.20 1: 0.20
69.60 +: 0.26
69.61 0.17
69.85 + 0.46
ST-Hard ST
84.23 + 0.61
84.34 = 0.25
84.12
83.76
iNat LEEC
50000
83.61 + 0.39
83.62 + 0.21
84.51 + 0.66
PL
83. 72 = 0.22
84,16 = 0.19
FixMatch
83.91 0.39
83.95 :: 0.40
Mapillary-IFCCC
2500
ST-Hard
31.09
31.04
31.41
31.23
31.78
Table 5: Baseline results -Ta--LT-- (accuracy in %). FinetunePrev significantly and consistently outperforms TrainScratch, suggesting representation trained coarse-grained grained classification can serve as powerful initialization for finetning later TPs. Furthermore, LabelNew and RelabelOld achieve almost the same performance across all TPs, but we will show in Table|ojtha once exploiting the old-ontology data and labels, can significantly improve the performance, approaching the "upperbound" AllFine.
coarse-
Benchmark
#images/ TP
TP' mAce
Training Strategy
Annotation
TPl mAce
TP² mAce
TP'l mAce
LabelNew
50.9 + 0.6
49.4 0.6
48.4 0.6
TrainScratch
RelabclOld
50.1 1.1
49.7 0.4
47.9 0.6
AllFine
62.7 + 0.5
63.8 + 0.3
58.0 0.2
iNa-44P--LECO
25000
46.6 1.2
LabelNew
55.1 0.7
61.7 :: 0.8
63.2 +: 0.4
FinetuneNew
RelabelOld
55.6 0.9
61.7 0.9
63.1 + 0.4
AllFine
67.7
78.8 + 0.4
84.6 0.3
6.3 Results
Table 4 shows the results of LPI. Loss and SSL strategies with pseudo-label refinement. We can see that simply adding the LPI. Loss LLP7. matches the SOTA (achieved by naive SSI. and Joint Training strategies in 3). Also, adding SSL loss along with pseudo-label refinement via either Filtering or Conditioning further bridges the gap to AllFine. For example, on iNat-LECO, the best strategy (+L1PL LSsL/Cond with ST-Hard) is only 0.2% lower than the AllFine (84.3% v.s. 84.5%). We acknowledge that the AllFine can be improved further if using LPL loss as well 8 but we use the current version just as reference of performance. We also tried combining LSSL/Fiter or Liss;/Cond with the Joint Training, and observe similar improvements upon LSsI. Loint (results in Table ?? and ??).
7 Further Experiments with More Time Periods
We further validate our proposed approahces with more TPs. In this experiment, we construct benchmark (Nat-4PP-LECO using iNaturalist by setting four TPs, each having an ontology at distinct taxa levels: "order" (123 classes), "family" (339 classes), "genus" (729 classes). "species" (810 classes). We show in Table 5 that FinetuneNew outperforms TrainSeratch from TP' to TP3 Furthermore, we study LabelNew annotation strategy with FinetunePrey training strategy (as in Table 3 and 4 for iNal-4TP-LECO, and report results Table b. summary, the proposed solutions generalize to more than two TPs, and simple techniques that utilize old-ontology labels (such as L.Joint) and label hierarchy (such as LI.PI.) achieve near-optimal performance compared to AllFine "upperbound".

---

Table 6: Results of INa--TTP-LECO with FinetuneNew training strategy and LabelNew annotation strat- egy (accuracy in %). We select two performing SSL methods on iNat-LECO in Table FixMatch and ST-Soft, for benchmarking. Clearly, all solutions proposed in the paper achieve consistent gains over baseline methods (Table 5). First, leveraging old-ontology data via SSL (e.g.. +LT.san)) can improve the classifica- tion accuracy from 63.2% 67.6% at TP3. Also, utilizing the old-ontology labels via simple joint training boost performance 83.0%. Lastly, incorporating taxonomic hierarchy through I.PL. and SSI. (e.g..
+LLPL + LSTsorrtee) further improves to 85.3%.
Benchmark
#images TTP
Combination of losses
TP mAce
TP MAec
TP mAce
AllFine
67.7 + 0.6
78.8 + 0.4
84.6 + 0.3
L only
55.1 + 0.7
61.7 0.8
63.2 + 0.4
CriaMach
56.7 + 1.0
61.1 1.0
65.0 + 0.9
+ LSTSOo
59.6 1.2
67.8 1.2
67.6 1.3
+Doiitt
64.7
74.3 0.9
8.9
iNat-4TP-LECO
25000
+ Loiint + LFisMunch
64.6 0.8
0.7
82.8 0.7
+ Losintt + SOT
64.4 1.4
7
83.0 +3.0+10 1.5
+LLPS.
67.5 +
0.9
83.6 1.0
+LLPE* + LFisMaccFFiler
67.7 0.8
7.9
85.3 0.6
+LLP + LHisMuchooodd
67.5
0.8
0.6
+ CLPE + LST-SotFiller
67.7 1 1.1
1.1
85.1 1.2
+ +LLPL + LST-SoCCond
6
7.37+13
7..4
8 Discussions and Conclusions
We introduce the problem Learning with Evolving Class Ontology (LECO) motivated by the fact that lifelong learners must recognize concept vocabularies that evolve over time. To explore LECO, set up its benchmarking protocol and study various approaches by repurposing methods of related problems. Extensive experiments lead to several surprising conclusions. First, we find the status-quo for dataset versioning. where old data is relabeled, is not the most effective strategy. One should label new data with new labels, but the mixture of old and new labels is effective only by exploiting semi-supervised methods that exploit the relationship between old and new labels. Surprisingly. such strategies can approach an upperbound of learning from an oracle aggregate dataset assumably annotated with new labels.
Societal Impacts. Studying LECO has various positive impacts it reflects open-world development of machine-learning solutions. For example, LECO emphasizes lifelong learning and maintainance of machine-leamnn learning systems, which arise in inter-discipliaary research (where bio-image analysis system needs to learn from refined taxonomy, of. iNaturalist), and autonomous vehicles (where critical fine- mne-grained classes appear as operations expand more cities). We are aware of negative societal impacts of LECO over those that arise in traditional image classification.
Future work address limitations. We notice several limitations and anticipate future work to address them. Firstly, the assumption of constant annotation cost between the old and new data needs to be examined, but this likely requires the design of novel interfaces for (re)annotation. Secondly, our work did not consider the cost for obtaining new data, which might include data mining and cleaning. Yet, in many applications, data is already being continuously collected (e.g.. by autonomous fleets and surveillance cameras). This also suggests that LECO incorporate unlabeled data. Moreover, perhaps surprisingly, finetuning previous TP's model significantly outperforms training from scratch. This suggests that the coarse-to-fine ontology evolution serve as a good curriculum of training, a topic widely studied in cognition, behavior and psychology [43
67 Future work should investigate this, and favorably study efficiently finetuning algorithms [13
57
[6). Lastly, we explore LECO on well-known benchmarks with static data distributions over time periods; the future, we
30
38
46
would like to embrace temporal shifts in the data distribution as well [4
Acknowledgments and Disclosure of Funding
This research was supported by CMU Argo Al Center for Autonomous Vehicle Research.
10

---

References
[1] Abdelsalam, M., Faramarzi. M.. Sodhani. S., and Chandar, S. (2021). lire: Incremental implicitlyrrefined refined classification. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11038-11047.
[2] Alshammari, S., Wang. Y., Ramanan, D., and Kong. S. (2022). Long-taile recognition weight balancing. In CVPR.
[3] Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., and Carlsson, (2015). Factors transferability for a generic convnet representation. IEEE transactions on pattern analysis and machine intelligence, 38(9),
[4] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C.,, and Gall, J. (2019). Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV.
[5] Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48.
[6] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. (2019a). Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 32.
[7] Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. (2019b). Remixmatch: semii-uuprrvied learning with distribution alignment and augmentation anchoring. arXiv preprint ar::v 19170975.
[8] Bertinetto, L., Mueller, R., Tertikas, K., Samangooei, S., and Lord, N. A. (2020). Making better mistakes: Leveraging class hierarchics with deep networks. In CVPR.
[9] Bucak. S. S., Jin, R., and Jain, A. K. (2011). Multi-label learning with incomplete class assignments. In CVPR 2011, pages 2801-2808. IEEE.
[10] Cai, Z.., Sener, O., and Koltun, V. (2021). Online continual learning with natural distribution shifts: An empirical study with visual data. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 828 9281-8290.
[11] Caruana, R. (1997). Multitask learning. Machine learning. 28(1), 41-75.
[12] Chang, M.-F., Lambert, J.. Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang. D., Carr, P., Lucey, S., Ramanan, D., et al. (2019). Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 8748-8757.
[13] Cohen, N., Gal, R., Meirom, E. A., Chechik, G., and Atzmon, Y. (2022). "his is my unicorn, fluffy": Personalizing frozen dision-languae representations. arXiv preprint 21144
[14] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler. M., Benenson, R., Franke. U., Roth, S., and Schiele. B. (2016). The cityscapes dataset for semantic urban scene understanding. In CVPR.
[15] Cour, T., Sapp, B., and Taskar, B. (201 Learning from partial labels. The Journal of Machine Learning Research, 12, 1501-1536.
[16] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 702-703.
[17] De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. (2021). A continual learning survey: Defying forgetting in classification tasks. IEFF transactions on pattern analysis and machine intelligence, 44(7), 3366-3385.
[18] Delange, M., M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, (2021). A continual learning survey: Defying forgetting in classification tasks. PAMI.
[19] Deng, J., Ding, N., Jia, Y., Frome, A.. Murphy. K.. Bengio, S., Li. Y., Neven. H., and Adam, H. (2014). Large-scale object classification using label relation graphs. In European conference on computer vision, pages 18-64. Springer.
[20] Donahue, J., Jia, Y.. Vinyals, O., Hoffmman, J., Zhang. N., Tzeng, E., and Darrel,, T. (2014). Decaft deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 655. PMLR.

---

[21] Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48(1), 71-99.
[22] Everingham, M., Eslami, S. A., Van Gool, L.., Williams, C. K., Winn, J., and Zisserman, A. (2015). The pascal visual object classes challenge: A retrospective. IJV,, 111(1), 98-136.
[23] Geiger, A., Lenz, P. and Urtasun, R. (2012). Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3360 3364.. IEEE.
[24] Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., and Cord, M. (2019). Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8059-8066 8068.
[25] Goel, A., Fermando, B., Keller, F., and Bilen, H. (2022). Not all relations are equal: Mining informative labels for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15596-15606.
[26] Grandvalet, Y., Bengio, Y., et al. (2004). Learning from partial labels with minimum entropy. Technical report, CIRANO.
[27] Gupta, A., Dollar, and Girshick, R. (2019). Lvis: dataset for large vocabulary instance segmentation. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5356-5364.
[28] He, K., Zhang, X., Ren, S., and Sun, (2016). Deep residual learning for image recognition. In CVPR.
[29] Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0553)
[30] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficicn transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
[31] Hsieh, C.-Y., Xu, M., Niu, G., Lin, H.-T., and Sugiyama, M. (2019). A pseudo-label method for coarse-to- fine multi-lahel learning with limited supervision.
[32] Hu, P., Lipton, Z. C., Anandkumar, A., and Ramanan, D. (2018). Active learning with partial feedback. arXiv preprint arX::::80222 172.:0700..
[33] iNaturalist 2017 (2017).
[34] iNaturalist 2018 (2018). http::/www.naauraalst..t//t////
[35] iNaturalist 2019 (2019). http://www.iaauuaallttccrrtttttt
[36] iNaturalist 2020 (2020).
httea//www..ntuuuallssscrr//ssss
[37] INaturalist 2021 (2021). httpe:/www..natuuualssttssssssss
[38] Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, and Lim, S.-N. (2022). Visual prompt tuning. arXiv preprint 12X11:2203.12111.
[39] Kang, B.. Xie, Rohrbach, M., Yan, Z., Gordo, A.. Feng, J., and Kalantidis, Y. (2020). Decoupling repre- sentation and classifier for long-tailed recognition. In International Conference on Learning Representations.
[40] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska Barwinska, A., et al. (2017), Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-33 1222....
[41] Kong, S. and Ramanan, D. (2021). Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822.
[42] Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
[43] Kruegor, K. A. and Dayan, P. (2009). Flexible shaping: How learning in small stops helps. Cognition, 110(3), 380-394.
[44] Lee, D. et al. (2013). Pseudo-label: The simple and efficient semi- semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning. ICML, volume 3, page 896.
12

---

[45] Lei. J. Guo, Z., and Wang, Y. (2017). Weakly supervised image classification with coarse and fine labels. In 2017 14th Conference on Computer and Robot Vision (CRV), pages 240-247. IEEE.
[46] Li, W.-H., Liu, X., and Bilen, H. (2022). Cross-domain few-shot learning with task-specific adapters. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7161-7170.
[47] Li. Z. and Hoiem, D. (2017). Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12), 2935-2947.
[48] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.. (2014). Microsoft coco: Common objects in context. In ECCV. Springer.
[49] Lin, 7., Shi, J., Pathak, D., and Ramanan, D. (2021). The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[50] Lomonaco, V. and Maltoni, D. (2017), Core50: a new dataset and benchmark for continuous object recognition. In Conference Robot Learning, pages 17-26. PMLR.
[51] Maltoni, D. and Lomonaco, V. (2019). Continuous learning in single-iicreementaal-skk scenarios. Neural Networks, 116, 56-73.
[52] Mapitlary- (2021).
Vi stsstas-22 2atas-2ddataser .tmml
[53] Marszalek, M. and Schmid, C. (2007). Semantic hierarchies for visual object recognition. In 2007 IEEE Conference Computer Vision and Pattern Recognition, pages 1-7. IEEE.
[54] achlaan, G. J. (1975). Iterative reclassification procedure for constructing an asymptotically optimal rule allocation in discriminant analysis. Journal of the American Statisticul Association, 70(350), 365-366
[55] Neuhold, G., Ollmann, T., Rota Bulo, S., and Kontschieder, P. (2017). The mapillary vistas datasct for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990-4999.
[56] Nguyen, N. and Caruana, R. (2008). Classification with partial labels. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 551-559.
[57] Peterson, G. B. (2004). day of great illumination: skinner's discovery shaping. Journal of the experimental analysis of behavior, 82(3), 317-328.
[58] Phoo, C. P. and Hariharan, B. (2021). Coarsely- labeled data for better few-shot transfer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9052-9061.
[59] Prabhu, A., Tom, P. H., and Dokania, K. (2020). Gdumb: simple approach that questions our progress in continual learning, In ECCV.
[60] Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. (2017). icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010.
[61] Ristin, M., Gall, J., Guillaumin, M., and Van Gool, L. (2015). From categories to subcategories: large-scale image classification with partial class label refinement. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 231-239.
[62] Rizve, M. N., Duarte, K., Rawat, Y. S., and Shah, M. (2021). In defense of pscudo-labeling! An uncertainty- aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.066 06329.
[63] Sanger, D. (1994). Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE transactions on Robotics and Automation, 10(3), 323-3333
[64] Sariyildiz, M. B., Kalantidis, Y., Larlus, D., and Alahari, K. (2021). Concept generalization in visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9629-9639.
[65] Seudder, H. (1965). Probability of error of some adaptive pattern-recoonitioo machines. IEEE Transactions Information Theory, 11(3). 363-371.
13

---

[66] Sharif Razavian, A., Azizpour, H.. Sullivan. J., and Carlsson, S. (2014). Cun features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference computer vision and pattern recogmition workshops, pages 806-813,
[67] Skinner, B. (1958). Reinforcement today. American Psychologist, 13(3), 94.
[68] Sohn, K., Berthelot, D., Carlini, N., Zhang. Z., Zhang, H., Raffel. C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurlPS.
[69] Su, J. and Maji. S. (2021a). Semi-supervised learning with taxonomic labels. In British Machine Vision Conference (BMVC).
[70] Su, J., Cheng, Z., and Maji, S. (2021). A realistic evaluation of semi-supervised learning for fine-grained classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[71] Su, J.-C and Maji, S. (2021b). The semi-supervised inaturalist challenge at the fgve8 workshop. arXiv preprint arXiv:2106.01366
[72] Sun, K., Xiao, B., Liu, D., and Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. In CVPR.
[73] Taherkhani, F., Kazemi, H., Dabouci, A., Dawson, J., and Nasrabadi, N. M. (2019). A weakly supervised fine label classifier enhanced by coarse supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6459-6468.
[74] Van de Ven, G. M. and Tolias, S. (2019). Three scenarios for continual learning. arXiv preprint arXiv: 994000734...
[75] Van Hom, G., Mac Andha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. (2018). The inaturalist species classification and detection dataset. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8769-8778.
[76] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang. X., Liu, W., and Xiao, B. (2019). Deep high-resolution representation learning for visual recognition. TPAMI.
[77] Wang, X., Wu, Z., Lian, I.., and Yu, X. (2022). Debiased learning from naturally imbalanced pscudo- labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14647-14657.
[78] Wilson, B., Qi, W., Agarwal, T., Lambert, J., Singh, J., Khandelwal, S., Pan, B., Kumar, R., Hartnett, A., Pontes, J. K., et al. (2021). Argoverse 2: Next generation datasets for self-driving perception and forecasting.
[79] Xic, Q., Luong, M.TT, Hovy, E., and Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10687-10698.
[80] Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? NeurlPS.
[81] Yuan, Y., Chen, X., and Wang, J. (2020). Object-contextual representations for semantic segmentation.
[82] Zenke, F, Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. In ICML.
[83] Zhai, X., Oliver, A., Kolesnikov, A., and Beyer, L. (2019). S4I: Self-supervised semi supervised learning.
In Proceedings of the IEEESCVF International Conference on Computer Vision, pages 1476-1485.
[84] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018), mixup: Beyond empirical risk minimiza- tion. In International Conference on Learning Representations.
[85] Zhang, J., Zhang, J.. Ghosh, S., Li, D., Tasci, S., Heck, L., Zhang. H., and Kuo, C.-. J. (2020a). Class- incremental learning via deep model consolidation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1131-1140.
[86] Zhang, J. O., Sax, A., Zamir, A., Guibas, L., and Malik, J. (2020b). Side-tuning: a baseline for network adaptation via additive side networks. In European Conference Computer Vision, pages 698 -144..1455 Springer.
[87] Zweig, A. and Weinshall, D. (2007). Exploiting object hicrarchy: Combining models from different category levels. In 2007 IEEE 1hth international conference computer vision, pages 1-8. IEEE
14


Reviews:
[{"review_id": "h96p-BtWSuC", "reviewer": "Reviewer_kuwA", "summary": "The paper introduces a new continual learning problem setup, where class vocabulary becomes more fine grained in a continual fashion. Different than classic continual learning this setup allows access to the historical examples. Thus it is not prone to catastrophic forgetting.\n\nThe paper explores several research questions, like when the vocabulary evolves whether to annotate new data or relabel the old data (without collecting new data), how to leverage coarse label (of old data), and whether to finetune the model trained on old data, or train from scratch.\n\nThe paper show that a semi supervised approach, that only requires labelling the new data without relabeling the old data, is almost equivalent to relabeling all the data (both old and new).\n\nThe approach uses the new data to provide pseudo labels for the old data, and use the old data coarse labels to resolve reconcile conflicts between the pseudo fine labels and the true coarse labels.\n", "questions": "\n1. It is not clear in what the new problem setup is different than LPL\n\n\n2. The approach is somewhat similar to [4], which was applied to a different problem setup. I suggest citing it in the related work.\n\n###References\n[4] Goel et al, Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation, CVPR 2022", "limitations": "Ok,\n\nSee my feedback in the weakness section\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n\n* The paper is well written, very easy to read, and provides decent baselines and upper bounds.\n\n* The problem setup is important from a practical standpoint, and novel to the best of my knowledge (I am not an expert in continual learning, or hierarchical learning).\n\n* The suggested approach saves relabeling efforts, which can become quadratic with the number of \"time periods\".\n\n\n### Weaknesses\n\n* Although the problem setup is general, and allowing multiple \"time periods\" (TP), in practice the experiment are only with a single TP. I think this is the major weakness of the paper, because it is not clear how well the approach generalizes, as the labels become more fine grained.\n\n* The approach was only evaluated on two datasets. It would be useful to provide experiments for a different data domain (not vision). \n\n* The problem setup ignores the fact that aside from storage, retraining on the old samples may be very costly from a compute standpoint.  E.g. retraining a model with a scale of Giga samples like a in autonomous driving may cost millions of dollars. There are recent works like [1,2,3] that try to alleviate this problem. I suggest to discuss it in the related work.\n\n* This setup may introduce biases. There may be a distribution shift when collecting more fine grained labels. It would have been beneficial to add this type of bias to the benchmark, and demonstrate how sensitive is the approach to such a shift.\n\n* It is unclear what are the open questions and the next steps. If the approach reaches the upper bound, does it mean that this problem is solved?\n\n* Experiments: There is a better upper bound. Which is utilizing both the old and new labels for the old data \n\n\n### References\n[1] Houlsby et al. Parameter-Efficient Transfer Learning for NLP, ICML 2019\n\n[2] Li et al, Cross-domain Few-shot Learning with Task-specific Adapters, CVPR 2022\n\n[3] Cohen et al, \"This is my unicorn, Fluffy\": Personalizing frozen vision-language representations, ECCV 2022\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VXQtlKLsLF2", "reviewer": "Reviewer_YxYF", "summary": "The paper introduces the problem of Learning with Evolving Class Ontology (LECO), a general case of Class Incremental Learning, where instead of introducing new classes at each time step, we refine existing class labels into more granular ones. The authors ask whether, given incoming data and a fixed annotation budget, it is better to annotate the new data with the new labels or re-annotate the old data. In other words, whether it is better to train a classifier on a smaller but homogenous dataset or a larger one with a mix of old (coarse) and new (fine) labels. Based on experiments on CIFAR and iNaturalist, the authors conclude that by using semi-supervised learning and incorporating the hierarchical taxonomy, it is possible to achieve excellent classification accuracy with a heterogeneous dataset. In particular, by using pseudo-labelling and Learning-with-Partial-Labels, it is possible to get close to the performance of a classifier trained on a homogenous dataset of the same size.", "questions": "In the \"Technical Insights\" section of the introduction, we read that \"Under the LECO protocol of the iNaturalist benchmark (Section 3), (1) labelling new data outperforms relabeling the old: 73.3% vs 70.9% in accuracy;\". I couldn't find these numbers in the result tables, so it is unclear how the label mismatch is handled in this base case. Is the strategy to learn a feature extractor on the old data and fine-tune it on the new data?\n\nIn the \"Benchmarks\" section of Chapter 3, the authors claim that \"using all the labelled data is realistic because, for example, safety-critical systems (e.g., autonomous vehicles) must not trade off recognition accuracy with parsimonious memory buffer.\" While the buffer size is indeed arbitrary, for any practical continual learning system, it has to be *finite*. If we extend the LECO setting to multiple TPs, the buffer size will eventually become an important consideration.\n\nOne of the strategies in Chapter 4 is to \"train an entire network from scratch with random weights.\" The phrasing is unclear here. Does it mean \"train a randomly initialised neural network\"?\n\nAnother strategy in Chapter 4 is to train a classifier on top of a randomly initialised feature extractor. What is the purpose of this? Is it to provide a lower bound on the classifier performance?\n\nIn Section 6.2., the first strategy is described as follows: \"we filter out pseudo-labels that do not align with the ground-truth coarse labels.\" What does filter out mean in this context? Is the data point rejected? Is the coarse label used instead?", "limitations": "The authors correctly identified an essential caveat in their research question: obtaining new data is generally equally or more costly than annotating it. Re-annotation is also usually quicker than annotating from scratch because the annotator can use existing labels to constrain the task to a handful of classes.\n\nOne significant problem mentioned but not explored enough is the assumption that the findings would generalise to multiple TPs. In particular, it would be interesting to see the performance of Learning-with-Partial-Labels with multiple levels of granularity.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is clarity. The structure is good, the writing is easy to understand, and the results are presented in a logical, incremental order. The authors precisely state the research questions and contributions and briefly summarise the conclusions in the introduction, which helps to guide the reader. The paper includes relevant details to reproduce the experiments.\n\nI have no reservations about the quality of the submission. The experimental setup is sound, and the proposed models seem appropriate to tackle the presented problem. The authors evaluate their methods on two different datasets, modified to match the evolving ontology scenario. As mentioned below, the evaluation could be improved by including related work.\n\nThe biggest weakness of the submitted work is originality. While the LECO setting appears novel and could be an interesting type of class-incremental learning, limiting it to two time steps and giving the model access to all the data turns it into fine classification with coarse supervision, a problem tackled by other omitted work, such as:\n\n- A Weakly Supervised Fine Label Classifier Enhanced by Coarse Supervision, Taherkhani et al. 2019\n- Weakly Supervised Image Classification with Coarse and Fine Labels, Lei et al. 2017\n- A Pseudo-Label Method for Coarse-to-Fine Multi-Label Learning with Limited Supervision, Hsieh et al. 2019\n- From Categories to Subcategories: Large-scale Image Classification with Partial Class Label Refinement\n\nThe authors combine existing methods in a novel way, which could be a valuable contribution. However, the lack of comparison to previous approaches makes it hard to judge whether their work advances state of the art and therefore undermines the significance of their findings. My suggestion would be to either extend the work to multiple TPs or focus on the problem of fine classification with coarse supervision: remove the LECO formulation and compare their method with existing work. I think either would make the submission much stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I0D9lbZHJbo", "reviewer": "Reviewer_6YbD", "summary": "This paper introduces a new problem called Learning with Evolving Class Ontology (LECO) where the objective is to learn finer classes over time. The authors have shown that labeling new data with fine-grained annotation is more valuable and proposed to use the techniques from semi-supervised learning and learning with partial labels literature to utilize the old coarsely-grained data. The authors have proposed a benchmarking protocol for LECO on top of two image classification datasets: CIFAR100 and iNaturalist and shown promising results.", "questions": "- How were the levels selected for the iNaturalist-LECO experimental setup?\n\n- The proposed method uses a large validation set. Is the final performance sensitive to the validation set size? Using such a large validation set to search hyperparameters is not very realistic. Were the upper-bound hyperparameters searched in a similar manner? \n\n- What happens if the fine-grained splitting of a corase-grained class is long-tailed. Most of the pseudo-labels for the coarse-grained old class will be assigned to the head fine-grained child class.  \n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Tackles a somewhat new and interesting problem\n\n- The proposed solution is simple and intuitive\n\n- Experiments are extensive and systematic\n\nWeaknesses:\n\n- The technical novelty of the proposed solution is low. This is not a bad thing in general. However, in this particular case, since the proposed problem (LECO) is somewhat similar to the existing continual learning problems and the proposed solution is an ensemble of multiple existing ideas I find this to be a major weakness.\n\n- I find the current problem setup and solution a bit restrictive. It overlooks a lot of practical concerns. (a) assumes a negligible cost for data acquisition, if new data is not available one can only relabel old data. (b) only expects finer labels in subsequent time periods. What happens if a new category is introduced which doesn't have a parent. The pseudo-labeling approach will mistakenly assign previous coarse-grained classes to this newly introduced category. Filtering the pseudo-labels based on class ontology might work in this case but if the data is long-tailed (the new class being a head class) this might remove a lot of old data from training.   \n\n- The findings are not very surprising. If data acquisition cost is ignored and a large portion of data is labeled (which is the case in the current experimental setup) it is expected that a recent SOTA semi-supervised method will be able to reach a performance closed to supervised upper-bound accuracy trained with all data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept