Semantic Prior for Weakly Supervised Class-Incremental Segmentation
Authors: Subhankar Roy
Riccardo Volpi
Gabriela Csurka
Diane Larlus
Venue: Submitted to ICLR 2023 (2023)
Keywords: class-incremental learning
weakly supervised semantic segmentation

Abstract: Class-incremental semantic image segmentation assumes multiple model updates, each enriching the model to segment new categories. This is typically carried out by providing pixel-level manual annotations for all new objects, limiting the adoption of such methods. Approaches which solely require image-level labels offer an attractive alternative, yet, such annotations lack crucial information about the location and boundary of new objects. In this paper we argue that, since classes represent not just indices but semantic entities, the conceptual relationships between them can provide valuable information that should be leveraged. We propose a weakly supervised approach that leverages such semantic relations in order to transfer some cues from the previously learned classes into the new ones, complementing the supervisory signal from image-level labels. We validate our approach on a number of continual learning tasks, and show how even a simple pairwise interaction between classes can significantly improve the segmentation mask quality of both old and new classes. We show these conclusions still hold for longer and, hence, more realistic sequences of tasks and for a challenging few-shot scenario.

Under review as a conference paper at ICLR 2023
SEMANTIC PRIOR FOR WEAKLY SUPERVISED CLASS-INCREMENTAL SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Ilass-incrementll semantic image segmentation assumes multiple model updates, each enriching the model to segment new categories. This is typically carried out by providing pixel-level manual annotations for all new objects, limiting the adop- tion of such methods. Approaches which solely require image-level labels offer an attractive alternative, yet, such annotations lack crucial information about the lo- cation and boundary of new objects. this paper we argue that, since classes rep- resent not just indices but semantic entities, the conceptual relationships between them can provide valuable information that should be leveraged. We propose a weakly supervised approach that leverages such semantic relations in order to transfer some cues from the previously learned classes into the new ones, comple- menting the supervisory signal from image-level labels. We validate our approach on a number of continual learning tasks, and show how even a simple pairwise in- teraction between classes significantly improve the segmentation mask quality of both old and new classes. We show these conclusions still hold for longer and, hence, more realistic sequences of tasks and for challenging few-shot scenario.
INTRODUCTION
When working towards the real-world deployment of artificial intelligence systems. two main chal- lenges arise: such systems should possess the ability to continuously learn, and this learning process should only require limited human intervention. While deep learning models have proved effec- tive tackling tasks for which large amounts curated data as well as abundant computational resources are available, they still struggle to learn over continuous and potentially heterogeneous sequences of tasks, especially supervision is limited.
In this work, we focus on the task of semantic image segmentation (SIS). A reliable and versatile SIS model should be able to seamlessly add new categories to its repertoire without forgetting about old ones. Considering for instance house robot or self-driving vehicle with such segmentation pability, we would like it to be able to handle new classes without having to retrain the segmentation model from scratch. Such ability is the core of continual learning research, the main challenge being to mitigate catastrophic forgetting of what has been previously learned (Parisi et al. 2019).
Most learning algorithms for SIS assume train- ing samples with accurate pixel-level annota- tions, time-consuming and tedious opera- tion. We argue that this is cumbersome and severely hinders continual learning: adding new classes over time should be a weightt process. This is why, here, we focus on the case where only image-level labels are provided (e.g., adding the 'sheep" class comes as easily as only providing images guaranteed to contain at least a sheep). This weakly supervised task is an extremely challenging problem in itself and very few attempts have been made in the context of continual learning (Cermelli al. 2022
Prodiction
Semantic similarty cless "sheep"
Input from model
sheep d
boat lkgg
Semartic embasding
Image
Lagend:
COW
boat
bkg
Our proposed Relation-aware Prior Loss (RaSP) is based on the intuition that pre- dictions from existing classes provide valuable cues better segment new, semantically related classes. This allows reducing supervision to image-level labels for incremental SiS.
Figure 1:

---

Under review as a conference paper at ICLR 2023
Additionally, we argue that tackling a set of SIS tasks incrementally can bring opportunities to learn to segment new categories more efficiently (allowing to move from pixel-level to image-level labels) and more effectively. This can be enabled by taking into account the semantic relationship between old and new classes as humans do. In this paper, we formalize and empirically validate semantic prior loss that fosters such forward transfer. This leads to continual learning procedure for weakly supervised SIS models that leverages the semantic similarity between class names and builds on top of the model's previous knowledge accordingly. If the model needs additionally learn about the 'sheep' class for instance, our loss can leverage the model's effectiveness in dealing with other similar animals, such as cows, and does not need to learn from scratch (see Fig.
We validate our approach by showing that it seamlessly extends state-of of the arr SIS methods. For all our experiments, we build on the WIL.SON approach of [Crmelli et all (202T). buildt itself on standard techniques for weakly supervised SIS (Borenstein & Ullman| 2004 Araslanov & Roth, 2020 We extend it with our Relation-awarr Semantic Prior (RaSP) loss to encourage forward transfer across classes within the learning sequence. It is designed as simple-to-adopl regularizer that measures the similarity across old and new categories and reuses knowledge accordingly.
To summarize, our contribution is threefold. First, we propose RaSP, a semantic prior loss that treats class labels, not as mere indices, but as semantic entities whose relationship between each other matters. Second, we broaden benchmarks previously used for weakly supervised class-incremental SIS to consider both longer sequences of tasks (prior art is limited to 2, we extend to up to 11), and few-shot setting, both with image-level annotations only. Finally, we empirically validate that the steady improvement brought by RaSP is also visible in an extended version of our approach that uses an episodic memory, filled with either past samples or web-crawled images for the old classes. We show that, in this context, the memory does not only mitigate catastrophic forgetting, but also and most importantly fosters the learning of new categories.
2 RELATED WORK
This work lies at the intersection of weakly supervised and class-incremental learning of SIS models. Due to the nature of our semantic prior loss, it also relates to text-guided computer vision.
Weakly supervised SIS. This term (Borenstein & Ullman 2004) encompasses several tasks for which SIS models are trained using weaker annotations than the typical pixel-level labels, such as image captions, bounding boxes or scribbles. Methods assuming bounding box annotations for all relevant objects, (or produced by a pretrained detector) focus on segmenting instances within those
bounding boxes (Dai et al. 2015 Ji Veksler 2021 Song et al, 2019 Kulharia et al., 2020). More
related to our work are the methods that with image-level labels, exploiting classification activation
maps (CAM) as pixel-level supervision for SIS (Zhou et al. 2016b Kolesnikov & Lampert 2016
Roy & Todorovic 2017 Huang et al., 2018 Ahn & Kwak 2018 Araslanov & Roth 2020).
Class-incremental SIS. Under the hood of continual learning (Parisi et al. 2019), class- class-incremenall
learning consists in exposing a model to sequences of tasks, in which the goal is learning new classes.
While most inasemnncremetall learning methods have focused on image classification (Masana et al.
2020 some recent works started focusing on SiS (Cermelli et al. 2020 Michieli & Zanuttigh 202 Douillard et al.] 2021; Maracani et al., 2021 Cha et al.) 2021). Yet, all aforementioned methods assume pixel-level annotations for all the new classes, which requires huge, often pro- hibitively expensive amount of human work. Therefore, weakly-supervised class-incremental SIS has emerged as a viable alternative in the pioneering work of Cermelll et al. (2022), which formalizes the task, and proposes the WILSON method to tackle it. WILSON builds on top of standard weakly supervised SIS techniques, and explicitly tries to mitigate forgetting using feature-level knowledge distillation, akin to the pseudo-labeling approach of PL.P (Douillard et al. 202Tb.
Text-guided computer vision. Vision and language have long history of benefiting from each other, and language, a modality that is inherently more semantic, has often been used as a source of supervision to guide computer vision tasks, such as learning visual representations (Quattoni et al.
2007
Gomez et al., 2017; Sariyildiz et al. 2020 Radford et al., 2021) object detection (Shi et al.
2017
zero-shot segmentation (Zhou et al.) 2016b Bucher et al. 2019; Xian et al. 2019: Li et al.
2020
Back et al. 202T). language-
segmentation (Zha et al.] 2017]]
| Li et al.) 2022; Ghiasi
al.
2022 Xu et al. 2022)
referring image segmentation (Hu et al. 2016 Liu al.
2

---

Under review as a conference paper at ICLR 2023
2022). among others. In some cases, this guiding process requires to create a textual embedding whose associated metrics plays a crucial role. Word2Vec (Mikolov et al.) 2013), Ve (Pennington et al. 2014) and BERT (Devlin et al. 2019) are among the most common. Similarly, our prior semantic loss assumes the availability of such similarity metrics between textual pairs. Others, such as Ghiasi al. (2022): Xu al. (2022; Wang et al. (2022) rely on large pre- trained text-image embeddings such as CLIP (Radford et al. 2021) or ALIGN (Ja et al. (02T] and combine those with pixel-level semantic labels to generalize SIS models for unseen concepts. Yet,
Ding et al. 2021: Wang et al.
none of these approaches has considered a weakly supervised incremental setting.
PROPOSED FRAMEWORK
We develop method for Weakly Supervised Inareennncelentll SIS (WSCIS). The goal is incre- mentally learning to segment instances from new classes by using image-level labels only, avoiding the need for pixel-level annotations. Before detailing our method, we formalize our setting.
Problem setup and notations. De = {(x]. y)) be a dataset for SIS, where x² E "xxx3
represents input image and is tensor containing the |C* [d1-dmmnnioonal one-hot label vectors for each pixel, in a H x W spatial grid, corresponding to a set of C semantic classes. As typical in SIS, the objects that do not belong to any of the foreground classes are annotated as a special background class ( ('bkg') ) included in Co, We refer to Do as the base task and do not make assumptions on its cardinality. Db is used train a base model, generally defined by an encoder E* and decoder Fo,, (E o F):: x + RIx(C",, where = H' x W is a spatial grid - corresponding to the input image size some resized version of and p EE* o F (x) is the set of class prediction maps, where of is the probability for the spatial location i E in the input image x to belong to the class c.
After this base training, we assume the model undergoes sequence of learning steps, as training sets for new tasks become available. Specifically, at each learning step i, the model is exposed to a new set D' = [(x}, 1 containing No instances labeled for previously unseen C³ classes, where 1' E R|C is the vectorized image-level label corresponding an image x'. Note that in each incremental step, only weak annotations (image-leve) labels) are provided for the new classes. This is in sharp contrast with the base task, in which the model is trained with pixel-level annotations.
The goal of WSCIS is to update the segmentation model at each incremental step & in a weakly supervised way, without forgetting any of the previously learned classes We learn the function (E' o FF): x + RTX|>" where y" Ul [C*} U Ch is the set of labels at step t (old and new ones). Note that, in general, we assume that data from previous tasks cannot be stored - that is, there no episodic memory. We revisit this assumption in Sec. 4.2
3.1 THE RELATION-AWARE SEMANTIC PRIOR Loss
In this paper, we propose to leverage the semantic relationship between the new and old classes. We argue that semantic object categories are not independent, i.e,, the new classes C' that are being learned at step may bear semantic resemblance with the old classes from yt-1, seen by the model during previous training steps. For example, the network may have been trained segment instances of the 'cow' class with dense supervision during the base training, and at any arbitrary incremental step the segmentation network can be tasked with learning to segment the 'sheep' class from weak-supervision. Since cow and sheep are closely related species sharing similar attributes (such as being four-legged, furry mammals), the old snapshot of the model FT-I Ft-1 (or, for brevity, (E o 1)) can provide valuable cues to localize the 'sheep' regions in an image labeled as sheep, despite having never seen this animal before (see Fig. D. Guided by this insight, instead of using the old model predictions to only give cues about the old classes, propose semantic-guides prior that uses old model predictions to discover more precise object boundaries for the new ones.
Concretely, step t and using the old model (E F)*-1, for each pixel x we assign the most probable class label y = arg max 3ff from old classes, yielding the label map y". Then, given the set of ground truth imaee-level labels [(x*) = {c|lt = 1} associated with image x', we estimate a similarity map sf between each class 1 in C(x²) and the predicted label map y":
s (Sn (w (()))
3

---

Under review as a conference paper at ICLR 2023
where wc) is a vectorial embedding of the semantic class in semantic embedding space Ω and So is a semantic similarity measure defined between the classes in Q. Different semantic embed- dings can be considered, such as Word2Vec (Mikolov al. 2013). GloVe (Pennington et al. 2014) or BERT (Devlin et al., 2019). These models were trained such that the dot product between em- bedding vectors, Sq, reflects the semantie similarity between given words. For example in Fig. D Sn(w("sheepp (arepp"),cee w('cow") > S1y(w("shepp') (("bkg')), as "sheep" lies closer to "cow" in the se- mantic space than 'background.. In this work, we use BERT (Devlin al. 2019) for all experiments (see comparisons with other embeddings in Supplementary).
Since one of our aims is rely on the boundaries of semantically similar classes, the background class plays crucial role. To ensure not to alter the original predictions made on the background class, we normalize the similarity map such that the score for the 'bkg" class equal to 1:
(I)) exp (w ('bkg') w(1))//
(2)
is a scaling hyperparameter. By exploiting such similarity maps we can convert the image labels into pixel-level label maps 8 (see Figs. and 2
The key question is, how to exploit these similarity maps to improve the learning of new classes? One might be tempted to use Eq. [2) as the only supervisory signal to learn the new classes: this would be sub-optimal, since the learning algorithm also needs to handle cases for which old and new classes are highly dissimilar, or for which the new class region is predicted as background.
To overcome this, build on the weakly supervised SIS literature, which typically relies localizer module trained with weak annotations. Its role is to select regions for each semantic class. It is often based on classification activation maps (CAM), which produce discriminative regions for each class that are then used as (pseudo) pixel-level annotations (Zhou et al., 2016a; Araslanov & Roth 2020). Yet, these methods often fail to provide well-defined maps. To overcome this, the localizer is combined with label propagation (Ahn & Kwak) 2018, Huang et al., 2018)
CRF (Kolesnikov & Lamperi 2016). Here, we assume strongly supervised based model, so we can rely instead on the semantic similarity maps defined above. We argue that they can provide a valuable supervisory signal for the localizer. In concrete terms, we define the following loss:
CRISp(Z, s)
ΣΣ VEI REC
log(o + (1 (x)) log(1 o (z
I
where 2f is the value assigned by the localizer for class at pixel i, is the sigmoid function. Given a generic loss for a localizer LetS (an instance of this loss will be detailed in the next section), can combine the two terms as L Las + ACRASP. Intuitively, our proposed loss serves regularizer that encourages forward transfer from the old classes to the new ones.
We call approach RaSP, which stands for Relation-aware Semantic- for the WSCIS task as it uses the semantic relationships between the old and the new classes as a prior in the semantic loss. This approach is generic and can be combined with any WSCIS method. In the next section we show how to combine it with the current state-of-the the--rt approach.
3.2 FULL INTEGRATION OF RASP
Without loss of generality, we implement our RaSP loss on top of the WILSON framework (Crr- melli et al. 2022). We chose WILSON since it is state of the art and since it relies localizer module AAaslaaov & Roth 2020) to tackle WSCIS and, hence, is a good fit to test our loss.
Background. WILSON is end-to-end method for WSCIS that incrementally learns segment new classes with the supervision of maps generated by a localizer trained with image-level supervi- sion. More specifically, step t, WIL.SON composed of shared encoder E' and a segmentation head Ft - which are both incrementally updated and a localizer head G², trained from scratch for every task. It also stores a copy of the model from the previous task. (E o F)
Given an image x from the current task, y = ((F ((F) E (I is the output produced by the old model. The scores obtained by the localizer, = (G' Ex) E RT*>" are aggregated into oneddimenii vector ý RDD* Each per-class aggregated score de is obtained using
4

---

Under review as a conference paper at ICLR 2023
Semantic similarity maps
Gld classe::
"oow", "aroteriike",
argmaai (K()
"nore" map
B Previous model
Semantic space
"bikee map
"horse" "blkee
knowlodge
RasP
RRSSP
poaling
Lels
Poeee bike
E
G
Irput image
"torse', : bike"
LaFG
New model
J
Figure 2: The RaSP framework. At training step i, the previous model (EoF) F (-1 has been trained to segment cows, motorbikes, and persons among other classes. Presented with an image labeled containing a horse and a bike, the two classes to learn, the old model almost perfectly segments out the horse and the bike, except that their pixels are predicted as "cow' and 'motorbike', respectively. We leverage the semantic relationships between old and new class names to produce similarity maps for the new classes, thus converting the image-level to pixel-Ievel (pseudo) labels.
normalized "Global Weighted Pooling" combined with a focal penalty term (see Supplementary) The score De be seen the likelihood for image x contain semantic class c. This allows training the model with image-level labels using the multi-label soft-margin loss:
Las(E,1)
1 |e
1' log(((())) + Σ (1-19) log(1 - -o()))
(4)
GEC
GEC
Note that, although the localizer outputs [)* -dimensioo vector, at task we are only provided with images and their image-level annotations for the new classes. Therefore, the sum in Eq. [4) is computed only over the new classes. In order to train the localizer for the old classes and prevent encoder from shifting towards the new classes and forgetting the old ones. ones, CCrmelli et al. 2022) pro- pose distill knowledge from the old model, by adding two knowledge distillation losses. The first one, LKDE. computes the mean-squared error between the features extracted by the current encoder E' and those extracted by the previous one Et-1, The second distillation loss LKDI. encourages consistency between the pixel-wise scores for old classes predicted by the localizer (E o G)* and those predicted by the old model (E F) (see details Supplementary).
Finally, WILSON combines the localizer output with the old model to generate the pseudo- supervision scores o that are used to update the segmentation module (E o F)', following
LSEG (p. q)
Σ
Σ RET
of log(o + - 7 log(1 (pf)) CE
(5)
where p (x) are the predictions from the main segmentation head and q is the supervisory signal containing i) the old model's predictions for the old classes, ii) the localizer's 's refined scores for the new classes (see Supplementary for more details) and iii) the minimum between the old
model and the localizer scores for the background. The final objective optimized by WILSON is the non-weighted sum of the different loss terms defined above, Lw LCLS + LXDI. LKDE LSEG-
maalizer-based approach designed for weakly supervised SIS, constitutes good starting point integrate and test our proposed semantic prior. Therefore, complement WILSON's ' losses with our loss introduced in Eq (3. Our semantic prior works seamlessly with WILSON's localizer without the need for any ad-hoc architectural changes. Eq. (3) simply requires as input the output from the localizer z E C) and ii) the semantic similarity maps between new and old classes, obtained via Eqs. ( and Concretely, with such definitions, simply optimize the joint loss Lj = Lw + XLRSPP- The hyperparameter controls the strength of our prior loss, which as a regularizer fostering forward transfer from the old to the new classes.
Extending WILSON with RaSP. Since WILSON exploits
prior loss also be applied together with WILSON losses, and
5

---

Under review as a conference paper at ICLR 2023
Input
GT
(F o E)
(x,)
g/:
RaSP
WIL.ONN
Figure 3: Visualizations. Qualitative figures from the multi-step overlap incremental protocol on 10-2 VOC. From left to right: input image, GT segmentation overlayed, predicted segmentation from old model, semantic similarity map corresponding to the image label (dog / sheep) computed between this label and old classes, predicted segmentation obtained with RaSP and with WILSON. Semantic similarity maps displayed in OpenCV colormap HOT (low
high similarity).
EXPERIMENTS
Implementation details. To be comparable with prior work (Cermelli et al. 2022 we use DeeplabV3 (Chen al. (017) SIS network with either ResNet- 101 (He 2016) (for VOC) or Wide-ResNet-388 (Wu et al. 2019) (for COCO). Following WILSON, the localizer is composed of 3 convolutional layers, interleaved with BatchNorm (lfffe 202T) and Leaky ReLU layers. For each step. we train the model with SGD for 40 epochs using batch size of 24. Since the localizer can produce noisy outputs in early training, we do not use LSGG for the first 5 epochs (the DeeplabV3 weights remain frozen). In 2 Sn is the dot product and wyy is 1.2 normalized. We set T 5 and X = and follow the values suggested by Cermelli et al. (2022 for all other hyperparameters. See Supplementary for different values for 7 and A.
Incremental settings. Our experiments consider several incremental learning scenarios. We name experiments following the notation N,,N,, to indicate that we first learn with pixel-level supervision from N₂ base classes, and then sets of N₂ new classes at a time, with image-level supervision only. Given total number of classes N, the number of tasks is (N No)/N, 11
Single-step settings consider only one incremental learning phase. For instance, in the 15-5 setting (see Tab. D. we first train the model on the 15 base classes and then learn the remaining 5 new classes in a single incremental step (bringing the total number of classes to 20). Multi-step settings add new classes to the model in multiple sequential steps. The 10-2 setting, for instance, considers 10 base classes and 5 incremental steps which learn 2 new classes at a time. In each table, we indicate results for base classes 1-N, and for the new ones as (N, 1)-N.
Each incremental setting can be designed in two ways: i) Overlap, if all training images for a given step contain at least one instance of new class, but they can also contain previous even future classes; ii) Disjoint, if each step consists of images containing only new or previously seen classes, but never future classes. In both protocols, annotations are available for the new classes only. We argue that multi-step protocol with overlap is the most realistic setting. That said, we also consider some other options, especially when it facilitates comparison with prior work.
Datasets. Following Cermelli et al. (2022). we experiment mostly with the Pascal VOC dataset (Ev- eringham et al., 2010). VOC consists of 10, 582 training and 1, 144 validation images covering 20 semantic categories. We divide the 20 categories into base and new classes according to the dif- ferent protocols described above. We further consider the COCO-to-VOR protocol, where the base classes are the 60 categories from the COCO dataset Lin et al. (014g (which has 164k training and 5k validation images from 80 semantic categories) that are not available in VOC. In the incremental steps, learn new classes from VOC categories following the same protocol above (e.g.. 60-5 a 5-step protocol where the 20 VOC classes are learned in incremental steps). In this case, in each step we evaluate the validation sets of both COCO and VOC.

---

Under review as a conference paper at ICLR 2023
Shoup
Shopp
miou (50
Pursen
+37.
Deg
Motoibkk
Potood
Method
RX5P
Horse
Dining table
-20
20
mioi gain
40
60
80
100
Figure 4: Left: Per-task and per-step mloU for the 10-2 VOC multi-step overlap incremental setting.
Right: Per class gain/drop of RaSP w.r.t WILSON, evaluated for each class in the step it was learned
Method
Supervision
15-5 (2 tasks)
10-10 (2 tasks)
1-15
16-20
All
1-10
11-20
All
Fine-Tuning
Pixel
12.5
36,9
18.3
7.8
58.9
32.1
LWF & Hoiem 2016
Pixel
67.0
61.0
70.7
63.4
67.2
PLOP Douillard al 202l)
Pixel
75.7
51.7
70.1
69.6
62.2
67.1
SDR (Michell & Lanutigh 202
Pixel
75.4
52.6
69.9
70.5
63.9
67.4
RECALL Marac
2021
Pixel
67.7
54.3
65.6
66.0
58.8
63.7
CAM (Zhou et all 6b]
Image
69.9
25.6
59.7
70.8
44.2
58.5
SEAM wane
2020
Image
68.3
31.8
60.4
67.5
55.4
62.7
SS (Araslanov & Koln 2020
Image
72.2
27.5
62.1
69.6
32.8
52.5
EPS Lee el al 2041
Image
69.4
34.5
69.0
57.0
64.3
WIL.SON (Cemnellee al.
2022
Image
74.2
41.7
67.2
70.4
57.1
65.0
WILSONF (Cermelli et al. 2022
Image
76.3
44.1
69.3
71.4
56.1
64.9
76.2
47.0
70.0
72.3
57.2
65.9
RaSP (Ours)
Image
(10.1%)
(16.6%) (1100%
(11.3%)
(11.6%)
(.55%
10-5 (3 tasks)
10-2 (6 tasks)
1-10
11-20
All
1-10
11-20
All
WILSONF (Cermelli et al. 2022)
Image
66.8
46.5
58.1
38.7
22.4
32.5
68.8
49.1
60.4
44.5
28.4
38.6
RaSP (Ours)
Image
(13.0%) (15.6%) (14.0%) (115.0%) (126.8%) (118.8%)
Table 1: VOC results. The mIoU (in %) scores for both single-step (2 tasks, top) and multi-step (bottom) overlap incremental settings on VOC. For each experiment, the three different columns indicate performance base, new and all 21 classes (including background), respectively. For RaSP (Ours), we further report the relative gain/drop performance (in %) w.r.t. WILSONF.
Evaluation metries. We evaluate all models using the standard mean Intersection over Union (mloU) (Everingham et al., 2010). For all scenarios in Tabs. S we report the mloU scores eval- uated after the last incremental step. Each time we report values: for base task (considering results on the base classes excluding the background), for the subsequent ones (new classes added during the incremental steps) and finally considering all the classes including the background (All).
4.1 MAIN RESULTS
Comparison to the state of the art. We compare our proposed RaSP with several state-of-the-an class-incremental learning methods that use either pixel-level or image-level annotations in the incre- mental steps. We mainly focus on WSCIS methods, which allow fair comparisons. Pixel-supervised methods are interesting but not comparable as they use prodigious amount of eutra-supervision. The best performing method with level and pixel-level supervision are respectively bolded and underlined in Tables. Since Cermelli et al. (2022) tested WILSON only for single-step incre- mental settings, we ran experiments the other settings using the official implementation provided For comparability,
by the authors (attps://gitaub com/ experiments on single-task settings. "WILSONT" indicates our reproduced results while "WIL- SON" corresponds to the original numbers from the paper. We further report in tables the relative gain/drop in performance (in %) of our RaSP w.r.t. WILSONF, within brackets.
also re-ran
show results for both single-step and multi-step incremental settings, VOC, using the overlap protocol. We observe that our RaSP outperforms WILSON in almost all the considered settings. In particular, the relative gain (in %6 w.r.t. WILSON grows wider as the number
Results. In Tab.

---

Under review as a conference paper at ICLR 2023
60-20 (2 tasks)
Method
Supervision
COCO
VOC
1-60
61-80
All
AlI
Fine-Tuning
Pixel
1.9
41.7
12.7
75.0
LWF & Hoiem 2016
Pixel
36.7
49.0
40.3
73.6
(Michell & ann 2019
Pixel
37.0
43.9
39.3
68.7
PLOF (oouiiard et u. 2021
Pixel
35.1
39.4
36.8
64.7
CAM
(Zhou et al. 2016b
Image
30.7
20.3
28.1
39.1
SEAM
Image
31.2
28.2
30.5
48.0
SS
(Araslanoy & Kotn 2120
Image
35.1
36.9
35.5
52.4
EPS
Image
34.9
38.4
35.8
55.3
WILSON (emelll el al. 2022
Image
39.8
41.0
40.6
55.7
WILSON Cermelli ct al.
2022
Image
41.1
41.0
41.6
54.8
41.1
40.7
54.4
RaSP (Ours)
Image
(0.0%)
(10.7%%
(0.0%)
(10.7%)
60-5 tasks)
60-2 tasks)
COCO
VOC
COCO
VOC
1-60
61-80
All
All
1-60
61-80
All
All
WILSONF
Cermelli et al. 2022
Image
30.1
28.0
30.2
42.0
10.2
14.8
12.2
24.1
33.0
28.2
32.5
41.7
14.6
16.5
15.9
26.9
RaSP (Ours)
Image
(L..75
11.6%)
Table 2: COCO-to-VOR VOC results. The mloU (in %) scores for both single-step (2 tasks, top) and multi-step (bottom) overlap incremental settings on COCO-to-VOC VOC. For each experiment, the first three columns indicate performance base, and all classes (81 including background) com- puted on COCO, respectively; last column indicates performance on all classes for VOC.
of incremental steps increases, with RaSP achieving +26.8% relative improvement over WILSON in new class performance, in the 10-2 setting. Not only our semantic-prior loss improves new class performance but also it leads to 15% lesser forgetting w.r.t. WILSON. We provide few qualitative examples in Fig. E showing how the semantic maps aid the final segmentation.
Fig. 3 (left) shows the mIoU scores per task and per step for the 10-2 VOC overlap setting, indicating which classes are learned at each step (for WILSON and RaSP). This plot shows how our method consistently improves
WILSON throughout the learning sequence. In Fig. (right), we report RaSP's per-class relative percentage improvement w.r.t. WILSON, computed at each step.
We show results for the COCO-to VOC benchmark (overlap protocol) Tab. 2 RaSP performs comparably with WILSON in the 2-task setting, but outperforms it when more increments and fewer classes per increment are considered from 60-5 tasks) to 60-2 (11 tasks). We observe both improvements for new classes and reduced forgetting on the old ones.
4.2 LEARNING ONE NEW CLASS AT TIME
A limitation of state-of ooffthe-an art approaches for WSCIS is their underwhelming behavior when learn- ing one new class at time: the model fails to learn and undergoes drastic forgetting. This is due to the fact that Eq. is optimized for single positive class: the lack of negative classes leads to gross overestimation of the foreground, causing the localizer to provide poor pseudo-labela to the segmentation head, with negative effect on old classes well. We show in Tab. (top-half) results of WILSON and RaSP for two single-class incremental settings (15-1 and 10-1), using VOC. Both methods struggle with learning new classes, yielding poor performance compared pixel- supervised methods. These fully-supervis methods can learn the new classes better since their annotations are composed of both positive foreground-object pixels and negative background pixels.
episodic memory. To circumvent this issue we store a small number of images from base classes in fixed-size memory M. Intuitively, samples in memory help the localizer by provid- ing negative samples. We show in Tab. (bottom-half) that storing little 100 past samples from the dataset significantly improves learning new classed for both WILSON and RaSP, with RaSP M outperforming WILSON M (28.3% 5% vs 20.8% in the 15-1 setting). Unsurprisingly. it also helps retaining performance on the base classes. Similar observations hold for the 10-1 setting.
A solution:
External data as alternative. Inspired by RECALL MMaraaai et al. 2021), consider the option of retrieving samples for base classes from external sources. We define this memory as Men- Concretely, retrieve 100 samples per base class from ImageNet (by creating mapping with VOC
8

---

Under review as a conference paper at ICLR 2023
15-1 (6 lasks)
10-1 (11 tasks)
Method
Supervision
T-15
16-20
All
1-10
11-20
All
wel memory
ILT (Michieli & Zanuttigh 2019
Pixel
4.9
7.8
5.7
16.5
1.0
9.1
MiB (Cermelli al. 2020
Pixel
35.1
13.5
29.7
15.1
14.8
15.0
WIL WONTT (Cermelli 2022
Image
0.0
2.3
0.6
0.0
0.2
0.1
RaSP (Ours)
Image
17.7
0.9
13.2
2.0
0.7
1.3
WILSONI + +M
Image
61.5
20.8
52.5
33.4
24.6
30.0
WP menory
RaSP (Ours) + M
Image
63.3
28.3
56.0
38.9
30,9
36.9
WILSONI MMax
Image
75.7
32.9
65.9
66.8
34.9
52.3
RaSP (Ours) Mad
Image
75.7
35.2
66.6
66.8
39.1
54.4
RECALL (Web) (Maracami al. 2021
Pixel
67.8
50.9
64.8
65.0
53.7
60.7
Table 3: Effect of memory. Results on single-class multi-step overlap incremental setting on VOC. M and Mexi indicate memories of previously seen external samples, respectively.
Method
Supervision
1-15
VOC (S-shot)
16:20
HM
1-15
VOC (2-shot)
16-20
HM
0:00
COCO (5-shot)
61-80
HM
0.60
COCO (2-shot)
61-80
HM
Fine-Tuning
Pixel
55.8
59.1
19.7
29.5
41.6
12.3
19.0
12.4
WI Qi et 2018
Pixel
63.3
21.7
32.3
63.3
19.2
29.5
43.6
8.7
14.6
44.2
7.9
13.5
AMP Stam al. 2019
Pixel
51.9
54.4
11.0
16.7
14.2
MiB
crmell el u. 4020
Pixel
65.0
28.1
39.3
63.5
12.7
21.1
44.7
11.9
18.8
44.4
6.0
10.6
PIFS
crmelli
Pixel
60.0
42.8
60.5
26.4
15.7
23.0
17.5
WILSONT (Cermelli el al 2022
Image
64.1
20.3
31.1
63.3
10.2
17.6
45.0
5.8
10.3
43.6
1.9
3.6
32.0
63.5
10,7
5.6
RaSP
Image
(10.5%X
2.9%
X10.3%
4..9%
4.0%%
(1..2%
(10.2%
Table 4: Few-shot results. The mIoU (in %) scores for the single-step (2 tasks) incremental few- shot SiS settings the PASCAL-5' and COCO-20' benchmarks, for 5-shot and 2-shot cases. We show the average results over the 4 folds as in (Cermelli et al. 202Ip. For each experiment, columns report performance old classes, new classes, and the Harmonic-Mean (HM) of the two scores.
classes). This further improves both WILSON Mext and RaSP Mex compared the previous episodic memory. RECALL performs better on new classes, but i) relies on pixel-level supervision and ii) uses significantly more web-crawled images; therefore, is not directly comparable.
4.3 CLASS-INCREMENTAL FEW-SHOT SEGMENTATION
We compare RaSP and WILSON the task of Incremental Few-Shot Segmentation (iFSS) (Cr- melli et al. 202T where the model learns incrementally from only few images. This a challenging setting, only tested so far with pixel-level supervision. Here, add the challenging constraints that the new training images are only weakly annotated. Following Cermelli et al. (202T), we consider 4 folds of classes for PASCAL-5 and the 4 folds of 20 classes for COCO-20' where each fold in turn is used as incremental setting with the other classes forming the base task.
Tab. reports average results over the 4 folds per-fold (per-fold results in Supplementary). Bottom lines contain results obtained by WILSON and RaSP; as expected, in the case of COCO-20' both methods perform poorly, which not surprising as even the strongly supervised methods (top lines) have difficulties to learn the new classes. On the other hand, on PASCAL-5 not only RaSP consis- tently outperforms WILSON, but in the 5-shot case it also outperforms or performs on par with some of the strongly supervised methods. Finally, we can appreciate that the performance of RaSP on the base classes remains comparable or often outperforms most of the strongly supervised methods, where the higher performance on the new classes tends to come with more severe forgetting.
5 CONCLUSIONS
We proposed a new method for Weakly Supervised Class-Incremental Semantic Image Segmenta- tion. Motivated by the observation that new classes to be added to the model often bare resemblance with the old ones that it already knows, we designed Relation-aware Semantic Prior loss (RaSP) that fosters forward transfer. It leverages semantic similarity between old and new class names in a loss that guides the learning of new categories. We validated idea broad variety of bench- marks. In particular, we demonstrated that our method is resilient to unexplored and challenging scenarios which increase the number of tasks and reduce data availability for new classes.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation. CVPR, 2018.
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2020.
Donghyeon Baek, Youngmin Oh, and Bumsub Ham. Exploiting Joint Embedding Space for Gen- eralized Zero-Shot Semantic Segmentation. In ICCV, 2021.
Eran Borenstein and Shimon Ullman. Learning to Segment. In ECCV, 2004.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-Shot Semantic Segmenta- tion. In NeurlPS, 2019.
Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, and Barbara Caputo. Mod- cling the Background for Incremental Learning in Semantic Segmentation. In CVPR, 2020.
Fabio Cermelli, Massimiliano Mancini, Yongqin Xian, Zeynep Akata, and Barbara Caputo. Prototype-based Incremental Few-Shot Semantic Segmentation. In BMVC, 2021.
Fabio Cermelli, Dario Fontanel, Antonio Tavera, Marco Ciccone, and Barbara Caputo. Incremental Learning in Semantic Segmentation from Image Labels. CVPR, 2022.
Sungmin Cha, Beomyoung Kim, YoungJoon Yoo, and Taesup Moon. SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremes Learning. In NewrIPS, 2021.
Liang-Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 2017.
Jifeng Dai, Kaiming He, and Jian Sun. BoxSup: Exploiting Bounding Boxes to Supervise Convolu- tional Networks for Semantic Segmentation. In ICCV, 2015.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-traine Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL HLT, 2019.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-Language Transformer and Query Generation for Referring Segmentation. In CVPR, 2021.
Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. PLOP: Learning without For- getting for Continual Semantic Segmentation. In CVPR, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling Open- Spen-Vocabulary Image Segmenta- tion with Image-Level Labels. In ECCV, 2022.
Spyros Gidaris and Nikos Komodakis. Dynamic Few-Shot Visual Learning without Forgetting. In CVPR, 2018
Lluis Gomez, Yash Patel, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In CVPR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from Natural Language Ex- pressions. In ECCV, 2016.
Zilong Huang, Xinggang Wang, Jiasi Wang. Wenyu Liu, and Jingdong Wang. Semantic Segmentation Network with Deep Seeded Region Growing. In CVPR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Sergey loffe. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch- Normalized Models. In NeurIPS, 2021.
Zongliang and Olga Veksler. Weakly Supervised Semantic Segmentation: From Box to Tag and
Back. In BMVC, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-LLanguage Representation Learn- ing With Noisy Text Supervision. In ICML, 2021.
Alexander Kolesnikov and Christoph H. Lampert. Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation. In ECCV, 2016.
Viveka Kulharia, Siddhartha Chandra, Amit Agrawal, Philip H. S. Torr, and Ambrish Tyagi. Box2Seg: Attention Weighted Loss and Discriminative Feature Learning for Weakly Supervised Segmentation. In ECCV, 2020.
Seungho Lee, Minhyun Lee, Jongwuk Lee, and Hyunjung Shim. Railroad is not train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF conference computer vision and pallern recognition, pp. 5495-5505, 2021.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven
semantic segmentation. In ICLR, 2022.
Peike Li, Yunchao Wei, and Yang. Consistent Structural Relation Learning for Zero-Shot Seg-
mentation. In NeurIPS, 2020.
Zhizhong Li and Derek Hoiem. Learning without Forgetting. ECCV, 2016.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, and Alan Yuille. Recurrent Multimodal Interaction for Referring Image Segmentation. In ICCV, 2017.
Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. RECALL: Replay-Based Continual Learning in Semantic Segmentation. In ICCV, 2021.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classifi- cation. arXiv preprint arXiv:2010.15277, 2020.
Umberto Michieli and Pietro Zanuttigh. Ineremental Learning Techniques for Semantic Segmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Umberto Michieli and Pietro Zanuttigh. Knowledge Distillation for Incremental Learning in Se- mantic Segmentation. CVIU, 205, 2021a.
Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1114-1124, 2021b.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. 1rXiv:1301.334 2013.
German Parisi, Ronald Kemker. Jose L. Part, Christopher Kanan, and Stefan Wermter, Continual Lifelong Learning with Neural Networks: A Review. Neural Networks, 113:54-71, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In In EMNLP, 2014.
Hang Qi, Matthew Brown, and David G, Lowe. Low-shot Learning with Imprinted Weights. In CVPR, 2018.
11

---

Under review as a conference paper at ICLR 2023
Ariadna Quattoni, Michael Collins, and Trevor Darrell. Learning visual representations using im- ages with captions. In CVPR, 2007.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Anirban Roy and Sinisa Todorovic. Combining Bottom-Up Top-Down, and Smoothness Cues for Weakly Supervised Image Segmentation. In CVPR, 2017.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020.
Miaojing Shi, Holger Caesar, and Vittorio Ferrari. Weakly supervised object localization using things and stuff transfer. In ICCV, 2017.
Mennatullah Siam, Boris Oreshkin, and Martin Jagersand. AMP: Adaptive Masked Proxies for Few-Shot Segmentation. In ICCV, 2019.
Chunfeng Song. Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven Class-wiss Region Mask- ing and Filling Rate Guided Loss for Weakly Supervised Semantic Segmentation. CVPR, 2019.
Yonglong Tian, Yue Wang, Dilip Krishman, Joshua B. Tenenbaum, and Phillip Isola. Rethinking Few-shot Image Classification: A Good Embedding All You Need? In ECCV, 2020.
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivari- ant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12275-12284, 2020.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. CRIS: CLIP-Driven Referring Image Segmentation. In CVPR, 2022.
Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90:119-133. 2019.
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic Pro- jection Network for Zero- and FeweLabel Semantic Segmentation. CVPR, 2019.
Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. ViTu Semantic Segmentation Emerges from Text Supervision. In CVPR, 2022.
Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open Vocabulary Scene Parsing. In ICCV, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, 201
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016b.
12


Reviews:
[{"review_id": "x5X9UiZvdY5", "reviewer": "Reviewer_PPnr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a weakly-supervised incremental semantic segmentation strategy, which starts from a base learning task with fully-annotated images and then incrementally learns novel classes with image-level labels only. To tackle this problem, it introduces a pseudo label generation method for the weakly supervised semantic segmentation (WSSS) at each incremental stage, which infers the pseudo labels of the new classes based on the model prediction of the old classes and the semantic similarity between new & old categories in a word embedding space. The proposed WSSS loss is then integrated into an existing weakly-supervised incremental semantic segmentation framework,  WILSON,  in order to build a full pipeline.  The authors evaluate the method on the PASCAL VOC dataset and COCO-to-VOC benchmark with comparisons to prior methods.   ", "summary_of_the_review": "The proposed weakly supervised ISS method seems less convincing due to the limitations on its novelty and pseudo-label generation, mixed performance, and the lack of sufficient experimental evaluation. ", "strength_and_weaknesses": "Strengths:\n- The problem setup of this work, which requires only weak annotation in incremental semantic segmentation, is useful in practice but less explored. \n- The idea of exploring semantic similarity for forward transfer seems novel for the task of weakly-supervised incremental segmentation. \n- The paper is mostly well-written and easy to follow.    \n\nWeaknesses:\n- The overall novelty of the proposed framework is limited. It heavily relies on the existing framework WILSON and the main contribution is an improved WSSS loss integrated within each incremental step. \n- The assumption on the background class is problematic for incremental segmentation. This work seems to assume a static background class, which is not true in this task. Due to this background drifting issue, the proposed pseudo label generation strategy seems less effective: In equation (2), the bkg class would have a high similarity to previous bkg class which includes novel classes. This would largely suppress the new classes and conflict with the CAM loss. To make this work, it seems to require a good base segmentation network to generate foreground masks, which is difficult for the incremental setup. \n- The experimental results are mixed or marginally improved compared to the WILSON framework. On some task settings of the VOC benchmark, the overall improvements are around 1% and on some task settings of the COCO-to-VOC benchmark, the proposed method is inferior to the baseline. \n- The experimental evaluation settings are limited. All of them start from a base learning task with many classes (15 or 60) and incrementally learn a smaller number of classes (5 or 20). It should be evaluated on more challenging settings with fewer base classes and more novel classes. Also, the evaluation on larger datasets in ISS, such as ADE, should be added.     ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "emiW9cvUCbm", "reviewer": "Reviewer_AMCk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new loss for weakly supervised (image-level labels) class-incremental segmentation that enables taking into account semantic relations between past classes and new classes. The proposed approach can be integrated to existing semantic image segmentation approaches that used a localizer module. In the paper, they integrate the loss to the recent WILSON framework. Experiments on Pascal VOC and COOC-to-Voc with different incremental settings are promising and show the interest of the approach.\n\n", "summary_of_the_review": "The paper tackles the relevant and important weakly supervised (image-level labels) class-incremental segmentation problem. It proposes to integrate a semantic loss term into an existing framework that improves the state-of-the-art but the novelty of the proposed approach is only marginal and important component of the proposed approach such as the notion of semantic priors is not deeply studied. ", "strength_and_weaknesses": "**Strengths** \n\n+ Weakly supervised class-incremental segmentation is very relevant and important and only a few approaches have been proposed in the literature\n+ The idea of using semantic relations between past classes and new classes makes sense.\n+ The paper is easy to follow and well-structured.\n+ Experiments and comparison to the state of the art are promising.\n\n**Weaknesses** \n\n+ My main concern is related to the novelty of the approach which just consists in adding new semantic losses in the WILSON framework and is thus marginal.\n+ Moreover, the choice and design of the semantic losses are very simple and could be more discussed. For instance, how to take into account stronger relations between classes than just embedding similarity ones? For instance, in semantic segmentation part-whole relations or spatial relations between classes could also be of great interest and the proposed approach can not differentiate between the different types of relations.\n+ Since the claim of the paper is to use conceptual relations between past and new classes, I would appreciate more experiments focusing on these specific points: for instance a setting with explicit relations between the different incremental steps and various configurations: specialization relations, generalization relations, mereo-topological relations... \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KJc5R6rEVNw", "reviewer": "Reviewer_RkMT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed approach address the problem of class-incremental for weakly supervised semantic image segmentation. The goal of this paper is to leverage the semantic relations among the labels to improve the weakly supervised learning for semantic segmentation. The idea is to generalize the WILSON approach for weakly semantic image segmentation. A new loss relation aware semantic prior loss (RASP) is developed to encourage the forward transfer across classes within the learning sequence. The proposed loss is validated in continuous learning setting.", "summary_of_the_review": "On the positive side, I think the proposed approach has good spirit as it has shown clear improvements on class-incremental setting for weakly semantic image segmentation. The proposed approach has also touched the drawbacks of the WILSON framework, and added the new loss RASP which encourages the forward transfer across classes within the continuous learning setting. However, the paper does not seem to show clear improvements. The aggregated metric improvements on the benchmarks do not guarantee that the improvements are from the proposed RASP loss. There are other technical design choices are not well justified.\n", "strength_and_weaknesses": "Pros\n\nThe approach naturally generalized the prior works on image-classification to the weakly semantic image segmentation, which is novel and inspiring. I think approaching the problem from the loss perspective is reasonable.\n\nThe proposed approach seems to be working well on the proposed settings. The proposed approach shows good results in the settings of effect of memory and few-shot. \n\nCons\n\nThe proposed approach is relative not incremental on top of the WILSON project. As the main idea in this paper is to add the RASP loss on top of the WILSON framework. However, it does not seem that the additional RASP loss adds the big advantages to the WILSON framework. More experiments should be provided to justify if the additional useful signals are provided from the localizer. Specifically, we need to see improvements of the slice and dice metric improvements of the new classes once we have applied the RASP loss and use the information from the old class. The Baseline in that setting is that we are not using RASP loss, but just run CRF-like approach. \n\nAnother question is about the RASP loss. There is no guarantee that the localizer for old tasks is providing the positive impact to the new tasks, unless we have made some careful choice on the old tasks and new tasks, which is not possible in the real-world use cases. How would one use the proposed approach in practice?\n\nOne design choice needs to be justified. In particularly, the proposed approach chooses to treat the new class region as background and build the proposed approach on a localizer module trained with weak annotations. In contrast, there are multiple approaches in literatures in salient region detection that shows promising results in segmenting the background, and plenty of the approaches using super-pixels. It is not clearly why the proposed approach pick up the route, which seems to be bias towards to give better thing-like object instance segmentation. The proposed technical choice might not work well on the stuff-like objects instances, which has extended boundaries. These are missed out in the discussion.\n\nThe experiments are mostly focus on the easy datasets such as Pascal and COCO. For the proposed to work, I would like to see how the proposed approach work on a more challenging dataset. The simulated experiments on new and old classes seem to be cherry picked.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CPgQmvYTQFw", "reviewer": "Reviewer_paY4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a weakly supervised approach that leverages semantic relations to transfer the previously learned semantic class knowledge into the new class. The main motivation is the similar class often has closer distance in the feature space. Extensive experiments are conducted to validate this method.", "summary_of_the_review": "Overall, this is a good paper with a novel class incremental learning that using semantic priors. Based on the strengths and the weakness, currently, the reviewer suggests a positive rating to this paper.", "strength_and_weaknesses": "Strength:\n1) The idea of using the semantic distance as a guidance is straightforward and has been verified effective. \n2) The experimental evaluation is sufficient and reasonable.\n\nWeakness:\n1) The proposed method could not deal with the case that new class has rare similarity with existing classes. In addition, how to calculate the semantic distance is not clear and need more discussion.\n2) The overall method is based on WILSON, which also weaken the contribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject