GAPS: Few-Shot Incremental Semantic Segmentation via Guided Copy-Paste Synthesis
Authors: Ri-Zhao Qiu
Peiyi Chen
Wangzhe Sun
Yu-Xiong Wang
Kris Hauser
Venue: Submitted to ICLR 2023 (2023)
Keywords: continual learning
incremental learning
incremental segmentation
few-shot learning

Abstract: Few-shot incremental segmentation is the task of updating a segmentation model, as novel classes are introduced online overtime with a small number of training images. Although incremental segmentation methods exist in the literature, they tend to fall short in the few-shot regime and when given partially-annotated training images, where only the novel class is segmented. This paper proposes a data synthesizer, Guided copy-And-Paste Synthesis (GAPS), that improves the performance of few-shot incremental segmentation in a model-agnostic fashion. Despite the great success of copy-paste synthesis in the conventional offline visual recognition, we demonstrate substantially degraded performance of its naive extension in our online scenario, due to newly encountered challenges. To this end, GAPS (i) addresses the partial-annotation problem by leveraging copy-paste to generate fully-labeled data for training, (ii) helps augment the few images of novel objects by introducing a guided sampling process, and (iii) mitigates catastrophic forgetting by employing a diverse memory-replay buffer. Compared to existing state-of-the-art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up to 80%. More notably, GAPS maintains good performance in even more impoverished annotation settings, where only single instances of novel objects are annotated.

Under review as a conference paper at ICLR 2023
GAPS: FEW-SHOT INCREMENTAL SEMANTIC SEG- MENTATION VIA GUIDED COPY-PASTE SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot incremental segmentation the task of updating segmentation model, as novel classes are introduced online overtime with small number of training images. Although incremental segmentation methods exist in the literature, they tend to fall short in the few-shot regime and when given partially-annotace train- ing images, where only the novel class is segmented. This paper proposes a data synthesizer, Guided copy-And-Pass Synthesis (GAPS), that improves the perfor- mance few-shot incremental segmentation in a model-agnostic fashion. Despite the great success of copy-paste paste synthesis in the conventional offiine visual recog- nition, we demonstrate substantially degraded performance of its naive extension in our online scenario, due to newly encountered challenges. To this end, GAPS (i) addresses the partial-annotation problem by leveraging copy-paste to generate fully-labeled data for training, (ii) helps augment the few images of novel objects by introducing guided sampling process, and (iii) mitigates catastrophic forget- ting by employing a diverse memory-rellay buffer. Compared existing state-of- the-art art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up 80%. More notably, GAPS maintains good performance in even more impoverished annota- tion settings, where only single instances of novel objects are annotated.
INTRODUCTION
Incremental segmentation is an important capability for open-world AI systems. For example, con- sider housekeeping robot that has been trained to segment common household objects, but once deployed in a user's home it encounters previously unseen type of furniture. For such practical ap- plications, incremental segmentation would be capable of expanding the set of recognized classes to contain the new object. There are few desired properties of incremental segmentation algorithms to operate under these scenarios. First of all, the algorithm should be equipped with few-shot learn- ing capability, which means that the algorithm can benefit from few one image provided by user rather than requiring hundreds of images annotated offline by professional annotators. Second, providing full segmentation annotation of an image is time-consuming. To avoid causing substantial burdens for untrained users, the algorithm needs to be trainable with parttally-annotated images where only novel classes are annotated.
A few attempts have been made by recent works (Cermelli et al., 2020; Cha et al., 2021; Douillard al., 2021: Zhang et al., 2022; Yan al., 2021) on non-few-shot incremental segmentation to inves- tigate learning with partially-annotated images, which is termed semantic background shift (Cermelli al., 2020). Background shift describes a challenge unique to incremental semantic segmentation where classes that are not in the current learning step are assigned 'background' labels, which pro- hitbits direct end-to-end training. Recent work uses either modified loss (Cermelli al., 2020; Zhang et al., 2022) or pseudo-labeling (Cha et al., 2021; Douillard et al., 2021; Yan et al., 2021) as proxies to train on partially-annotatee strate good performance under the non-few-shot settings, they rely on rich annotations and fall short when only limited amount of data is presented to the model, due to a lack of diversity of data. An restrictive setting occurs when users label only single instance of the novel class, which can dramatically hurt performance of proxy models, due to the training containing non-annonn non-annottted instances of the novel class (which are treated negative pixels).
images. However, although these proxying methods demon-

---

Under review as a conference paper at ICLR 2023
Deople
Momoecycle
I F'ully-labelee Syntheic Samples
Copy-paste
Training
SOVEREIC
Nove-obijest-olly
Arnotation
Memory-rephe Buffer
Incremental
Novel Task
Updated Model
Generalization
Learning
Our proposed method utilizes guided copy-paste augmentation to synthesize diverse train- ing data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As result, the model incorrectly signs learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt the novel motorcycle class with an annotation of single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view color.
Figure 1:
To address the aforementioned challenges, we propose GAPS (Guided copy-And- Paste Synthesis), which improves the training of incremental segmentation models by synthesizing fully-an fullt-annotated images from partially-annotale play module into different incremental learning algorithms, e.g., standard fine-tuning PIFS (Cer- melli et al., 2021). Copy-paste generates diverse training data to boost performance under few-shot settings, enables the model to learn with partially-annotate images with few novel instance out of many novel instances in an image (e.g., as illustrated at the lower left part of Fig. I), which is stricter setting than semantic background shift (Cermelli et al., 2020).
examples. It is model-agnostic, and can be inserted as a plug-and-
annotated
To the best of our knowledge, we are the first to introduce copy-paste as a synthesis technique to cre- diverse data source for few-shot incremental segmentation. Although copy-paste (Ghiasi al., 2021) has been shown to be an effective data augmentation technique for offline visual recognition tasks, we identify new key technical challenges to adapting few-shot incremental settings. First, how should the synthesizer pick representative samples from the base dataset to construct diverse pool fully-annotated base scenes? Second, given constructed pool of fully-annotated images, how should it select the most suitable base images to be pasted on? Third, after an informative image selected, from what distribution should it sample current and previously learned novel objects to mader-sampling? Our GAPS method differs
balance sample frequency and avoid over-sampling from a na've (e.g., uniform random sampling) copy-paste process by a guided strategy that considers diversity of the memory-replay buffer, imbalanced class frequencies between base classes and novel classes, and contextual similarity of images.
In summary, contributions are as follow:
1. We are the first to introduce copy-paste as synthesis technique to address partially-laben labeled images for incremental segmentation.
2. To address the gaps between copy-paste under the offline setting an augmentation tech- nique and under the online setting as a synthesis technique, we design guided copy-paste process that improves the distribution of synthesized images by enforcing diversity of the memory-replay buffer, exploiting contextual information, and balancing class frequencies.
3. The proposed GAPS technique consistently boosts the performance of a variety of incre- mental learning algorithms from simple fine-tuning sophisticated state-of-the-arts under
2

---

Under review as a conference paper at ICLR 2023
the few-shot setting. Furthermore, demonstrate the strength of GAPS to cope with more challenging task setting where only one instance out of many novel instances in an image is annotated, which highlights co copy-paste as a better alternative to pseudo-labeling or modified loss for practical incremental segmentation applications.
2 RELATED WORK
Incremental Learning for Semantic Segmentation. It is known that many learning-based mod- els suffer from catastrophic forgetting (McCloskey & Cohen, 1989), a phoenomenon that causes models to perform significantly worse old tasks when they are fine-tuned to adapt to new tasks. Incremental learning studies how to enable models to adapt to new classes while mitigating catas- trophic forgetting without accessing the old dataset full-scale re-training. This problem has been studied extensively in image classification (Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Yoon al., 2018; Rebuffi al., 2017; Mallya al., 2018: Castro et al., 2018 88; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of seman- tic segmentation (Cermelli et al., 2020; Michieli & Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021: Zhang et al., 2022). Noticeably, a few attempts have been made by recent work to ad- dress the semantic background shift problem proposed by Cermelli al. (2020) via either pseudo- labeling (Cha et al., 2021; Douillard et al., 2021; Yan et al., 2021) or modified loss (Cermelli et al., 2020; Zhang et al., 2022) to train on partially-annotaa work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with few data via guided copy paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a model-agnosti data pre-procssoo,, which is or- thogonal to incremental learning techniques such as regularization (Li & Hoiem, 2017).
images of novel classes. However, existing
Few-Shot Semantic Segmentation. Few-shot semantic segmentation methods predict segmentation masks of novel classes using only few training examples of the novel class. Many meta-learning- based methods (Shaban et al., 2017; Wang et al., 2019; Tian et al., 2020b; Zhang et al., 2020) and even specialized datasets (Li al., 2020) have been proposed to address such a problem. However, few-shot semantic segmentation methods produce novel-class-o segmentation. In comparison, our proposed method works in more challenging and realistic setting where both base classes and novel classes need to be segmented.
binary songround-backkroundd
Few-Shot Incremental Segmentation. While there are many works few-shot incremental image classification (Tao et al., 2020; Cheraghian et al., 2021), relatively fewer works have been done to investigate few-shot incremental segmentation (Tian et al., 2020a; Cermelli al., 2021; Ganea et al., 2021). Tian et al. (2020a) designs a meta-learning- based classifier that adjusts learned prototypes by modeling interaction between base classes and incoming novel class. Unlike Tian et al. (2020a), which only performs a single update of weights in the classifier, PIFS (Cermelli et al., 2021) apply regularization techniques to allow fine-tuning of the entire network, achieving state-of-the-at result in few-shot incremental semantic segmentation. However, PIFS (Cermelli al., 2021) is fine-tuned on only a small number of samples, which leads to sub-optimal performance due to overfitting. In addition, PIFS requires fully-annotated images as input, which hinders its potential for practical applications.
Copy-Paste Augmentation. Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi al., 2021; Dvornik et al., 2018) have been done investigate how sopy-and-paste augmentation can help with various visual tasks. Dvoriik et al. apply copy- paste augmentation in object detection by designing a neural network to consider context and guide copy-paste, However, the context guidance method proposed by Dvomik et al. can not be trivially applied to our application since it requires abundant fully-annotated training data. More recently, Ghiasi et al. conduct extensive experiments to demonstrate the effectiveness of simple copy-pac copy-paste in the instance segmentation problem. We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naïve copy- Synthesis (NPS) to adapt it to our online task setting. However, as we will demonstrate, such naive adaptation gives unsatisfactory performance in task setting because of gaps between the static offline learning and continual
3

---

Under review as a conference paper at ICLR 2023
Memory-rppay Baffer Diversiry-awar Excmplar Selection
Underlying Midel
f
Conrest Sampler
Syatheszer
LCB
Purtially-annotuae -nnotattd Image
Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few one annotated instance of a single image. It is more probable for GAPS to select scene contextually similar to the provided image from memory-replay buffer Do. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli al., 2021) example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.
online learning. In our work, we propose a series of techniques to guide the copy-paste synthesizer address these gaps, whose effectiveness evident from the significant improvement from NPS.
METHOD
Problem Setup. Let X C R #xxx33 be set of RGB images with size H x W. C C N be set of category labels, and y C Hx be set of label masks (i.e., per-pixel category labels in C). In semantic segmentation, we aim learn model ø that maps an image X' to segmentation mask E y Different from standard semantic segmentation, in few-shot incremental segmentation, C expanded over time through two stages. During the base learning stage, the model is provided with a base dataset Do {(z,, Vi) I; X. Us yCo), where Co is set of classes in the base dataset. Dn generally contains many fully-annotated image-mask pairs and is used to train model do - yCo from scratch.
During the incremental learning stage, sequence tasks (D1, D2.... } with novel categories presented to the model, where D, {(x,, B:)læ; X. ys ye and C, set of classes for task D,. In few-shot learning, the size of the training sets for the novel tasks is small, i.e., Di1 < Dol After adapting to task Dj. the model is updated as éj X yU The goal of incremental learning is to optimize the model performance jointly on both previous tasks and the current task. To enforce the partially-anne annotated image setting, we follow Cermelli et al. and assume that only novel classes are annotated, i.e., C, nC, Ø for all #
Method overview. Fig. illustrates our proposed Guided copy-Paste Synthesis (GAPS) framework for few-shot incremental segmentation. It is a generic and model-agnostic data synthesis framework that generates falle-labeled scenes from partially-annote the underlying segmentation model. After the standard base learning stage with base dataset Do. we build memory-replay buffer Do using an iiversttygguieed exemplar selection strategy (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copy- ing from the masked novel objects in D,...., Dj and pasting onto base exemplars from the replay buffer Dr- The strategy by which we choose base exemplars and novel segments is context context-guided (Section 3.3) and flass-frequecy--uul (Section 3.4).
images of novel objects as a preproces-
4

---

Under review as a conference paper at ICLR 2023
3.1 FEW-SHOT INCREMENTAL SEGMENTATION MODEL
In principle, GAPS is model-agnostic, which means that it can work with many incremental segmen- tation models as a diverse data source to improve their performance. Here we adopt PIFS (Cermelli et al., 2021) as the main baseline underlying segmentation model for its state-of-the-art perfor- mance on few-shot incremental semantic segmentation and support for end-to-end training. The PIFS segmentation model Φ is composed of convolution-based feature extractor f and per-pixel classification layer g using prototypical representation - 9 configured to classify the pixels into classes, so it is parameterized with prototypes W = [w1, W2,,... ................................ Wnn]- Intuitively, f maps every pixel in an input image onto the unit hyper-sphere in a high-dimensional representation space. then generates probability prediction by comparing cosine similarity of feature vectors with learned class prototypes W; in the representation space and applying softmax of the resulting similarities.
We want to note that re- re-implementation of Cermelli ct al. (2021) uses C2 regularization rather than the prototype distillation loss proposed by Cermelli et al. We found experimentally that when diverse data source is used (i.e., proposed GAPS), C2 regularization works better. To be more precise, we construct a penalization term LrGG to regularize the output before the classifier. For incremental learning task Dj with image-mas -aak pairs (Ir y), we have
LEGG = (z) - 1
(1)
The final training loss is given by
g(x,y) (o) (z). + XLREG,
where CCE is either the standard cross-entropy loss or the modified cross-entropy loss from Cermelli al. (2020). X hyper-parameter used to weight the regularization loss. All other components are the same as in (Cermelli et al., 2021). We denote our re-implementation of PIFS with La regu- larization loss as PIFS(C2).
3.2 DIVERSITY-GUIDE EXEMPLAR SELECTION WITH LEARNED PROTOTYPES
For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work di- rectly top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), propose a diversity-guit guided exemplar selection process that builds a small yet diverse memory-replay buffer Do from Do to mitigate catastrophic for- getting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.
Algorithm 1 Construct Memory-replay Buffer
Require: number of exemplars n + FLOOR(n/ Yol) // Sample per class for from to |Yo| do
Se < { yi) E Doo E yi}
for (I,, yi) E S. do
+ MAP(I;, yr, c) // Pred. Proto. 8; + COSINESIMILARITY (pic. we)
end for
Sort S. by similarity score s;
F.S <- (} // final exemplar set of class c
for j = 1,2, ........ k do
Here, we present an algorithm (Algorithm 1) to construct Do by exploiting the Masked Av- erage Pooling (MAP) function from Cermelli et al. (2021). Intuitively, we approximate the difficulty of every sample by their similarity be- tween the estimated prototype with learned pro- totypes. Estimated prototypes that are close to the learned prototype are considered easy sam- ples and vice versa. After building a list of base samples sorted by difficulties, we select sam- ples from equally-spacee intervals to ensure samples of all difficulties are selected for diversity.
Lide + j j-1Sol/k
Unds + MIN (Lower + Scl/k,
(x. y) < SAMPLE(S. Usaul)
ES.- + ES. U (x.
end for end for
Do UNIFORMSAMPLER. (... II......
E.S,,n)
During the incremental learning stage, we select at most k samples for each novel class using the same algorithm to memorize novel classes. To maintain the size of the memory-replay buffer, we remove old samples from the memory-replay buffer but keep least 80% of the samples to be fully-annotated samples, that we have diverse base images for copy-pasting.
5

---

Under review as a conference paper at ICLR 2023
3.3 CONTEXT-GUIDED SAMPLING
We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, TV in an apattment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design context- guided sampling algorithm to select images from Do that are contextually similar to the provided partially-labeled images.
GAPS uses a scene embedding network h X Td to estimate pairwise contextual similarity between any two given images. The embedding h maps an input image to a scene embedding vector in metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan & Zisserman, 2015) and replace its last fully connected layer with possie-similarrty-baee based classifier. The network h is trained using the Places365 (Zhou et al., 2017a) dataset, which contains 365 scene categories and roughly 1.6 million images in its training split. During training. the scene prediction is done by comparing the predicted embedding with existing prototypes. After training, the network / is frozen and the 365 prototypes are discarded.
To find contextually similar base images to each novel image, we evaluate the cosine similarity of the novel image to each of the examples in Do and constuct a contextually similar subset S with 1Do//10 most contextually similar examples. When there are multiple novel images, we take a union of selected examples. To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from S with a probability of a, and sample from Dg with a probability of 1 1- a, where is hyperparameter set to 0.9 in our implementation. Note that we only need to compute scene embedding once for every image in Do and incoming partially-annotatee images. Hence, the context- guided sampling algorithm poses only minor computational overhead to GAPS.
3.4 CLASS-PREQUENCY-GUUDEE PROBABILISTIC SYNTHESIS
Now the final question is, given fully-annotated image rg and an image of novel object zN, how frequent should we apply copy-paste? There is a trade-off between oversampling and un- dersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes. On the other hand, rarely pasting novel instances would lead to undersampling of the novel class. Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis.
To apply vRFS, we first need to compute category- wise sampling factor Te for every c as in RFS. If E Co. we set r. 1 as since during the construction of Do already consider class balance by class-wise uniformly sampling. If c E C₁ with j 2 1, we first compute its class frequency by fe nShot/ Dol. where n.Shot denotes the number of images in Dj with at least one pixel of c. Then, the category-wise sampling factor for is given by rc = MAX(1, vt/fe). Note that in (Gupta et al., 2019), é is chosen as a hyperparameter to be tuned. However, we empirically found that setting
be the multiplicative inverse of total number of classes, or t 1/1 1/| Ui.... Y,J. is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings.
During the synthesis process, we first randomly select a novel class cn from C,. and another class Co from Cy \ {. We first decide if c, should be pasted onto BB. To apply vRFS resampling, we hallucinate two virtual samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of rÚ = MAX = Te.- Thus, the probability to synthesize class co onto z is given by Te,/ /1 re.). We then repeat the process for the novel class oN. Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of cN in the final image.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASETS
We follow literature in few-shot segmentation and few-shot incremental segmentation (Shaban ct al., 2017: Nguyen & Todorovic, 2019; Tian et al., 2020a; Cermelli et al., 2021) and evaluate model on the PASCAL-5' dataset (Shaban et al., 2017) and the COCO-20' dataset (Nguyen & Todorovic, 2019). PASCAL-5 is artificially built from the PASCAL VOC 2012 Semantic Segmentation dataset (Everingham et al., 2010) with additional annotations from the SBD (Hariharan et al., 2011) dataset. The original VOC segmentation dataset provides segmentation annotations for 20 object categories. The PASCAL-5' dataset manually splits the original dataset into 4 folds for cross-validation. For each fold, 5 categories are selected as novel categories, while the remaining 15 categories are re- garded as base categories. In our experiments, images containing at least one pixel of the novel categories are excluded from the base dataset. The construction of the COCO-20' dataset handles the 80 thing classes in COCO in similar manner, where the dataset is split into 4 folds and each fold contains 20 categories. The rest of the process to construct the base dataset and the novel dataset in COCO-20' is same as the PASCAL-5 dataset.
4.2 EVALUATION PROTOCOLS
In the base learning stage, the model is trained using the entire base dataset. In incremental learning stages, sequences of tasks are presented to the model. We use the same evaluation protocol as pro- posed in Cermelli et al. (2021) for fair comparisons, where 5 incremental learning tasks are used for PASCAL-5' and each task contains 1 class from the novel split. On the COCO-20* dataset, there are incremental learning tasks, and each task contains 5 classes from the novel split.
We evaluate the performance of the model on the entire validation set of the corresponding dataset after every step. For fair comparisons with Cermelli et al. (2021), we average results across dif- ferent steps and exclude completely unseen classes from evaluation of current step. We use three different metries to evaluate the performance the model: mean Intersection-over-UN over base categories, mloU over novel categories, and harmonic mean of the base mloU and the novel mloU. Unless otherwise noted, the numbers are computed by averaging results over splits in cross- validaaal fashion.
(mloU)
To average out randomness due to few training samples, we also average results over multiple runs with different set of few-shot training samples. For experiments on splits on PASCAL-5 we found that averaging results over 10 runs with randomly sampled few-shot novel images yields stable results. For CO'O-20', we found that averaging results over runs is enough to yield stable results.
4.3 MAIN RESULTS
In Table we evaluate various incremental segmentation methods on the PASCAL-5' dataset and the COCO-20* dataset, and combine them with GAPS where appropriate.
Baselines. There are two main baselines we are comparing to. The first one is SSUL (Cha al., 1021), which is the state-of-the-an method in non-few-shot incremental segmentation. The second one PIFS (Cermelli et al., 2021), for it is the state-of-the-ar art method few-shot incremental semantic segmentation. We also report performance of our re-implementation PIFS((2) described in Sec. 3.1. In addition, we also evaluate simple fine-tuning and MiB (Cermelli et al., 2020).
GAPS consistently increases performance under few-shot settings. Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require anloytannotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotale images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.c., SSUL), GAPS still steadily increases performance on novel categories.
7

---

Under review as a conference paper at ICLR 2023
METHOD
BASE
NOVEL
HM
BASE
NOVEL
HM
PASCAL-5'
1-SHOT
PASCAL-5 5 5-SHOT
MIB (Cermelli et al., 2020)
43.9
2.6
4.9
60.9
5.8
10.5
FINETUNE"
47.2
3.9
7.2
58.7
7.7
13.6
FINETUNE+
64.2(-17.0)
16.2x+123)
25.9(+18.7)
66,8(-8.1)
38.1(-30.4)
48.5(+34.9)
SSUL (Cha et al., 2021)
73.9
16.4
26.8
74.8
27.8
40.5
SSUL+GAPS
74.0(-0.1)
19.9(43.5)
31.3(+4.5)
(4.9(-0.1)
30.0k+2.2)
42.8(+2.3)
PIFS (Cermelli et 2021)
64.1
16.9
26.7
64.5
27.5
38.6
PIFS(C,)
64.6
19.7
30.2
57.7
24.5
34.4
PIASS
66.8(+2.2)
23.6(+3.9)
34.9(+4.7)
(8.2(+10.5)
44.2(-19.7)
53.7(+19.3)
COCO-22
1-SHOT
COCO-20'
5-SHOT
MIB (Cermelli et al., 2020)
40.4
3.1
5.8
43.8
11.5
18.2
FINETUNE"
38.5
4.8
8.5
39.5
11.5
17.8
FINETUNE+GAPS
44.5(=6.0)
11.0(46.3)
17.7(e9.5)
46,4(-6.9)
24.9(-13.4)
32.4(+14.6)
SSUL (Cha al., 2021)
51.0
6.3
11.3
51.6
15.0
23.2
SSUL+GAPS
50.8(-0.2)
11.0(4.7)
18.1(46.8)
51.9(-0.3)
17.1(-2.1)
25.7(+2.5)
PIFS (Cermelli et al,, 2021)
40.4
10.4
16.5
41.1
18.3
25.3
P)FS(C.) P 2 +GAPS
45.7
10.3
16.8
46.2
20.2
28.1
46.7(-1.0)
12.8(+2.5)
20.1(+3.3)
48.8(+2.6)
25.8(+5.6)
33.7(+5.6)
Table 1: Methods augmented with our proposed GAPS consistently outperform their un-auumented counterparts across different few-shot settings COCO-20' and PASCAL-5'. Methods noted with' use fully-annotated images, others use same sets of images with novel-class-only partial annotation.
:: our re-implementation using L2 regularization. Highest results are colored red and the second highest results are colored blue. HM stands for harmonic mean. (Best view in color).
4.4 ABLATION STUDY
In Table 2, we ablate guidance designs in GAPS to illustrate how different types of guidance con- tribute to the final incremental learning performance than naive copy-paste synthesis. Due to the highest harmonic mean of PIFS(C))+GAPS, here we use GGSSSC))+GAPS for the ablation study.
Our diversity-guider exemplar selection method consistently increases performance on base categories, which suggests that it is capable of choosing diverse samples to construct representa- tive memory-replay buffer and mitigate catastrophic forgetting.
Context-guided sampling steadily improves performance novel classes, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis.
Frequency-guinn guideed probabilistic synthesis boosts results on novel classes. On the other hand, its influence on base categories is negligible. We take closer look step-wise performance (whose visualization is available in Appendix A. and found that the reason is due to unguided copy- paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-rephee buffer.
4.5 MORE CHALLENGING SINGLE- INSTANCE EXPERIMENT
Though the semantic background shift proposed by Cermelli et al. (2020) relaxes the requirement to provide full segmentation annotations, still requires all novel instances in images to be annotated, which can be time-consuming to obtain in cluttered scenes and hinder potential applications. Here we consider a more challenging task setting, which we term single-instance incremental learning- Namely, for training images provided in incremental learning stages, if there are multiple instances of a novel class in the image, we assume that only one instance will be annotated.
To simulate this setting, use the instance-level segmentation annotation provided by the COCO dataset to enforce only annotation of one novel instance in every image is available to the model. Since oatte-o-thh--ee incremental segmentation approaches pseudo-labeling (Cha et al., 2021), we design& a method PIFS((2)) which simulates combining PIFS(C,) with pseudo-labeling to cope with partially-annotatee sample. Here we allow PIFS( to have access additional information
8

---

Under review as a conference paper at ICLR 2023
MEM
COPY-PASTE
F-GUIDE
D-GUIDE
C-GUIDE
BASE
NOVEL
HM
46.2(+ 0.3)
20.2(= 0.7)
28. [[ 0.3)
49.3 0.2)
27.9 19
0.20
0.6)
0.)
/
/
0.22
25.2(# 0.6)
32.9(1
48.2 0.2)
25.0 0.7)
32.9(1 0.3)
48.8(+
25.8(= 0.6)
33.7(
Table 2: Ablation study of components in GAPS on PIFS((2) on the COCO-20' dataset under 5- shot setting. Note that when only combined with the memory-replay buffer, the base loU is higher because model has access to additional full annotations. When diversity guidance (D-guide) dis- abled, Dg consists random examples from the base dataset, resulting in worse base performance. When context guidance (C-guide) is disabled, a base image is uniformly sampled. When frequency guidance (F- (F-guide) is disabled, a novel instance is sampled uniformly and is always pasted onto the base image. 95% confidence intervals over 20 trials reported assuming that trial results are fully-annotated masks when copy-paste is turned off.
normally distributed.
METHOD
BASE NOVEL
HM
BASE
NOVEL SINGLE-INSTANCE ONLY
HM
ALL INSTANCES
PIFS(C,) 46.2
20.2
28.1
46.1 (-0.2%)
17.6 (-12.9%)
2-.. 33.2 (-1.5%)
PIFS(C))+GAPS
48.8
25.8
33.7
49.1 (+0.6%)
25.1 (-2.7%)
Table 3: Performance of pseudo-labeling methods and GAPS under the more challenging single- instance learning setting on COCO-20' 5-shot. Only novel instance out of potentially many stances in individual training images is annotated. The pseudo-labeling baseline, PIFS((2) yields substantially worse performance; whereas PIFS(C))+GAPS has only minor performance decreases.
the annotation of other non-novel background pixels to simulate an oracle gseudo-labeling model which perfectly segments learned classes but recognizes unseen novel classes as background.
The results are given in Table 3. We can observe that the pseudo-labeling baseline, PIFS(C2) yields substantially worse performance when the model receives single-instance annotations. We reason this is due to noisy labels generated by the pseudo-labeling process, where novel instances are incor- rectly labeled as background. On the contrary, PIFS(C2) +GAPS shows only a minor performance decrease with single instances. This highlights the potential of copy-paste synthesis as alternative to the existing ggeudo-labeling paradigm to cope with the more realistic single-instance setting.
More results and visualization. Due to space limit, kindly refer readers the appendix for more guantitative results from Appendix A.2 and visualized qualitative results in Appendix A.
5 CONCLUSION AND DISCUSSION
In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially- annotated images. Our proposed GAPS technique selects representative exemplars in the memory- replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.
In future work, we are interested in further application of copy-paste as synthesis technique to cope with the background shifting problem for incremental segmentation. We believe that copy- paste can serve as promising alternative to pseudo-labeling and modified loss to enable learning on partially-annotatee
images. We also believe that further optimizing exemplar selection and sampling strategies can lead to better guidance and lead to even better performance. Finally, the ability to learn with as few as one annotated instance an image raises several intriguing possibilities. For example, integrating work with learning-basee interactive segmentation will enable human operators to continually and adaptively teach novel classes and correct failed predictions. This workflow has many interesting applications such robot teleoperation where sparse annotations preferable. Learning with weaker annotations, like bounding boxes or single clicks, and even self supervveer,s,nn,, also interesting direction explore.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jihwan Bang, Heesu Kim, Youngloon Yoo, Jung-Woo Wno Ha, and Jonghyun Choi. Rainhow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, 2018.
Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Mod- eling the background for incremental learning in semantic segmentation. In CVPR, 2020.
Fabio Cermelli, Massimiliano Mancini, Yongqin Xian, Zeynep Akata, and Barbara Caputo.
Prototype-based incremental few-shot semantic segmentation. arXiv preprint arXiv:2012.01415, 2021.
Sungmin Cha, YoungJoon Yoo, Taesup Moon, al. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. NewrlPS, 2021.
Liang-Chiel Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint wrXiv:/766.05887 2017.
Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Peters- son, and Mehrtash Harandi. Synthesized feature based few-shot class-incremental learning on mixture of subspaces. In ICCV, 2021.
Arthur Douillard, Yifu Chen, Amaad Dapogny, and Matthieu Cord. Plop: Learning without forget- ting for continual semantic segmentation. In CVPR, 2021.
Nikita Dvornik, Julien Mairal, and Cordelia Schmid. Modeling visual context is key to augmenting object detection datasets. In ECCV, 2018.
Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy syn- thesis for instance detection. In ICCV, 2017.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In CVPR, 2021.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. In ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Xiang Li, Tianhan Wei, Yau Pun Chen, fu-Wing Tai, and Chi-Keung Tang. A 1000-class dataset for few-shot segmentation. In CVPR, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 2017.
David Lopez-Pat and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. NewrIPS, 2017.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul-
tiple tasks by learning to mask weights. In ECCV, 2018.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
10

---

Under review as a conference paper at ICLR 2023
Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVW, 2019.
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In ICCV, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few- shot class-nncrementl learning. In CVPR, 2020.
Zhuotao Tian, Xin Lai, Li Jiang, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. Generalized few- shot semantic segmentation. arXiv preprint arXiv:2010.05210, 2020a.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang. Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. TPAMI, 2020b.
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In ICCV. 2019.
Shipeng Yan, Jiale Zhou, Jiangwei Xie, Songyang Zhang, and Xuming He. An framework for online incremental learning of semantic segmentation. In Proceedings of the 29th ACM Interna- fional Conference on Multimedia, 2021.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018
Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng. Represen- tation compensation networks for continual semantic segmentation. In CVPR, 2022.
Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE transactions cybernetics, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017a.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017b.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.

---

Under review as a conference paper at ICLR 2023
Split
Categories
person, airplane, boat, parking meter, dog elephant, backpack, suitcase, sports ball, skateboard wine glass, spoon, sandwich, hot dog, chair dining table, mouse, microwave, refrigerator, scissors
COCO-20-0
bicycle, bus, traffic light, bench, horse bear, umbrella, frisbec, kite, surfboard cup, bowl, orange, pizza, couch toilet, remote, oven, book, teddy bear car, train, fire hydrant, bird, sheep zebra, handbag, skis, baseball bat, tennis racket fork, banana, broccoli, donut, potted plant Iv, keyboard, toaster, clock, hair drier
COCO-20-1
COCO-20-2
motorcycle, truck, stop sign, cat, cow giraffe, tie, snowboard, baseball glove, bottle knife, apple, carrot, cake, bed laptop, cell phone, sink, vase, toothbrush
COCO-20-3
Table 4: COCO-20' class splits under multi-step settings. Every novel split is further split into steps sequentially sorted by their original class indices in (Cermelli et al., 202 A row indicates classes in a step. For example, in COCO-20-L classes presented to the model in the first incremental learning step are bicycle, bus, traffic light, bench, and horse.
Split PASCAL-5-0 PASCAL-5-1
Categories
aeroplane bicycle bird boat bottle bus car cat chair cow
PASCAL-5-2
table dog horse motorbike person
PASCAL-5-3
potted plant sheep sofa ttain tv-monitor
Table 5: PASCAL-5' class splits under multi-step settings. Every novel split is further split into steps sequentially sorted by their original class indices and each step split by the 'I'.
A APPENDIX
A.1 DATASET SPLITS
We use the PASCAL-5' splits (Shaban al., 2017) and the COCO-20 splits (Nguyen Todorovic, 2019) as described in previous works. We follow the multi-step setup described in (Cermelli et al., 2021) to ensure fair comparisons. Details are given Table 4 and Table 5.
A.2 MEMORY-REPLAY BUFFER CONSTRUCTION
In Table 6, we study how different memory buffer construction strategies influence the performance of the model. We experiment with multiple baseline implementations to construct the memory- replay buffer: Full-set-random denotes samples that are uniformly sampled from the base training set. Classwise-randon denotes randomly selecting even number of samples from each base class. To consider multiple classes in images, we design two baseline methods. RFS (Repeat Factor Sam- pling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights rare classes and has been shown work well long-tailed task settings. The other baseline, "GreedyClass' is achieved by greedily picking samples every base class with the most number of classes. To consider the region sizes in the image, we design a base- line method, "BalancedRegion" which uses similar algorithm our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.
12

---

Under review as a conference paper at ICLR 2023
Base Set
Base
Novel
HM
Full-set-random
47.9 + 0.2
24.6 0.7
32.5
Classwise-random
48.3 0.2
24.7
32.7
RFS
47.6 77.6 0.3
25.2
33.0
Greedy
4.2
25.0
33.0
BalancedRegion
24.9 +
32.9
Easy
4.2
24.2
32.2
Hard
47.7 + 0.2
25.2+
33.0
Easy-Hard-Mix
48.4 08.40..3
24.8 0.7
32.8
Diverse Sampling (Ours)
48.9 0.2
2.9
33.3
Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 are reported.
Method
Base 47.1 0.3
Novel
HM
Always Paste (Ghiasi et al., 2021)
18.7 1.7
26.8
50-50-Paste
49.7 0.4
24.2 1.5
32.6
CAS (Gupta et al., 2019)
49.4 0.3
1.30
19.3
vRFS (Ours)
48.9 0.4
1.6
33.3
Table 7: Ablation study of different strategies compute synthesis probability COCO-20-1. Our proposed vRFS method computes synthesis probability to balance class frequencies training data, and achieves best performance in terms of harmonic mean. Mean and standard deviation over 10 runs are reported.
Intuitively, easy samples help model maintain learned prototypes during incremental learning, and hard samples help model distinguish decision boundaries. Hence, based ranking method based on prototypical distances, we also construct a few variants: Easy denotes samples whose predicted embeddings are closest learned prototypes; Hard represents difficult samples whose predicted embedding are most distant to learned prototypes in the learned metric space; Easy-Hard-Miz denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, Diverse-Sampling is our proposed method, where samples of all difficulties are equally drawn.
sample difficulty
We experimented with all of these variants and found that the diversity- construction yields the best result in metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture easiest samples and hardest sample also good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity- aware sampling. It is also worth noting that both the 'GreedyClass' method and the "Bal- ancedRegion' method achieve almost comparable results to our methods, but still slightly underper- form our diversity guidance.
A.3 CLASS-FRRQUERNE AWARE SYNTHESIS
We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS. The results are illustrated Table 7. Here, vRFS (virtual RFS) our proposed method. CAS CClass-wware-sampa sware-samplingg) is sampling strategy from (Gupta et al., 2019) which first uniformly samples class and then selects a sample containing instances from the selected class. In our im- plementation, the probability to synthesize novel objects is simply given by 1/M, where M is the sum of total number of base classes and seen novel classes. For every selected base image, the Always Paste method randomly picks novel category and always synthesizes selected novel category onto the given image. Finally, the 50-50-Passe method an extension of the naïve copy-pe copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability. Note that these two options mutually exclusive.
instance of the
13

---

Under review as a conference paper at ICLR 2023
Method
Base 48.5 + 0.4
Novel
Standard Jittering
24.1 1.5
Large Scale Jittering (Ghiasi et al., 2021)
48.9
0.4
25.3 + 1.5
Table 8: Ablation study of magnitudes random resizing in copy-paste on COCO-20-1. Large Scale Jittering is proposed in (Ghiasi et al., 2021) with more aggressive resizing scale. Mean and standard deviation over 10 runs are reported.
0.5
0.3
lou
0.2
PIPS Base
0.1
PIFS(L2) Base
+GPPS Novel
PIFS(L2) Novel
0.0
3
12
15
18
# of novel tasks introduced
Figure Long-term continual learning performance of PIFS(C2) and PIFS(C))GAAS on the COCO-20-1 split under 5-shot setting. Best view in color.
In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU. On the other hand, the 50-50-Paste baseline and the CAS baseline achieves better results possibly because they undersample novel classes.
A.4 EFFECT OF GEOMETRIC VARIATIONS
In Table 8, we investigate how geometric variations affect the performance in our method. Our conclusion consistent with what was found in (Ghiasi et al., 2021): using more aggresive scale jittering helps increase the diversity of samples and increases performance. Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique when there is few data presented, highlighting possibility to further improve the performance by using even stronger augmentation technique.
A.5 ROBUSTNESS AGAINST CATASTROPHIC FORGETTING
We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) used. The step-wise result can be found in Fig. 3. PIFS(C2), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS.
A.6 EVALUATION ON DISJOINT ADE20K 100-5 5-SHOT SETTING
To validate our approach on the ADE20k dataset (Zhou et al., 2017b; 2019), we ran suite of exper- iments where follow the ADE100-5 (11 steps) setup discussed in (Cha al., 2021) and modify 5-shot learning. We tested PIFS(C2) and GGPSS on this setting. The results are given Table 9. We observe steady increase in both the base loU and novel IoU in this challenging task setting methods augmented with GAPS. This further validates the effectiveness of approach.
14

---

Under review as a conference paper at ICLR 2023
0.50
0.40
0.35
MFS(L2) Base lou
(LF)(22) Novel loU
RTS(L2/+GAPS
0.30
MFS(L2)+GGM GAPS Novel IoU
0.25
0.15
Memorr Size
Figure 4: Performance of PIFS(C))+GAPS the COCO-20-1 split under 5-shot setting with vary-
ing memory-replay buffer size. Best view in color.
Method
Base
HM 6.3 5.9 + 1.4
Novel
PIFS(C2)
33.8 + 0.9
3.5 + 1.5
PIFS(C>)+GAPS
36.2 + 0.6
10.1
Performance of 2IFS(C2) and GIPSS on the ADEI00-5 disjoint 5-shot setting, Mean and standard deviation over 5 runs reported.
Table 9:
A.7 COMPARISONS WITH EXEMPLAR-ONI. METHODS
We explore an alternative memory-replay buffer implementation to add examples learned during the incremental learning stage to the mepory-replay buffer, in an attempt to help performance on novel classes. Since examples observed during the incremental learning stage are only partially annotated, they can not be directly added to the memory-replay buffer. We experiment with an alternative method. Here, "PIFS(C))+MM-H partially annotated samples before adding them to memory. The results are given in Table 10. We can observe that storing samples observed during the incremental learning stage leads to only moderate increase in novel IoU, Our PIFS(C,)+GAPS +GAPS significantly outperforms both baselines.
means that we follow SSUL and perform pseudo-labeling on
A.8 EFFECT OF MODELS PRETRAINED ON DIFFERENT DATASETS FOR CONTEXTUAL GUIDANCE
We experiment with several other pretraining models as the contextual guiding model. In particu- lar, use ImageNct-pretranng model and pretrained image encoder from CLIP (Radford al., 2021) to compute scene embedding for incoming images. The other settings remain unchanged. The results given in Table 11. We observe from the results that the ImageNett-retraange model per- forms worse than Places365- Places365-pretraiined model. On the other hand, the CLIP--prtrrained model yields comparable performance to method.
In summary, we conclude that Places365 is appropriate dataset for training the image-level con- textual sampler from both quantitative. Qualitative discussions can be found in Figure 6.
A.9 PERFORMANCE WITH VARYING MEMORY SIZES
We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where vary the number of fully- annottadd images in the memory-replay buffer investigate the per-
15

---

Under review as a conference paper at ICLR 2023
Method
Base
Novel
HM
PIFS(C2)
46.4 0.2
20.7 = 1.0
28.0
(I +MEM
49.4
1..6
27.0
PIFS MEEM-LL
4.2
2..
29.8
PIFS(C))+GAPS
0.2
0.9
33.3
Comparisons of GAPS and various exemplar-only methods with PIFS((2). Mean and Table 10:
95% confidence intervals over 10 runs are reported.
Pretrained Model
Base
Novel
HM
ImageNet CLIP
48.2 0.2
24.3 0.9
32.3
48.5 0.2
2.
33.3
Places365
48.9 0.2
0.9
33.3
Table 11: Performance different pretrained models for contextual guidance under the COCO-20-1 5-shot setting. Mean and 95% confidence intervals over 10 runs are reported.
formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS((2) method.
A.10 IMPLEMENTATION DETAILS
To allow fair comparisons with existing methods (Cermelli et al., 2021; Cha et al., 2021; Cermelli et al., 2020),
the same Deeplab-V3 (Chen et al., 2017) architecture with ResNet-101 (He et al., 2016) backbone. Following existing works in incremental few-shot segmentation literature (Cermelli et al., 2021; Tian et al., 2020a), before the beginning of base learning stage, the ResNet- 101 backbone is initialized using weights pre-trained on ImageNet.
For fine-tuning and PIFS(C2), during the base training stage, we an initial learning rate of 0.007 and polynomial learning rate schedule with batch size of 32 and train for 20 epochs on both the PASCAL-5' dataset and the COCO-20* dataset. SGD optimizer is used with 0.9 momentum coefficient and 0.0001 weight decay. During the incremental learning stage, we apply initial learning rates of 0.001 and 0.01 for the feature extractor and the classifier, respectively. Polynomial learning rate schedule is used. For the PASCAL-5' dataset, we use batch size of 16 for 200 iterations and 0.1. For the COCO-20' dataset, we use batch size of 16 for 400 iterations and 0.5. 100 iterations are used for 1-shot cases COCO. For data augmentation, we follow implementations in PIFS (Cermelli et al., 2021) and SSUL (Cha et al., 2021) and use standard random horizontal flipping. random resizing, and random cropping to 512 x 512.
The reproduction of SSUL and the iemmleementation SSUL+GAPS uses the same set of hyperpa- rameters and augmentations from the open-source release of SSUL except for the batch size which changed to 16.
The scene classification network is the original VGG16 (Simonyan & Zisserman, 2015) network without batch normalization layers. We replace the last linear classification layer with prototypical learning layer to make the network generate unit vector embedding for comparison using cosine similarity. The network is trained on the training split of the Places365 dataset (Zhou et al., 2017a) for 6 epochs. We use initial learning rate of 0.01 and polynomial learning rate schedule with batch size of 64. All images are bilinearly interpolated to size 224 x 224. The final scene classification network achieves 0.51 accuracy on the validation split of the Places365 dataset.
A.11 QUALITATIVE RESULTS
Fig. 5 shows qualitative results of PIFS(C2). PIFS(C2)+NPS (naive copy-paste). and PIFS(C))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can that PIFS and PIFS+NPS gives reasonably good predictions that are adapted recently (c.g.. the cow class the last row). However, they tend to exhibit catas- trophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the
classes
16

---

Under review as a conference paper at ICLR 2023
Train
Test
PIFS(L))
PIES(LZ)NNPS PIESLL2)GAPS
GT
BG
Plane
Bicycle
Bird
Beat
Bottle
Table
Dog
Bus
Car
Cat
Horse
Motorbike
Person
Pot Plan
Sheep
Sofa
Train
Monitor
Chair
Cow
Novel
Figure 5: Qualitative results on the PASCAL-5-1 split under 1-shot setting. Sequential adaptation five classes are performed in the order indicated by the blue arrow on left. Visualization done after all adaptation steps are completed. Observe, particularly in rows 1, and 4, that method mitigates catastrophic forgetting and improves performance on both base and novel classes compared to PIFS and PIFS+NPS. (Best viewed in color)
model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.
In Fig. 6, we give some qualitative examples of the scene embedding network. We can see that given an image of traffic lights a railway, scene embedding model is able to select contextually- similar scenes such as railways and street scenes for subsequent synthesis.
Fig. 7 and Fig. 8 demonstrate how PIFS(C)) compare to the baseline PIFS(C2) across mul- tiple step. The set up the same as Pascal-5' 1-shot incremental learning. These visualizations demonstrate how PIFS(C2) forgets learned classes earlier more incremental learning steps occur. On the other hand, PIFS(C))+AAPS is able to maintain good performance on learned classes as adapts classes.
17

---

Under review as a conference paper at ICLR 2023
Top-5 contextually similar images selected by our scene embedding model
Training image of novel class 'traffic light'
Examples of other images not selected by our embedding model
Figure 6: Qualitative visualization of samples selected in the context-awarr sampling process. Given an image of traffic lights railway, our scene embedding network is able select contextually- similar scenes (railway, street).
Test
Train
Ground Truth
Figure 7: Multi-step qualitative results of our method (GAPS) on the PASCAL-5-1 split under 1- shot setting. (Best view in color.)
18

---

Under review as a conference paper at ICLR 2023
Test
Train
Ground Truth
Figure 8: Multi-step qualitative results of PIFS(C2) on the PASCAL-5-1 split under 1-shot setting. (Best view in color.)
19


Reviews:
[{"review_id": "YnihnvxZ2P", "reviewer": "Reviewer_2ZbE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a data augmentation technique to improve the performance of the existing state-of-the-art methods in the incremental few-shot semantic segmentation setting. \nThe proposed technique, called GAPS, augments the few-shot samples by pasting the new classes' pixels onto a subset of exemplar images coming from the previous training steps. There are three critical aspects of this technique. First, exemplar selection: the subset of exemplar images is selected by uniformly sampling the images considering the similarity with the class prototype. Second, where to paste the new classes: the images are selected based on the similarity to the context of the new classes ranked with a VGG pre-trained on Places-365. Third, how frequently apply copy and paste: the paper proposes a dynamic strategy called virtual Repeated Factor Sampling.\nThe results show that GAPS is able to improve state-of-the-art methods by a substantial margin on both Pascal-5 and COCO-20 settings, either using one or five shots. \n", "summary_of_the_review": "Overall, the paper proposes a new interesting data-augmentation technique for exemplar-based Incremental Few-Shot semantic segmentation. However, the paper misses some important comparisons, both in the ablation studies and in the main tables.\n", "strength_and_weaknesses": "Strengths:\n1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.\n2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.\n3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.\n\nWeaknesses:\n1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset. \n2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)? \n3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).\n\n[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. \"Gdumb: A simple approach that questions our progress in continual learning.\" European conference on computer vision. Springer, Cham, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LywXPUUtBOS", "reviewer": "Reviewer_oYb5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a copy-paste data augmentation strategy for the few-shot incremental semantic segmentation task.  It starts with a base learning stage to train an initial segmentation network with fully-annotated images. In the subsequent incremental learning stages, the proposed method develops a guided sample selection strategy to build a memory-replay buffer and to generate fully-labeled training examples with the copy-paste technique from the buffer and a small set of images with novel classes annotated. The guided sample selection strategy considers three factors: diversity in difficulty levels, context similarity, and class frequency balance. The authors integrate their strategy with several incremental segmentation pipelines and evaluate them on two few-shot segmentation benchmarks, PASCAL-5i and COCO-20i.  ", "summary_of_the_review": "While the proposed copy-and-paste strategy seems interesting for the few-shot ISS task, there are several concerns on this work, including the novelty of the method for ISS, the effectiveness of the memory for data synthesis in ISS and the insufficient experimental evaluation. As a result, my preliminary rating leans toward the negative side. ", "strength_and_weaknesses": "Strengths:\n- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination. \n- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.\n- The paper is most clearly written and easy to follow.   \n\nWeaknesses:\n- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods. \n- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS: \n1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful. \n2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor. \n- There are several concerns on the experimental evaluation, which make the conclusion less convincing:\n1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE. \n2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results. \n3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.       ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AqILXAXQlI", "reviewer": "Reviewer_bVGy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a few-shot incremental semantic segmentation methods via guided copy-paste synthesis. To  achieves this, authors provide three kinds of guidance, i.e., diversity-guide, context-guide, and frequency-guide. Ablation study and comparison with baseline model show clear performance improvement with copy-paste synthesis.", "summary_of_the_review": "This paper introduces a copy-paste techniques to incremental few-shot segmentation, and designs three guidance to generate more representation training images. The novelty is a concern, because  copy-paste in segmentation is more about a data augmentation technique, which has already been proved to be effective.  Extensive experiments show clear performance improvement with such technique.", "strength_and_weaknesses": "Strength:\n (1) This paper introduces copy-paste as a synthesis technique to few-shot incremental segmentation.\n (2) Experiments show clear improvement on performance with the proposed method.\n\nWeaknesses:\n(1)  My main concern is the novelty of the paper, copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation task, it is not novel enough to be accepted as an ICLR paper.\n(2) The paper is not clear at least for me, especially figure 1 and figure, it's really hard for readers to understand the task, although the idea is clear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject