A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation
Authors: Tomoya Sakai
Haoxiang Qiu
Takayuki Katsuki
Daiki Kimura
Takayuki Osogami
Tadanobu Inoue
Venue: NeurIPS 2024 poster (2024)
Keywords: few-shot learning
semantic segmentation
catastrophic forgetting
Primary area: machine_vision
TL;DR: A simple approach without resorting to, e.g., complicated modules and meta-learning improved GFSS performance.

Abstract: The goal of *generalized* few-shot semantic segmentation (GFSS) is to recognize *novel-class* objects through training with a few annotated examples and the *base-class* model that learned the knowledge about the base classes.
Unlike the classic few-shot semantic segmentation, GFSS aims to classify pixels into both base and novel classes, meaning it is a more practical setting.
Current GFSS methods rely on several techniques such as using combinations of customized modules, carefully designed loss functions, meta-learning, and transductive learning.
However, we found that a simple rule and standard supervised learning substantially improve the GFSS performance.
In this paper, we propose a simple yet effective method for GFSS that does not use the techniques mentioned above.
Also, we theoretically show that our method perfectly maintains the segmentation performance of the base-class model over most of the base classes.
Through numerical experiments, we demonstrated the effectiveness of our method.
It improved in novel-class segmentation performance in the $1$-shot scenario by $6.1$% on the PASCAL-$5^i$ dataset, $4.7$% on the PASCAL-$10^i$ dataset, and $1.0$% on the COCO-$20^i$ dataset.
Our code is publicly available at https://github.com/IBM/BCM.

A Surprisingly Simple Approach to Generalized Few-Shot Semantic Segmentation
Tomoya Sakai IBM Research Tokyo tomoya. 201 bm. com
Haoxiang Qiu IBM Research Tokyo haoxiang qiu@ibm... com
Takayuki Katsuki IBM Research Tokyo kats@jp. ibm. com
Daiki Kimura
Takayuki Osogami IBM Research Tokyo osogami jegaai@jp... ibm. com
Tadanobu Inoue IBM inouet@jp ibm. com
IBM Research Tokyo daiki@jp ibm. com
Abstract
The goal of generalized few-shot semantic segmentation (GFSS) is to recognize novel-class objects through training with few annotated examples and the base- class model that learned the knowledge about the base classes. Unlike the classic few-shot semantic segmentation, GFSS aims to classify pixels into both base and novel classes, meaning it is a more practical setting. Current GFSS methods rely several techniques such using combinations of customized modules, carefully designed loss functions, meta-learning, and transductive learning. However, we found that simple rule and standard supervised learning substantially improve the GFSS performance. In this paper, we propose a simple yet effective method for GFSS that does use the techniques mentioned above. Also, we theoretically show that our method perfectly maintains the segmentation performance of the base-class model over most of the base classes. Through numerical experiments, we demonstrated the effectiveness of our method. It improved in novel-class segmentation performance in the 1-shot scenario by 6. 1% on the PASCAL-5* dataset, 4.7% on the PASCAL-10* dataset, and 1.0% the COCO-20' dataset. Our code is publicly available https:/ //g thub. com/ wwt/IBM/me// IBM/BCM.
Introduction
Semantic segmentation is vital task in various visual understanding systems, and the goal is to obtain pixel-level semantic categories [1]. Recent developments in convolutional neural networks [2] and vision transformers [3] have pushed the limits of semantic segmentation. With a large amount of annotated images, we can obtain an accurate model that can recognize objects in the training data. In real-world applications, however, the learned model will encounter novel-class objects that are not classified in base classes, i.e., classes that are not annotated in the training data.
To solve this probsem,.ew-shot semantic segmentation (FSS) aims recognize novel-class objects with a few annotated images while using the learned model, which has knowledge about the base- class information. Although various FSS methods have been proposed [4-15], FSS only handles novel-class object recognition, which restricts its applicability since base classes will still appear at inference in practice.
Generalized FSS (GFSS) aims to recognize both base- and novel-class objects [16] and is regarded as more practical setting than FSS. Current GFSS methods rely several techniques such as using combinations of customized modules [16-19], carefully designed loss functions [20]. meta-learning [16, 18, 19], and transductive learning [20]. These techniques improved GFSS performance the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classes
Figure I: Illustration of base-class min- ing (BCM) with three novel classes: "train", "tv", and "couch". Basc-class model first outputs prediction. pre- diction is one of chosen base classes, corresponding model outputs predic- tion. Otherwise, prediction of base- class model is used as it is. Simple rule finds which base class related novel classes. Models for novel classes are trained by standard supervised learning.
base classes
novel classes
background
Input
basa-class
background
madel
bed
bed
model
couch
cost of implementation and computation time. For example, such customized module is not always supported the target framework, methods based on meta-learning require several hours to train customized modules, and transductive learning optimizes models during inference, which is not suitable for applications that require quick responses. However, we found that a simple rule and standard supervised learning improve GFSS performance.
In this paper, we propose a simple yet effective GFSS method that does not use the above techniques. As illustrated in Fig. our idea is mining base classes closely related to classifying novel classes. We thus refer to our method as base-class mining (BCM). Surprisingly, BCM perfectly maintains the segmentation performance of the base-class model over subset of base classes. Since maintaining the segmentation performance of the base-class model is critical in GFSS, BCM will be beneficial, especially when performance difference from the base-class model confuses users.
Our contributions are summarized as follows.
We propose simple yet effective GFSS method based on a simple rule and well-known supervised learning techniques, which can be regarded strong alternate baseline without transductive learning.
We theoretically show that the performance of the base-class model for a subset of base classes is perfectly maintained, which is the first theoretical finding about base-class segmentation performances in GFSS, the best of our knowledge.
We demonstrated the effectiveness of BCM the PASCAL-5', PASCAL-10', and COCO-20* datasets. BCM substantially improved novel-class segmentation performance the 1-shot scenario by 6.1% on PASCAL-5' and 4.7% on PASCAL-10',
2 Related work
2.1
GFSS setting
In GFSS [16], multiple novel classes need to be classified, i.e., multi-class classification, in addition classifying base classes. This differs from the FSS setting of single novel-class classification, i.c.. binary classification.
We consider the practical GFSS setting [20], in which the available resources are few-shot annotated images for novel classes and the base-class model trained using standard learning methods. The existing setting [16] requires annotated base-class samples for training the base-class model and tuning GFSS model that can recognize both base and novel classes. For example, the number of base classes is 60 (excluding the background)
COCO-20 case, meaning that need collect 300 annotated images for tuning in the 5-shot scenario, other than training the base-class model. Such additional samples for base classes are not necessary the practical GFSS setting, resulting in using those samples for training the base-class model, which is an advantage of the practical GFSS setting.
2.2
GFSS methods
The major challenges with current GFSS methods are i) attaining better recognition performance for novel classes and ii) maintaining the segmentation performance of the base-class model. Context- aware prototype learning (CAPL) [16] enhances prototypes with a few annotated images and employs

---

balancing mechanism of prototypes for base and novel classes. Base and meta (BAM) [14, 18] designed customized modules learn knowledge from shot data and combined predictions of both base and novel classes on the basis of thresholds. POP [17] and PCN [19] use similar approaches. Distilled information maximization (DlaM) [20] does not depend on customized modules, instead, uses the information maxinization principle [21] and designs loss function on the basis of knowledge distillation [22] to preserve base-class knowledge. DIaM uses the transductive learning approach [23], which is not suitable for applications that require quick responses.
Compared with the above methods, BCM does not rely on carefully customized models, various combinations of loss functions, or transductive learning.
2.3 Continual semantic segmentation
The GFSS setting relates to another emergence problem setting known as continual semantic segmen- tation (CSS) [24-26] in which the new classes appear in a continual learning manner. The CSS setting reduced the GFSS setting if several novel classes appear with few annotations in a single step. Coincidentally. a previous study [26] empirically found phenomenon similar to our idea that a novel class classified as a base class, through qualitative analyses of their method. However, their method was not designed for the few-shot learning setting, meaning that how to improve GFSS performances is unclear. In contrast to their findings, BCM explicitly integrates the idea of the relation between base and novel classes with the architecture. These differences in problem settings and architectures differentiate CSS-based methods from BCM.
3 Preliminaries
3.1 Problem settings
We consider the following practical GFSS setting [20].
Let X E #x333 denote an RGB image of height H and width W, and Y E yHw be its corresponding segmentation map, where y C {0, 2,3, 2....... is a set of object classes. Let [- indicate the j-th element of a matrix, where (l,..., HWJ. If [Y]; = y. the object y exists at the j-th pixel.
Let 3b. and Vn be the sets of base and novel classes, respectively, such that 36 n Vn 0 and Van = -Y,UYa. We use the class '0' for background, which is often the case with implementation. For the sake of simplicity,
include the background into Dh, e.g. V1- (0, 1,2,3) and V (4, 5).
We have a learned base-class model Gb. which is trained with a large amount of annotated images by using standard semantic- semantic-segmentation segmentation methods [1]. Given X, Th returns a base-class segmentation
map Y, >'" Similarly to the practical GFSS setting [20]. we do not assume the customized architecture for Go. enabling us to easily use cutting-edge foundation models [27].
A K-shot dataset contains K examples with its ground-truth mask for each novel class y E Vn. e.g.. K 5 and D.n = 5, we have 25 annotated images. Note that K examples for base classes are not necessary, as discussed in Sec. 2.1.
Our goal is to obtain the segmentation map Y BBCM E yy xW computed using the prediction model for GFSS, denoted as ŪBCM-
3.2 Evaluation metric
The mean intersection--ver- (mloU) is widely used in reporting the performance of segmentation methods [1]. Let us first define the IoU for a class y' E ) as
Σ レ Ty 181)
HW
IY =
(1,)' Uy : -1
WN U.
where and [condd] the indicator function taking 1 cond is true, 0 otherwise. Here, we consider single sample for evaluation, but it can be easily extended to multiple samples by adding the summation over
(y. = -1
3

---

co-ocaurences samples
novel cleas id
Base
Set of
T-shot
dell
9
0
(0, 3.
baze plass 12
5 6
0
4. 5
classes
1,11, 8):3
CO-OcOurennces
nssmange
B (0,1)
(a) Counting co-occurrences
(b) Co-occurrence count table
(c) Base-novel mapping (BNM)
Figure 2: Illustration of BNM creation. For illustration purpose, image of size 3 x 3 is used. (a) Count co-occurrences, i.e., (base class, novel class) pairs. There are three (0, 4) and one (2,4) co-occurrences. (b) Aggregate co-occurrence counts for all samples and create co-occurrence count table. (c) Create BNM from co-occurrence count table, where top-1 strategy finds base class with largest co-occurrences (shaded cell in Fig. 2b) for each novel class.
samples to the numerator and denominator in Eq. (1), respectively. Finally, the mloU is computed by
mloUy(Y, 9) = D Σ
(Y, (Y,)). P).
(2)
y'Ey
For example, y Vn, will be the mIoU over novel classes.
Proposed method
We now present BCM.
4.1 Training
Training is divided into two steps: 1) finding the relationship between base and novel classes, and 2) training models for classifying novel classes for each chosen base class.
Step 1. We input into Zb and obtain Y b. For each pixel of the annotated object, we compare Y, and Y and record co-occurrences of base and novel classes. We then count the co-occurrences, find the top-s co-occurred base class for each novel class, referred to the top-s strategy, and obtain chosen base classes denoted as B. Finally, we construct the mapping from a base class to novel classes, called base-novel mapping (BNM). Figure 2 illustrates the creation of BNM with the top-1 strategy from the 1-shot dataset.
Step For each chosen base class 3 E B, train model ga with the modified K-shot dataset where labels are converted into /3 if they are novel classes irrelevant to 3 or the background. Taking the example in Fig. 2c, when / 1, the irrelevant novel-class labels '4' and '5' and the background label '0' are replaced with '1'. Then, Ø3-1 returns either '1' or '6' as the prediction.
To obtain the learned model G.s, loss or effective losses used in the previous studies.
any learning method, such minimizing the cross-entropy
4.2 Inference
Inference analogous training. For test image X, we first obtain the base-class prediction Y,- For each pixel j. [Y,], 3, we then obtain the prediction the corresponding model ğo and overwrite [Ybly with the output of ğg. Figure illustrates how we obtain the segmentation map of BCM. We summarize the flow of BCM in Fig. 4.
4.3 Preventing catastrophic forgetting
Maintaining the base-class segmentation performance is crucial in GFSS. We theoretically show that BCM perfectly maintains the segmentation performance of most of the base classes without resorting to, e.g., knowledge distillation [22] for training models.

---

3=D
each @ B
2
X
9b
2
Find Index
2
5-
Y3
Yb = B
Overwrite
BCM
98
Feature
Figure 3: Illustration of inference. Yb. Y f=0. and Y BCM are predic- tions of Ib. -8=0. and BCM, spectively.
Figure Flow of BCM. For each B, BCM finds pixels of Y 8 and overwrites Y,, with Y3.
We first formulate the prediction of BCM described in Sec. 4.2. Let Vh be the prediction of the base-class model the j-th pixel and 0g be the prediction of Da at the j-th pixel. The prediction of BCM at the j-th pixel, ÚBCM, is obtained by
Tb (0
ih &
(3)
otherwise.
The above prediction mechanism leads to the following proposition:
Proposition 4.1. Let Y, b and Y BCM be the predictions of the base-class model and BCM, respectively. The mloUs of Y and Y BCM over Vo \ B are the same:
mloUy, Y,) yAl Y BCM).
(4)
If [B] is small, the segmentation performance of most of the base classes is perfectly maintained.
Proof. For any y' E B, if Yf = then ( BCMlj = [Ybl) bl5 by definition of the BCM prediction. Then, for any 1// E 3i, B.
IoU (Y, . = IoU,/ (Y, BCM).
(5)
Taking the average of loU,, over 3A, \ B, we obtain Eq. (4).
Intuitively, since BCM uses the prediction of the base-class model as it is for a subset of base classes, the mloU over those base classes the same as To- Proposition 4.1 shows that BCM partially prevents catastrophic forgetting [28, 29]. In our experiments, [B] tended to be small, resulting in the mloU over base classes being almost maintained.
4.4 Lightweight implementation
Since the size training data for novel classes is not large in GFSS, training deep neural networks for Øø is impractical. We thus use the base-class model 9b as the feature extractor of go and train linear models as the last layer of ğB with the K-shot data.
To train linear models, we can use off-the-shelf libraries, such as Scikit-learn [30], meaning that training time will be fast, compared with the end-to-end training GPU. Since the number of background pixels much larger than that of objects of interest, we applied sampling techniques for imbalanced data [31] training data, such under-sampling.
Regarding the top-s strategy, we used = from the performance and computation time viewpoint. The effect of in the top-s strategy is discussed in Sec. 5.6.
4.5 Further performance improvement
Since our implementation is to train simple linear models, we can easily use various techniques to improve GFSS performance. We explain two effective and easy-to-use techniques used in our experiments follows.
5

---

Pre-processing, We can use Tukey's 's ladder of powers transformation [32], known as the effective transformation in sew- learning [33]. Specifically, let f be the d-dimensional feature vector extracted by lb- Tukey's ladder of powers transformation is defined as
[S" logg
if +
(6)
otherwise,
where is a hyper-paramettr, and the power and logarithm operations are element-wise. When 7 1, the original feature used. In a previous study [33], T = 0.5 was recommended as the default value; thus, used it in our experiments.
These pre-processed feature vectors are used for To only since the change in feature representation for Io without retraining would downgrade performance. Note that to apply similar pre- pre-processing to the existing methods, crucial to take into account the adverse effect on total performance.
Ensemble learning. We can use ensemble learning [34, 35] to improve GFSS performance. Unlike the existing GFSS methods, the computation time of BCM will be short since training a linear model for 98 is lighter than tuning deep neural networks in an end-to-end manner.
We introduce shot-wise ensemble learning few-shot learning when K > 1. This involves first preparing multiple T-shot datasets (I. S K) by drawing samples from the K-shot dataset then aggregating outputs of models trained with the T-shot datasets. In our experiments, we split the 5-shot dataset into five 1-shot datasets and obtained six models by using five 1-shot and one 5-shot datasets. In inference, we computed a weighted average of the outputs of the models. The weights can be determined by, e.g., validation data or pre-defined values. In our experiments, we set one for the model with the 1-shot dataset and five for the model with the 5-shot dataset for simplicity.
5 Experiments
5.1 Setup
Datasets. We used three FSS datasets: PASCAL-5' [4, 36, 37], PASCAL-10" [20, 36, 37], and COCO-20* [6, 38]. The PASCAL-10* dataset was introduced to investigate the impact of increasing the number of novel classes [20].
Methods for comparison. We compared BCM with CAPL [16]. BAM [14, 18], and DIaM [20]. Note that DIaM was regarded as a simple method since it trains the last linear layer only, similarly to the simple methods [39, 40] proposed for few-shot object detection.
Evaluation. We report the mloUs over base and novel classes, referred as the Base and Novel scores, respectively, where the background was not included in the Base score, similarly to a previous study [20]. We also report the average of the Base and Novel scores, called the Mean score. All reported scores are the average of five independent trials.
Base-class model. We used the publicly available pre-trained model for GFSS,2 pyramid scene parsing network (PSPNet) [41] with the pre-trained ResNet-50 backbone [2]. It was trained with labeled data for base classes by using the stochastic gradient descent optimizer with an initial learning rate of 2.5 x 10 momentum of 0.9, and weight decay of 10 The batch size was 12, and number of epochs was 20 for COCO-20f and 100 for PASCAL-5' and PASCAL-10'.
Detailed implementation. The implementation of BCM is based on the publicly available DIaM code. We followed the same data- -oadilg and evaluation procedure and replaced the method part with BCM. Specifically, to train novel-class models ga in Sec. 4.1, we used the logistic regression in Scikit-learn [30].' which uses the L-BFGS-B [42] method with line-search strategy as the default solver. The regularization parameter was determined from the five-fold cross-validation from the ten candidates {10-5 5 10a) The default values were used for the other hyper-parameters.
'hhe details of how can modify BAM to output multiple novel classes are explained in [20].
DlaM: /etps://githhe com/ a www.aann//iii /
We used an accelerated version of Scikit-leamd https: : / wtt.tttbb.ommmite//eeetetoe -ete

---

Table Average mloU over five trials. Base and Novel represent mloU scores over base and novel classes, respectively. Mean shows average of Base and Novel scores, Results of comparison methods were obtained from [20]. All methods use ResNet-50 as backbone.
PASCAL-5
1-shot
5-shot
Method
Base
Novel
Mean
Base
Novel
Mean
CAPL [16]
CVPR'22
64.80
17.46
41.13
65.43
24.43
44.93
BAM [14]
CVPR'22
71.60
27.49
49.55
71.60
28.96
50.28
DIaM [20]
CVPR'23
70.89
35.11
53.00
70.85
55.31
63.08
BCM
(Ours)
71.15
41.24
56.20
71.23
55.36
63.29
MASCAL-10*
CAPL [16]
CVPR'22
53.78
15.01
34.40
57.02
20.40
38.71
BAM [14]
CVPR'22
69.02
15.48
42.25
69.18
21.51
45.35
DIaM [20]
CVPR'23
70.26
31.29
50.77
70.25
51.89
61.07
BCM
(Ours)
70.07
35.94
53.01
70.12
53.49
61.81
COCO-20'
CAPL [16]
CVPR'22
43.21
7.21
25.21
43.71
11.00
27.36
BAM [14]
CVPR'22
49.84
14.16
32.00
49.85
16.63
33.24
DIaM [20]
CVPR'23
48.28
17.22
32.75
48.37
28.73
38.55
BCM
(Ours)
49.43
18.28
33.85
49.88
30.60
40.24
5.2
Main results
Table summarizes the average performance over the five trials for each method in the practical GFSS setting. BCM outperformed the other GFSS methods regarding the Novel and Mean scores. Notably, the Novel scores in the 1-shot PASCAL-5' and PASCAL-10" settings substantially improved with BCM. Regarding the Base score, BCM achieved comparable/best performance thanks preventing catastrophic forgetting, discussed in Sec. 4.3. We discuss these results from the viewpoint of our theory in Sec. 5.4.
These results indicate that BCM achieved the best performance without resorting to various techniques used with the other methods, such as meta-learning [43, 44], information maximization principle [21], and transductive learning [23]. The implementation of BCM was to train the final linear layer only, as described in Sec. 4.4, but we can use cutting-ede edge architectures and training techniques in practice, leading to further performance improvement.
5.3 Ablation study
We investigated the effect of pre-processing (Tukey's ladder of powers transformation) and ensemble learning, explained in Sec. 4.5.
Table shows the performance of four variations of BCM, i.c., with and without pre-processing and ensemble learning. Compared with the results in Tab. 1, BCM without data pre-processing and ensemble learning outperformed the other methods in the 1-shot setting, showing that the simple rule and standard supervised learning improved the GFSS performance. The effectiveness of data pre- pee-cesessing was much higher when the number of novel classes was small (see the PASCAL-5' and PASCAL-10' settings). However, the pre- pre-processing decreased this performance slightly in the 1-shot COCO-20' setting.
Our ensemble-learning approach consistently improved GFSS performance, with a roughly 5% improvement on datasets. Note that we can use standard ensemble-learning approaches in the 1-shot setting, meaning that further performance improvement is possible in practice.

---

20
Table 2: Effect of pre- pre-proeessing and ensemble learning. 'P' and 'E' denote data pre-processing and ensemble learning, respec- tively.
1-shol 5-shot
Frequency
15
PASCAL-5'
1-shot
5-shot
Base
Novel
Mean
Base
Novel
Mean
[A]
71.16
38.13
54.65
71.23
49.83
60.53
(a) PASCAL-5"
71.15
41.24
56.20
71.23
50.83
61.03
71.23
53.53
62.38
1-shot 5-shot
V
/
71.23
55.36
63.
14
Frequency
12
PASCAL-10*
70.07
34.56
52.32
70.12
48.80
59.46
70.07
35.94
53.01
70.12
49.88
60.00
70.12
52.27
61.19
70.12
53.49
61.81
(b) COCO-20*
COCO-20
49.48
18.03
33.76
49.88
26.73
38.30
Figure 5: Frequency of [B] over four splits and five trials. Size of tended to decrease with increasing number of shots.
49.43
18.28
33.85
49.86
26.76
38.31
49.90
30.48
40.19
49.88
30.60
40.24
2012.2
Table 3: Chosen pairs on COCO-20' (5-shot)
PASCAL-5
Inference time [me]
2000
Training time [s]
100
COCO-20'
1500
Base class
Novel class
1000
50
502.5
baseball glove sports ball
5.2
18.0
24.9
17.7
24.6
bed
couch
K-shot
K-shot
tie
person
baseball glove
Figure 6: Training time [s] with Figure 7: Inference time [ms] BCM
on COCO-20"
5.4 Number of chosen base classes
Figure shows the frequency of [B], the number of chosen base classes, over 20 runs (four splits and five trials). Overall, |B] tended to decrease with the increase in K. We hypothesize that noisy pairs appear relatively smaller than frequent pairs when K 5, and the top-1 strategy ignored such a noisy pair. In particular, the median value of the frequency in 5-shot was 1 PASCAL-5 and COCO-20'.
Another observation is that [B] was much smaller than the number of base classes. For example, the largest [3] was four the 5-shot COCO-20* setting, meaning that less than 7%(*= 4/61) of base classes (including background) were chosen in the BCM training step. These results indicate that we do not need to prepare 93 for many base classes, and training and inference times do not increase rapidly to the number of base classes. Moreover, [B] tends to be small in practice, IoUs most of the base classes are perfectly maintained, as shown in Proposition 4.1.
5.5 Which classes were chosen?
We explored which class consists base- and novel-class pairs the BNM and recorded pairs the 5-shot COCO-20* setting. In most cases, the background was chosen, meaning that the base-class model recognized novel-class objects as the background. Sometimes, the pairs summarized Tab. were chosen, showing that the related classes were chosen with BCM. The results empirically confirm

---

Base soote
70
Novel soore
Size of
50
60
40
3
10
1
3
10
2
10
Top-s
Top-s
Top-s
(a) Base score
(b) Novel score
(c) Size of B
Figure 8: Effect of 8 in top-s strategy in 5-shot PASCAL--5' setting
our idea that novel class is classified as the background or a similar base class with the base-class model.
5.6 Effect of top-s strategy
We varied in the top-s strategy and investigated how s affects the segmentation performance and size of B in the 5-shot PÁSCAL-5 setting. The results in other settings can be found in Appendix A.1.
Figures 8a and 8h respectively illustrate the Base and Novel scores with respect to (1, 5, 10), showing that the median and minimum scores decreased as increased.
We also show the size of B in Fig. 8c. The number of chosen base-classes increased, but was upper- bounded by a certain value. Even though we increased s, the chosen base-classes might be the same. addition, the increase in s also led to longer training time. this sense, smaller is preferable from the perspective of computation time.
In summary, higher may result in selecting redundant base classes and cause performance degradation, and larger [ requires many models for 9a, resulting in longer computation time for training. We thus recommend computation time.
since it is the best choice based GFSS performance and
5.7 Computation time
The computation time was measured on a machine equipped with an NVIDIA @ V100, 16 CPU cores, and 32GB memory.
Training time. We plot the training time [s] with BCM in Fig. 6. CAPL and DIaM are not shown since CAPL requires hours of training time due meta-learning, and DIaM, which is based on transductive learning, does not optimize models other than the inference phase. In the 1-shot scenario, training time less than min. The training time increased the number of novel classes increased. Although we used ensemble learning in the 5-shot scenario, the training time the 5-shot COCO-20' setting about 2 min. Note that training was done by CPU computation, meaning that further acceleration is expected by GPU computation.
Inference time. Figure 7 shows the inference time [ms] of CAPL, DlaM, and BCM. Since DIaM is based on transductive learning, the inference time was slower than the inductive methods, i.c. CAPL and BCM. BCM requires computations of the novel-class models in addition to that of the base-class model, but the total inference time was comparable to that the end-to-end model, i.e., CAPL.
Note that BCM has [B] final linear layers for novel classes, leading to subtle slowdown when switching layers, unlike the end-to-end computation CAPL. Our implementation used CPUs for training the final linear layers, Scikit-learn is used, unlike CAPL. on GPU. This device difference might another reason for the slowdown. practice, more sophisticated implementation will shorten the gap between CAPL and BCM.

---

6 Conclusion
Summary. We presented simple yet effective GFSS method called BCM. BCM based on a mapping between base and novel classes and trains novel-class models by simple supervised learning without resorting to meta-learning and the information maximization principle. Since we can use standard supervised learning, training can be done efficiently using off-the-shelf software. We can use the featurizer and base-class model without modifying their weights, enabling us to use cutting-edge public foundation models with BCM. We theoretically showed that the mloU over most of the base classes is perfectly maintained. Through numerical experiments, we demonstrated the superior performance of BCM method against state-of-the- art GFSS methods.
Limitations. BCM has limitations that need to be resolved. First, although the final performance of BCM outperformed the other GFSS methods, the performance improvement in the 5-shot scenario was slight, meaning that there is room for improvement. Second, while BCM perfectly maintains the segmentation performances of most of base-classes, it does not improve such performance using novel-class data. A possible direction to resolve these limitations is to investigate the strategy for creating mapping other than the top-s strategy and use more recent powerful supervised/few--soo learning methods.
Acknowledgments
The authors would like to thank the anonymous reviewers for their helpful comments and suggestions.
References
[1] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE TPAMI, 44(7):3523-3542 2022. 1,3
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6,6
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[4] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In BMVC, 2017. 1.6
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. CANet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In CVPR, 2019.
[6] Khoi Nguyen and Sinisa Todorovie. Feature weighting and boosting for few-shot segmentation. ICCV, 2019. 6
[7] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. PANet: Few-shot image semantic segmentation with prototype alignment. In /CCV, 2019.
[8] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In ECCV, pages 763-778, 2020.
[9] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In ECCV, pages 730-746, 2020.
[10] Malik Boudiaf, Hoel Kervadee, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation without meta-leaming: A good transductive inference is all you need? In CVPR. pages
2021.
[11] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmenta- tion. In CVPR, pages 8312-88 0221, 2021.
[12] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang. Zi-Zhe Song. and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In ICCV, pages 8741 8750, 2021.
10

---

[13] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang. Ruiyu Li. and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE TPAMI, 44(2):0500-1633 2022.
[14] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning what not to segment: new perspective on few-shot segmentation. In CVPR, pages 8057-8067, 2022. 3, 6,
[15] Nico Catalano and Matteo Matteucci. Few shot semantic segmentation: a review of methodologies and open challenges, 2023.
[16] Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. Generalized few-shot semantic segmentation. In CVPR, pages 1 11572, 2022. 1, 2, 6,
[17] Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Learning orthogonal prototypes for generalized few-shot semantic segmentation. In CVPR, pages 11319-11328, 2023. 3
[18] Chunbo Lang, Gong Cheng. Binfei Tu, Chao Li, and Junwei Han, Base and meta: A new perspective on few-shot segmentation. IEEE TPAMI, 4569):50669-100000
2023. 1, 6
[19] Zhihe Lu, Sen He, Da Li, Yi-Zhe Song, and Tao Xiang. Prediction calibration for generalized few-shot semantic segmentation. IEEE TPAMI, 32:3311-3322 133233,,3,,, 2023. 1,3
[20] Sina Hajimiri, Malik Boudiaf, Ismail Ben Ayed, and Jose Dolz. A strong baseline for generalized few-shot semantic segmentation. In CVPR, pages 11269-11278 2023. 1, 2, 3, 6,7
[21] R. Linsker. Self-organization in a perceptual network. Computer, 21 2163)105-117, 1988. 3, 7
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. 3,4
[23] Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 2010. 3,7
[24] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In CVPR, 2020. 3
[25] Ze Yang, Ruibo Li, Evan Ling, Chi Zhang, Yiming Wang. Dezhao Huang, Keng Teck Ma, Minhoe Hur, and Guosheng Lin. Label Label-guided knowledge distillation for continual semantic segmentation on 2D images and 3D point clouds. In ICCV, pages 18601-18612, 2023.
[26] Beomyoung Kim, Joonsang Yu, and Sung Ju Hwang. ECLIPSE: Efficient continual learning in panoptic segmentation with visual prompt tuning. In CVPR, pages 3346-3356, 2024. 3
[27] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman. Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis. Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespic, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko. Giray Ogut, Laurel Orr. Isabel Papadimitriou. Joon Sung Park. Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan. Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohami, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xic, Michihiro Yasunaga, Jiaxuan You. Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. 3
[28] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Academic Press, 1989. 5
11

---

[29] James Kirkpatrick, Razvan Pascanu. Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinshe Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetiing neural networks. Proceedings of the National Academy of Sciences, 114(13):351-3355 -3226, 2017.
[30] Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel. Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-leam: Machine learning in Python. JMLR, 12:2825-2830. 2011. 5.6
[31] Nitesh V. Chawla. Data Mining for Imbalanced Datasets: An Overview, pages 853-867. Springer, 2005.
[32] John Wilder Tukey, Exploratory Data Analysis. Addison-Wesley Publishing Company, 1977. 6
[33] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021.
[34] Trevor Hastie, Robert Tibshirami, and Jerome Friedman. The Elements of Statistical Learning. Springer New York Inc., 2009. 6
[35]
Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms. Chapman & Hall/CRC, 2012.
[36] Mark Everingham, SM Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. IJV,, 111(1):98-13)) 136, 2015.
[37] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014.
[38] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740-755, 2014.
[39] Xin Wang, Thomas Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu. Frustratingly simple few-shot object detection. In ICML, volume 119, pages 9919-9999999 9919999,,, 2020. 6
[40] Ze Yang. Chi Zhang, Ruibo Li, Yi Xu, and Guosheng Lin. Efficient sho- object detection via knowledge inheritance. IEEF Transactions Image Processing, 32:321-334 2023.
[41] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 6
[42] Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23(4):550-560
1997. 6
[43] Jurgen Schmidhuber. Evolutionary principles in self-refcrential learning. On learning now learn: The meta-meta- mneta-meta...-hoo Diploma thesis, Technische Universitat Munchen, Germany, 1987.
[44] Chelsea Finn, Pieter Abbcel, and Sergey Levine. Modcl-agnostic meta-Ieariing for fast adaptation of deep networks. /CML, volume 70, pages 1126 1135, 2017.
12

---

A Appendix
A.1 Effect of top-8 strategy
Figs. and 10 show the effect of in the top-s strategy in the 1-shot PASCAL-5' and COCO-20' settings, respectively. We can observe similar tendency that discussed in Sec. 5.6.
45
Novel scare
Size of B
6
Base
35
2
65
2
10
10
3
5
10
Top-s
Top-s
Top-S
(a) Base
(b) Novel score
(c) Size of B
Figure 9: Effect of s in top-s strategy in 1-shot PASCAL-5' setting
score
50
Novel score
20
Size of
48
10
46
2
10
10
10
Top-s
Top-5
Top-s
(a) Base
(b) Novel score
Size of B
Figure 10: Effect of in top-s strategy in 1-shot COCO-20* setting
A.2 Example when > 1
We explain obtaining the prediction when 1.
Even when s > the inference procedure works as explained in Sec. 4.2 since a single base class is mapped to multiple novel classes in BNM. We show the case when = 2 with tiny example: the base classes are '0' and '1', and the novel class is '2'. Suppose that we have the following BNM when = 2 in Tab. 4. This table shows when the base class '0' is mapped the novel class "2' and '1' is also mapped to '2', In this case, we have the two models: gæ=0 returns '0' or '2', and g==1 returns '1' or '
Table 4: Example BNM when s =
Base class
Set of novel classes
2
For each pixel, the base-class model outputs either '0" or '1'. We then compute the prediction of the corresponding model and overwrite it. Since BCM does need overwrite the same pixel multiple times, we can straightforwardly combine predictions of for the final prediction.
A.3 Broader impact
The idea behind BCM will positively affect future studies on few-shot learning. Our future work will lead to more powerful visual understanding systems. Regarding negative societal impact, we expect BCM will not have direct path to harmful applications. However, harmful actors may maliciously use visual understanding systems. To prevent such malicious use of technology, we need pay attention to events in our society.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We stated the main claims based on the results of the theoretical analysis and numerical experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should diseuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provided the proof of the theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclosed the details of settings and learning procedures in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: However, we will release the code after we get our organization's 's permission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ sellic/guides (oaiies//oddSummiii ssi ssionPoiccy) for more details.
While we encourage the release of code and data, we understand that this might not possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCode/uddeiiiiiiiceeee / //nips. soublic/guides guiiiiieeiid//oo iss oniooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We wrote the details. Also, we used the pre-trained models and data loading procedure provided by the publicly-availabee repository of existing method.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Partially yes. We used the box plot to show the deviation of the performance on some experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap. etc.)
The assumptions made should be given (c.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclosed the computing resources used our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ctw.idelsllllllecccccccccc NeurIPS Code of Ethics https: /ttpps:://euurre htt///nurrppps/p// http:/rrrrp..ccppppcceeeepp htcp/uubll/thhhhhccideeeeeeeeee ines?
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consider- ation due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed broader impacts in Appendix A.3.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We will not release data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recogmize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited the assets used in paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. com/ /ataatts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "ol3I8TMEY3", "reviewer": "Reviewer_HhLp", "summary": "The paper presents a new and efficient BCM technique aimed at tackling the issue of generalized few-shot semantic segmentation. It identifies how base and novel classes relate to each other by examining the overlap between the base model's predictions and the true labels of the novel images. Utilizing these insights, the approach trains new models for the novel classes, allowing them to better differentiate from similar base classes, which in turn enhances the segmentation results.", "questions": "please refer to the weakness section.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The writing in the paper is easy to understand and direct. \n\nBesides, The performance is commendable, and it also maintains a pleasing level of efficiency. \n\nThe approach appears to be general and independent of the model's architecture.", "weaknesses": "Given that the base-novel mapping (BNM) facilitates a transition from base to novel classes, there's no necessity to further refine the base-class model and its feature extractor, nor to adjust their weights.\n\nThe BCM employs inductive learning instead of transductive learning, which results in swift inference and also ensures that the training process of the BCM is both quick and user-friendly.\n\nThe quantitative assessment appears to indicate that the performance of the BCM is state-of-the-art when juxtaposed with traditional methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. \nIt appears you have listed some of the strengths in the weaknesses section.\nIf you forgot to copy and paste questions from your memo, e.g., questions about explanations in our paper, please let us know during the discussion.\nWe would like to improve the presentation of our paper through your comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lfmsn8DexQ", "reviewer": "Reviewer_NeFB", "summary": "This work introduces an interesting method for generalized few-shot segmentation. Unlike previous methods that mainly focus on meta-learning, the proposed method maintains the performance of base classes while achieving decent performance for novel classes. Feature pre-processing and model ensembling techniques are used to further enhance performance. Experiments on PASCAL-5i, PASCAL-10i, PASCAL-20i, and COCO-20i show better or comparable performance to SOTA methods.", "questions": "Typo: line 43, \"beneficial, especially.\"", "limitations": "The authors have already discussed their limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed method is simple but effective.\nExperimental results are generally good.\nThis work provides a new perspective on maintaining base class performance in GFSS. The performance for base classes that are less relevant to novel classes is exactly maintained by design.", "weaknesses": "1. It should be clarified whether the co-occurrences matrix and BNM are calculated per-dataset or per-batch.\n2. Since the proposed method uses feature pre-processing and model ensembling techniques, the comparison may be somewhat unfair.\n3. The written language can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Please find our answers to your questions below.\n\n> It should be clarified whether the co-occurrences matrix and BNM are calculated per-dataset or per-batch.\n\nThe current implementation computes BNM per dataset.\nWe will clarify this point in the final version.\n\n\n> Since the proposed method uses feature pre-processing and model ensembling techniques, the comparison may be somewhat unfair.\n\nWe presented the ablation study, showing that our method without feature pre-processing and ensemble learning techniques achieved better novel-class segmentation performances in the 1-shot setting.\n\nThe existing methods often use meta-learning, information maximization principle, and transductive learning in addition to the simple supervised learning method, e.g., minimization of the cross-entropy loss.\nEach existing method uses various techniques to improve its performance further.\nGiven that, our comparison is reasonable from the viewpoint of standard practice.\n\nFrom another perspective, the advantage of our method is that we can use feature pre-processing and model ensemble learning techniques at a low cost, as shown in Figure 6 (training time).\nPre-processing to feature vectors, rather than input images, would downgrade the performance of the existing methods.\nEnsemble learning with the existing methods causes computation issues.\nFor instance, ensemble learning further slows down the speed of the existing methods based on transductive learning, such as DIaM.\n\n\n> The written language can be improved.\n\nWe will revise the written language.\n\n> Typo: line 43, \"beneficial, especially.\"\n\nThank you very much. We fixed the typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lrZsvHZz1b", "reviewer": "Reviewer_MDAS", "summary": "The authors propose a simple yet effective method termed base-class mining (BCM) for GFSS that does not employ the techniques mentioned earlier in the existing method. Experiments show some improvements on COCO-20i, PASCAL-5i and PASCAL-10i.", "questions": "1. When s > 1, each novel class may have multiple mapped base classes. In this case, how to combine the final predictions for the novel classes is not elaborated.", "limitations": "No negative societal impact is expected.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article is well-written.\n2. The proposed method is simple yet effective too some extent.\n3. The computational cost is low as the approach only requires to update several final linear layers.", "weaknesses": "1. The overall novelty is relatively limited, as the idea that \"a novel class is classified as the background or a similar base class by the base-class model\" has already been explored in continual semantic segmentation [1], where novel class \"lake\" is classified as base class \"water\" before learning the new class lake.\n2. The improvement across different settings is relatively marginal.\n3. The computational cost is about 1.5 folds compared to CAPL as in Fig. 7. From my view, the base model is shared among different g_beta. The only computation increase should only be the added final linear layers, which is roughly neglectable. Therefore, I expect more explanations for the increase.\n\n[1] Kim, Beomyoung, Joonsang Yu, and Sung Ju Hwang. \"ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Please find our answers to your questions below.\n\n\n> The overall novelty is relatively limited, as the idea that \"a novel class is classified as the background or a similar base class by the base-class model\" has already been explored in continual semantic segmentation [1], where novel class \"lake\" is classified as base class \"water\" before learning the new class lake.\n\nThank you for pointing out the paper [1].\nIn [1], a novel class is classified as a base class, and then misclassification is corrected by post-processing, so-called logit manipulation.\nIn contrast, we explicitly integrate the idea into the model architecture, which vastly differs from [1].\nThe architecture allows us to see which novel class is related to which base class through BNM, as illustrated in Figure 2(c).\nMoreover, we showed that our method prevents catastrophic forgetting in theory (Proposition 4.1) and improved performance, particularly in the 1-shot setting.\nAlthough the two settings (continual learning in panoptic segmentation [1] and GFSS) have some overlap, achieving better performance by [1] in GFSS would not be straightforward since the number of training samples is quite limited in GFSS, particularly in the 1-shot setting.\nWe believe that our paper contains various quantities that contribute to the *overall* novelty.\n\nWhen writing our paper, we did not realize [1] since it does not consider GFSS, and it was published very recently in CVPR2024 and appeared on arXiv roughly a month before the submission deadline.\nIn the final version, we will cite [1].\n\n\n> The improvement across different settings is relatively marginal.\n\nThe improvement of the Novel score in the 5-shot PASCAL-$5^i$ is marginal, but except for that, our method improved the Novel score by at least 1% and a maximum of 6%.\nFrom the viewpoint of inference time, our method is much faster than DIaM, as shown in Figure 7.\nConsidering these points, the results of experiments would be sufficient to show the effectiveness of the proposed method against the existing methods.\n\n\n> The computational cost is about 1.5 folds compared to CAPL as in Fig. 7. From my view, the base model is shared among different g_beta. The only computation increase should only be the added final linear layers, which is roughly neglectable. Therefore, I expect more explanations for the increase.\n\nOur method has $|\\mathcal{B}|$ final linear layers for novel classes, leading to a subtle slowdown when switching the layers, unlike the end-to-end computation of CAPL.\nAlso, our current implementation uses CPUs for the final linear layers, as Scikit-learn is used, unlike CAPL on GPU.\nThis device difference might be another reason for the slowdown.\nBesides, other miscellaneous things may cause about six milliseconds of slowdown.\nNevertheless, the current implementation would show the usefulness of our method.\nIn the final version, we will mention the above points and explain that a more sophisticated implementation will shorten the gap between CAPL and our method.\n\n\n> When s > 1, each novel class may have multiple mapped base classes. In this case, how to combine the final predictions for the novel classes is not elaborated.\n\nEven when s > 1, the inference procedure works as it is since a *single* base class is mapped to multiple novel classes in BNM.\n\nWe show the case when s > 1 with a tiny example: the base classes are 0 and 1, and the novel class is 2.\nSuppose that we have the following BNM when s = 2:\n| Base class | Set of novel classes |\n| ---- | ---- |\n| 0 | 2 |\n| 1 | 2 |\n\nThis table shows when the novel class 2 is mapped to *two* base classes, 0 and 1.\nIn this case, we have the two models: $g_{\\beta=0}$ returns 0, or 2, and $g_{\\beta=1}$ returns 1 or 2.\nFor each pixel, the base-class model outputs either 0 or 1.\nWe then compute the prediction of the corresponding model $g_\\beta$ and overwrite it.\nSince our method does not need to overwrite the same pixel multiple times, we can straightforwardly combine predictions of $g_\\beta$ for the final prediction.\nTo improve the clarity of our paper, we will add the above explanations in the final version.\nWe hope that this answer will resolve your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by three experts in the field. The recommendations are (Weak Accept x 2, Borderline Accept). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns (especially the unclear paper writing and presentation by Reviewer NeFB, technical novelty claim and comparisons with previous literature by Reviewer MDAS) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (poster)