OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation
Authors: Dongjun Hwang
Yejin Kim
Minyoung Lee
Seong Joon Oh
Junsuk Choe
Venue: NeurIPS 2025 poster ()
Keywords: Open-Vocabulary
Segmentation
Continual Learning
Primary area: applications
TL;DR: We propose an Mixture-of-Experts based continual learning method for Open-Vocabulary Segmentation to handle sequentially collected datasets.

Abstract: Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.

OVS Meets Continual Learning: Towards Sustainable O Open-Vocabulary Segmentation
Dongjun Hwang!
Yejin Kim
Minyoung Lee!
Seong Joon Oh
Junsuk Choe
'Sogang University
2 University Tübingen
Tübingen Al Center
Abstract
Leen-Voabullary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each them has clear limitations. To address these issues, we propose ConOVS, novel continual learning method based Mixture-of Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre- treitrann incremental, and zero-shot datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially. Code is available https://github.. http:///ww...cooooo///////////n/
Introduction
In fields such robotics [9] and autonomous driving [24, 47], there is growing demand for models that can segment novel objects not included in the training dataset. However, conventional closed-set segmentation models, which are restricted to recognizing only the classes seen during training. fall short in meeting this demand. To address this limitation, Open- Vocabulary Segmentation (OVS) has emerged, aiming enable segmentation of unseen classes that are not included in the training dataset. OVS continues to be an active area of research, particularly through methods that leverage foundation models such CLIP [52, 58].
Most previous studies [49, 52, 55, 59] on OVS assume scenario in which the model is trained once using a pre-training dataset. However, in practice, trainable datasets often arrive sequentially as new data are collected over time. Considering this setting, we first discuss how existing OVS models perform under such conditions. To facilitate a clearer discussion, we measure the relative performance of OVS models on seen and unseen classes using a reference baseline. We adopt OneFormer [18] for this role. It represents the state-of-the-art in closed-set segmentation and shares the same ConvNeXt backbone [27] as the OVS model [52], enabling fair comparison in model capacity.
The most straightforward approach is to use the existing OVS model as is. In our experiments (Figure 1a), the existing OVS model achieves 86.4% of OneFormer's performance
the pre-training dataset COCO [26]. In contrast, its performance drops to 46.9% on new dataset ADE20K [57], which contains unseen classes as well. These results indicate that the OVS model fails to perform well on datasets it has not encountered during training.
Conresponding author (Jachoelsogang ac sc..k kt).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DVS pre-trained solely
Base ine
Fine tuning
Retraining
ConOVS (ours)
ADEZOK zero-shet dataset.
LVIS (2ero-shot)
cityscapes
200
COCO
ADE20K
BDD100K (zero-shot)
(Incremental)
ADE2OK Mccementall
150
100
0%
3%
Mapillary
20.5
44.0
Relative Po
125
88.8%
(zero shot
26 0
23:5
100
COCO
86.4% +ADE20K
39,3
pre-training
75
PC $4.5
T5.9
13,3
(zero:shot)
50
100.0%
79.8%
95 0%
94 4%
847 (zera shot)
PC 459 (zero-shot)
90.9
PAS 21 (zero-shot)
One Former
Hin-tuneCCnOOS (ours)
PAS 20 (zero-shot)
(b)
Figure 1: (a) Comparison of the performance of the OVS model (fe-clip [52]), Retraining, Fine-tuning, and ConOVS against the closed-sct segmentation model OneFormer. (h) Performance of the Baseline (fe-clip (52]), Fine-tuning, Retraining, and ConOVS on the pre-training, incremental, and zero-shot test datasets. PQ used.
To determine whether this performance gap is due to the inherent difficulty of the unseen classes or simply because the model has been trained on them, we retrain the OVS model using both the pre-training dataset and the new dataset. As shown in Figure 1a, the model's performance on ADE20K improves significantly from 46.9 46.9% 79.8% relative to OneFormer. This result confirms that the low performance on the new dataset is primarily due to the lack of exposure during training. also suggests that this limitation the OVS model can be effectively mitigated by training on newly collected data.
However, retraining the model from scratch demands substantial computational resources. par- ticular, this approach becomes impractical when the pre-training data is no longer accessible or computational resources are limited. To address these limitations, we consider an alternative ap- proach: transfer learning. Specifically, we fine-tune the pre- rre-trained OVS model on a new dataset. However, as shown in Figure and 1b, this approach also has a limitation. leads to performance degradation not only on the pre- vre-training dataset but also on zero-shot tasks. This issue appears to stem from well-known drawback fine-tuning, namely, catastrophic forgetting. Therefore, we consider continual learning (CL) methods, which are designed to address catastrophic forgetting. However, most CL. approaches are developed under the assumption that the number of classes is finite, making them unsuitable for open- voccbvouabllryy tasks where the number of classes be potentially infinite [60, 61].
As result, in scenarios where new datasets are continuously collected-as as assumed in this paper-itt still unclear how to effectively utilize the incoming data, and finding viable solution in OVS remains a non-trivial challenge. To address this, we propose ConOVS, based continual learning method that incrementally trains an existing OVS model on new datasets. Our method begins by fine-tuning the pre-trained OVS model to build a distinct expert for each new dataset. During inference, we estimate the probability that a given input sample is close to the distribution of each training dataset, based on their statistical representations. The model then computes an interpolation factor from these probabilities and dynamically combines the experts by interpolating their weights. This allows method to produce optimal model for predicting each input sample.
oixtture-of-Experee (MoE)
To simulate the scenario assumed in this paper, we sequentially introduce incremental datasets to an existing OVS model and evaluate the resulting models on three validation sets: the pre-training dataset, the incremental dataset, and zero-shot datasets. As shown in Figure 1b, our method not only significantly improves performance on the incremental dataset compared to standard retraining and fine-tuning, but also consistently enhances performance on both the pre-training and zero-shot datasets. Furthermore, compared to existing continual learning methods, our approach achieves superior performance across all three evaluation settings.

---

2 Related Works
2.1 Open- Open-Vocabulary Segmentation
Recent open- vven-vocabulary segmentation (OVS) research has focused on leveraging models capable of open-vocabulary classification, such as CLIP [36]. to recognize classes that are not included in the training dataset. For example, fc-elip [52] identifies unseen classes by combining class embeddings from the model's decoder with those from CLIP. Moreover, a recent study [35] has explored an approach that retrieves LoRA modules trained on different datasets according to the input and utilizes them in conjunction with CLIP. Other methods further enhance the recognition of unseen classes by either applying visual grounding techniques like GradCAM [40] to CLIP [29, 42, 58] or distilling knowledge from both CLIP and the segmentation foundation model Segment Anything Model (SAM) [43, 54]. Meanwhile, there are also OVS approaches that do not rely on CLIP. For instance, methods such as X-Decoder [55, 62, 63] train both the encoder and decoder from scratch using segmentation datasets along with large-scale image- pair datasets.
Most existing OVS studies are based on scenario in which the model is trained only once. However, this setting inherently limits performance on unseen classes (see Section 1). To overcome this limitation, we analyze strategies for training OVS models in scenario where new datasets are introduced sequentially.
2.2 Continual Learning
Acquiring additional knowledge in an already trained model is not straightforward. When a model is further trained on new data, often tends to forget previously learned information while learning the new content [30]. This phenomenon is widely known as catastrophic forgetting. To address this issue, the field of continual learning (CL) has emerged, CL. explores methods that enable models to learn from new data while retaining prior knowledge.
CL techniques are typically categorized into three types. First, replay-based methods store a subset of previously seen data and retrain the model using it to preserve prior knowledge [2, 38]. Second, regularization-base methods introduce penalty terms in the loss function to constrain parameter updates. preventing significant deviations during training on a new dataset [1, 21, 25]. Third, parameter-isolation based methods mitigate interference by freezing previously learned parameters and allocating separate parameters for learning new data [20, 46]. Several approaches extend this idea into a fixtuure-of-Experee (MoE) framework, where additional parameter sets are treated as distinct experts, and gating module selects the appropriate expert based on the input [22, 45].
However, existing CL methods are designed under the assumption that the number of classes is finite, which limits their applicability in open-vocabulary settings [56, 60, 61]. Therefore, there need for novel approaches that enable continual learning in Open-Vocabular Segmentation (OVS) scenarios, where new data are introduced incrementally. To address this, we propose a novel MoE-based continual learning technique that effectively expands the capacity of OVS models.
Motivation
In this section, we expand on the discussion from Section 1 and explore in greater detail how newly collected datasets can be leveraged improve the performance of OVS models.
The most straightforward approach is to retrain the model from scratch using a joint dataset that combines the original and newly collected data. In practice, this strategy effectively preserves performance on seen classes while substantially improving performance on unseen classes. However, it suffers from two major limitations: (1) it incurs significant computational costs, as the model must be retrained from scratch every time new data are added; and (2) retraining becomes entirely infeasible if access to the original dataset has expired.
Due to these limitations, fine-tuning the model using only the newly collected dataset may appear to be a practical alternative. However, this approach compromises the model's 's original performance. As shown in Figure 2a, fine-tuning the OVS model results a significant drop in performance not only on the pre-training dataset but also the zero-shot test dataset. Qualitative examples provided in the
3

---

neremental Dataset (Cityscapes)
COCO
Citscappe (fine-tuning)
ADE20k (zero-shot)
70
SOTA (closed set) Cityscapes
100
(pre-train)
65
SOTA (closed set on COCO
80
SWC Fine tuning
After fine-tuning
20.1
Retraining
40
Bascline
-2277
Retraining
20
-10.3
Fine-tuning
ECUPSE
Continual Learning
Before Fine-tuning Model
After Fine-tuning Model
25
35
40
45
55
60
Pre-training Dataset (COCO)
(a)
(b)
Figure 2: (a) Performance degradation the pre-training and zero-shot datasets after fine-tuning. fe-clip is used. (b) Comparison the performance of OneFormer [18], the baseline (fe-clip [52]), retraining, fine-tuning, existing continual learning methods [20, 21, 25), and ConOVS on the pre-training and incremental datasets. All methods use the same iterations. FQ is used.
Appendix further illustrate this phenomenon. This degradation likely caused by a well-known issue in fine-tuning, known catastrophic forgetting [21, 25].
Another potential direction to apply continual learning (CL) methods to OVS models. However, most existing CL methods are built on the assumption of a finite set of classes, making them difficult to directly apply to open-poocceeee vocabulary tasks [56, 60, 61]. For instance, [1, 20] apply CL segmentation tasks by treating all unseen classes as background, which fundamentally conflicts with the goal of OVS models that aim to recognize potentially unlimited categories.
Even when existing CL methods are adapted for OVS (see Appendix A.2 for implementation details), our experimental results show that their effectiveness limited. As shown in Figure 2b, OVS models trained with adapted CL methods perform significantly worse than the closed-set segmentation model OneFormer on both the pre-training and incremental datasets. We believe this arises because existing CL methods assume a closed-set set segmentation with finite label space, whereas OVS involves potentially infinite label space, which these methods do not account for.
To address these issues, improves the performance of OVS models. Specifically, ConOVS (1) reduces training cost by using only newly collected data, unlike retraining: (2) avoids catastrophic forgetting, unlike fine-tuning; and (3) effectively improves performance on the incremental and zero-shot test dataset, unlike existing CL methods.
we propose ConOVS, new continual learning method that sequentially
4 Background
Open- Vocabulary Segmentation (OVS) aims predict segmentation mask-class pairs from input image King and text description Tyxx which may include both seen (trained) and unseen classes. OVS models typically consist of three components: image encoder, text encoder, and decoder, denoted as f= [mmg frett f Jaec)- The image encoder fing produces an image embedding Zimg- and the text encoder fiesi produces text embedding Ziext- These fed into the decoder faec. which, given N learnable object queries, outputs pairs of predicted masks and class embeddings, (m,, c,) Each m, predicted mask, and e; its associated class embedding. Final class labels are assigned by matching each e; to the most similar text embedding.
Continual Learning Setup. We consider continual learning scenario in which datasets containing new classes arrive sequentially, and the set of seen classes gradually expands over time. The model pre-training dataset D pre, and then incrementally updated using sequence of datasets Dine,1; Pmm. ... At each time step f E {1, ... n). the model is trained only on D Dine.t without access to Dpee, Dimc,,, D-m The class set C, from each incremental dataset is added to the previously seen class set, resulting in Cree = U' s C, U Cpre The model is evaluated on test sets of all datasets up to time to assess both its ability to learn new classes and retain prior knowledge. To additionally evaluate generalization, we use zero-shot test set Drmm containing unseen classes Cunson C Crotal \ Coeen that never appeared during training.
is first trained

---

OVS model
CS (ours)
Algorithm 1 Interpolation factor estimator
class
mask
(b) Mixture-of- Expertsss
Require: Input (Trimg Prea). encoders
dec
Jimgs fex.- decoder sase: MVN parame-
dec,
der
den
ters { sing: P ) i==== PDF p(:)
dec,ine
Ensure: Interpolation factor
Zsmg
Pleart
Au
X
1: Extract embeddings: Zime < Sime(
dec.pp
t- Sueu (x
Estimate likelihoods: Limg f- {p(ziie
Jimg
Stent
A
Vime)] Lust + [pp(ziest Bisx))
T Input image
(a) Interpolation factor estimator
3: Compute: Pine f
softmax softmax(lime)
Input text
Puatt softmax
return max(Pimg:
Combine: max (Ping Prese)
Figure 3: Overview of the inference process of our proposed method.
The Proposed Method: ConOVS
In this section, we propose ConOVS, a novel MoE-based continual learning method designed to train OVS models in scenarios where new datasets are sequentially collected. For clarity, we describe the proposed method in two parts: Training Phase and Inference Phase.
5.1 Training Phase
During training, we derive expert models and multivariate normal (MVN) distributions for each dataset. Specifically, we first train an OVS model from scratch using the pre-training dataset. Then, we fine-tune only the decoder on each incremental dataset to obtain expert model specific to that dataset. For each dataset, we also compute the mean and covariance matrix of the image and text embeddings, which define the MVN distributions. These are represented as 4 img (Ping ing)) and
& = (Mient Ejgg) for each dataset E (0. .n]. Here, 0 corresponds to the pre-training dataset, while E {1,..., ,n} refers to the incremental datasets.
5.2 Inference Phase
We perform inference by dynamically combining expert models based on the MVN distributions derived during training. Specifically, we first compute task vectors v; for each expert model, defined as the arithmetic difference between the decoder weights of the i-th incremental expert Odecine and the pre-trained decoder weights Quec.r- Given an input sample, we feed the image Tang and class descriptions Frex into the image and text encoders, respectively, embeddings Zing and Ziex- We then evaluate the likelihoods of these embeddings under the MVN distributions for all datasets, and collect them into the vectors ling. hext E R=+1
obtain the corresponding
After that, apply the softmax operation to the log- likeelioooood vector to normalize the proximity scores of each domain into the [0, 1] range. This decision motivated by a prior study [16], which reported that merging performance degrades when the interpolation factor exceeds 1. Finally, compute the element-wise wise maximum of the two probability vectors to obtaín the final interpolation factor vector À R .+1 The detailed procedure is provided in Algorithm 1, and ablation studies the choice of softmax and element-wise maximum are presented in Appendix F.
The final decoder weights Odec.eew are computed as:
i dec Oseccpr
+
That is, the decoder is dynamically constructed by linearly combining task vectors v, with interpola-
tion weights X; relative to the pre-trained decoder (see Figure 3b). Note that while Xg is not directly used in this computation, is included in the softmax operation and thus indirectly affects the other À elements. As result, when the input is close to the pre-training distribution, Åo approaches 1, pushing the remaining X; values toward
The effectiveness and justification of this design are empirically validated in Section 6.
5

---

Table I: Comparison of performance across Baselines (fe-clip, X-Decoder), Retraining, Fine-tuning, four existing continual learning methods, and ConOVS when the incremental dataset is Cityscapes (b) ADE20K. PQ used.
(a) Cityscapes
(b) ADE20K
COCO
Citysaaes (incremental)
ADE20K (zero-shot)
COCO
ADE20K (incremental)
Cityscapes (zero-shot)
Method
Method
(pre-training)
(pre-training
Jo-clip
50.1
44.0
23.5
fe-clip
50.1
23.5
44.0
Fine Fine-tuning
-2.77
+20.1
10.3
Fine luning
-7.7
+24.1
-3.0
Retraining
+0.6
+17.9
+1.7
Retraining
X
+1.4
+16.5
-1.2
ER
-1.6
+19.0
+0.3
ER
+0.4
+21.5
-3.5
LwF
-10.7
+12.2
LwF
-3.8
+13.7
-1.0
EWC
-2599
-2.6
ECLIPSE
ECLIPSE
-5.9
ConOVS (ours)
+0.3
+20.2
+2.5
ConOVS (ours)
+1.7
+23.8
+0.9
X. Decoder
56.7
36.3
16.7
X Decoder
56.7
16.7
36.3
Fine-tuning
-5044
+26.6
.1.9
Fine-tuning
-37.3
+28.2
.3.7
ConOVS (ours)
-0.4
+26.6
+0.1
ConOVS (ours)
-1.5
+29.2
+1.4
6 Experiments
Learning Sequences. This study assumes scenario where trainable datasets arrive sequentially and evaluates OVS models that are incrementally trained on them. In the main paper, we examine three learning sequences. In Scenario 1 (S1), the model is pre-traue pre-trained on COCO [26], incrementally trained on Cityscapes [7], and evaluated on ADE20K [57] as the zero-shot test set. In Scenario (S2), the model is again pre-trained COCO but incrementally trained ADE20K, with Cityscapes used for zero-shot evaluation. In Scenario (S3), the model is pre-trained on COCO and incrementally trained on both Cityscapes and ADE20K. For zero-shot evaluation,
a diverse collection of datasets: LVIS [10]. BDD100K [51]. Mapillary Vistas [33], PC-59, PC-459 [31], PAS-20, PAS-21 [8], and A-847 [57]. We further validate our method on larger number of incremental datasets in Scenario 4 (S4), with the results provided in Appendix E. Evaluation is conducted on the test sets of the pre-training and incremental datasets, as well as the designated zero-shot test sets.
Implementation Details. We apply our method to two OVS models: fc-clip with ConvNeXt-L [27] and X-Decoder with Focal-L [50]. During the pre-training phase, fc-clip trains only the decoder, while X-Decoder trains both the encoder and decoder. In the fine-tuning phase, both models train only 0.01, and log-likelihood is used to compute
the decoder. The temperature T' in the softmax is probabilities from the MVN distributions. All experiments are run on two NVIDIA A5000 GPUs.
Metrics. We evaluate panoptic, instance, and semantic segmentation using PQ, mAP, and mIoU, respectively. Due to space constraints, we report only PQ in the main paper, with the others in the Appendix J. Some zero-shot test datasets support only specific segmentation tasks; for example, LVIS supports only instance segmentation. In such cases, we evaluate performance only on the supported task.
Evaluation
6.1 Main Results
In this section, we compare the performance of the proposed ConOVS and other approaches under the three scenarios. We first analyze the results for scenarios S1 and $2 followed by scenario S3. We then provide a more in-depth analysis of our method, including an investigation into the behavior of the interpolation factors. All methods were trained with the same number of iterations to ensure a fair comparison, and detailed information on the training cost of each method is provided in Appendix D.I.
In scenarios S1 and S2, where only a single incremental dataset is used for training, our method consistently outperforms existing approaches across all datasets, whether the incremental dataset ADE20K or Cityscapes (see Table 1). particular, compared to retraining, our method almost maintains or even improves performance on the pre-training dataset, despite not using it during additional training (e.g., Retraining: +1.4 vs. Ours: +1.7 in S2). It also achieves superior perfor- mance the incremental dataset itself (c.g., Retraining: +16.5 vs. Ours: +23,8 in S2). Moreover,

---

compared to fine-tuning and conventional continual learning, our method improves performance on the incremental dataset without compromising performance on the pre-training dataset. This improvement is attributed to the dynamic interpolation of expert models in our method, which helps mitigate catastrophic forgetting.
Our method also achieves the best performance on the zero-shot test dataset. For instance, in scenario S2, performance on the Cityscapes improves by +0.9, whereas all other methods show performance drops. This result indicates that our method enhances recognition of a wider range of classes while preserving previously learned knowledge.
In scenario S3, our method consis- tently achieves superior performance compared to both fine-tuning and re- training. Specifically,
Table 2: Performance comparison scenario S3. The best per- formance for each dataset is underlined. "City "City->DEE" means fine-tuning Cityscapes first, then ADE20K, PC is used.
shown in Table 2, fine-tuning performs well only on the most recently trained dataset, whereas our method consis- tently achieves strong results on all three datasets. By contrast, retrain- ing shows lower performance than our method, likely due to its need for more iterations converge. In comparison, our method yields better results with the same number of training iterations, demonstrating greater training efficiency. Note that the analysis related to the number of training iterations in retraining is provided in Appendix G.3.
Learning
COCO
ADE20K (incremental)
Cityscapes
Method
Sequence
(pre-training)
(incrmmental)
fe-elip
50.1
23.5
44.0
Fine-tuning
ADE - City
20.8
15.4
65.2
Fine-tuning
City ADE
39.3
48.3
46.0
Retraining
COCO, City ADE
48.6
35.5
60.5
ConOVS (ours)
City, ADB
51.6
47.0
64.3
Table Performance comparison on unseen datasets in scenario S3. The best performance for each dataset underlined. PQ is used.
Learning Sequence
BDD10OK (PQ)
Mapillary (mIoU)
PC-59
PC-459
PAS-21
A-847
Method
(mAP)
(mIoU)
(mIoU)
(mIoU)
(mIoU)
(mIoU)
fc-clip
20.5
19.0
26.0
53.0
16.9
93.1
80.2
13.8
Fine-tuning
City - ADE
21.7
19.7
27.8
52.1
17.2
92.3
76.7
16.0
Fine-tuning
ADE - City
10.4
21.3
24.2
45.9
13.5
87.4
70.7
11.5
Retraining
COCO, City, ADE
21.5
21.8
28.0
53.2
17.3
93.3
80.9
15.2
ConOVS (ours)
City, ADE
23.1
22.6
29.1
54.9
17.9
93.6
80.7
16.3
In addition, our method also consistently outperforms other approaches in various zero-shot eval- uations. As shown in Table 3, achieves superior performance across all eight zero-shot test datasets. This result suggests that the dynamic interpolation of expert models in method facilitates recognition of a broader range of unseen classes
Table Comparison of performance on seen unseen classes in zero-shot test dataset ADE20K. mIoU used. (b) Comparison of PQ. SQ, and RQ between fc-clip and ConOVSS in the zero-shot test dataset ADE20K.
(b)
Method
Seen Classes
Method
RQ
fc-clip
35.0 (+0.0)
28.6 (+0.0)
fe-clip
23.5 (+0.0)
61.7 (+0.0)
28.3 (+0.0)
ConOVS (ours)
37.9 (+2.9)
30.9 (+2.3)
ConOVS (ours)
25.9 (42.4)
73.1 (+11.4)
31.2 (+2.9)
Evaluation of the Truly Unseen Classes. Some classes in the zero-shot test datasets may overlap with those in the training data. For instance, ADE20K shares 38 of its 150 classes with COCO. To more accurately assess zero-shot performance, we separately evaluate the model on truly unseen classes that do not appear in the training data. Therefore, split ADE20K into seen and unseen subsets and measure performance on each in scenario S1.
As shown in Table 4a, our method improves performance by similar margin on both seen and unseen classes (seen: +2.9, unseen: +2.3). This suggests that the performance gain is not solely from improved recognition of seen classes, but also reflects better generalization to unseen classes.
7

---

(a) Distribution of the interpolation factors on A-847
(b) Distribution of the interpolation factors on BDD
2000
COCO
1000
COCO
Cityscapes ADE20K
Cityscape
Court
Count
800
ADEE0K
1500
200
0.2
0.4
0.5
0.8
1.0
0.2
0.4
0.6
0.8
1,0
Interpolation
Interpolation Factor
(c) Example of Interpolation Factors
(oooo, City, ADEZOK]
6
dec Jnc
Legend
10,00,0. 0.10 0.99
d
des nare
Mistao Exparria
Interpolation Factor Estimator
Sample from A-847
Training Data Distribation
Unsocn Data Distribution
[6 λI, Aal
6 dec
10,21,0.4 0.47, 0.36]
Sample from BDD100K
O devenes
O 0
Embed Space
arc,ine
Figure 4: Interpolation factor behavior across different input sample distributions.
Analysis of Improvements in Unseen Classes. To better understand the source of performance improvements unseen classes, we analyzed results on the zero-shot dataset ADE20K by comparing the PQ, SQ, and RQ scores of the baseline and our proposed method. As shown in Table 4b, incorporating ConOVS into the baseline model improves both PQ and RQ. The most notable gain, however, is observed in SQ, which evaluates the quality of the predicted segmentation masks. These results indicate that the improvements in unseen classes are primarily driven by enhanced segmentation quality rather than improved mask classification.
Understanding the Behavior of the Interpolation Factor. We analyze how the proposed method adapts to different imput sample distributions. To this end, we examine the distribution of interpolation factors À estimated by the interpolation factor estimator across two zero-shot test datasets. One is A-847, which shares a similar distribution with the incremental training dataset ADE20K, and the other is BDD100K, which differs significantly from all training datasets.
As shown in Figure 4a, the interpolation factors for A-847 tend to be close to 0 or In particular, the expert trained ADE20K receives value close to 1, while other experts receive values close O. This shows that when input samples are similar to a previously trained distribution, our method selectively activates the corresponding expert to maximize performance (see Figure 4c top-right).
In contrast, as illustrated in Figure 4b, the interpolation factors for BDDI00K are more evenly distributed between 0 and 1. This suggests that the input samples do not clearly belong any of the known training distributions.
such cases, our method disperses the À values to avoid over-reliance on single expert. Instead, it combines the weights of multiple experts based the probability that the input sample belongs to each distribution. This allows the model to leverage knowledge from various datasets and produce more accurate predictions even for samples from unfamiliar domains (see Figure 4c bottom-right).
6.2 Ablation Study
In this section, we conduct ablation studies to analyze the contribution of each component in the proposed method. All experiments are conducted in scenario S1.
Ablation Study of Image and Text Distribution. Our method computes the interpolation factor of an input sample using the MVN distributions of image and text embeddings for each training dataset. To analyze how the interpolation factors are affected by the distribution design, we compare three configurations: image only, text only, and combined image-text.
As shown in Table 5a, using both image and text distributions yields the best performance on the incremental dataset. This suggests that combining both modalities enables more accurate estimation of the input sample's 's proximity to training distributions, leading to better expert selection.

---

Table 5: (a) Comparison of the interpolation factor estimator when using both image and text distributions versus using only one of them, PQ is used. (b) Performance comparison when the MVN distribution is replaced with K-means clustering or KDE. fe-clip and PQ are used.
(b)
COCO (pre-training)
Cityscapes (incremental)
ADE20K
COCO (pre-training)
Cityscapes (incrementa))
ADE20K
Distribution
Methods
(zero-shol)
(zero-shol)
image only
43.4
25.8
k-means clustering
text only
$1.9
60.7
25.9
kemnel density estimation
48.1
57.4
26.1
image text
51.6
64.3
26.0
MVN distribution
50.4
64.3
26.0
Evaluating Alternative Approaches against the MVN Distribution. We evaluate and compare two alternative techniques to the MVN distribution used in our method for estimating interpolation factors. Specifically, we replace the MVN distribution with K-means clustering or Kernel Density Estimation (KDE), and analyze the resulting performance changes. Detailed descriptions of the K-means and KDE are provided in the Appendix B.5.
As shown in Table 5b, both K-means and KDE yield lower performance the pre-training and incremental dataset. These results suggest that the MVN distribution enables more accurate estimation of interpolation factors for in-distribution data. We attribute this to its relatively simple structure and low dimensionality, which make it less sensitive to outliers than K-means or KDE.
Replacing Softmax with Argmax. The proposed method uses the softmax function to compute interpolation factors for each dataset. We compare the performance on eight zero-shot datasets when replacing the softmax function with the argmax operation. Table 6 presents the evaluation results. The experimental results show that softmax consistently outperforms argmax across all zero-shot datasets (e.g., LVIS, argmax: 21.3, softmax: 23.1).
Table 6: Performance comparison between the argmax and softmax operations the interpolation factor estimator. We use fe-clip with our method and fine-tune it on both Cityscapes and ADE20K. PQ is used.
Incremental
LVIS
BDD100K
Mapillary
PC-59
PC-459
PAS-20
PAS-21
A-847
Decision
Dataset
(mP)
(PQ)
(mIoU)
(mIoU)
(mIoU)
(mIoU)
(mIoU)
(mIoU)
Argmax
Cityscapes, ADE20k
18.3
26.9
53.1
17.0
80.2
16.3
Sollmax
Citscapes, ADE20L
23.1
22.6
29.1
54.9
17.9
93.6
80.7
16.3
Specifically, datasets such as LVIS and BDD100K, softmax demonstrates clearly superior perfor- mance. However, for PAS-20, PAS-21, and A-847, the performance difference between softmax and argmax is minimal. This occurs because, when the input sample is close to the distribution of the pre-training or incremental dataset, the interpolation factor obtained from softmax tends to close to 0 or 1. As result, softmax behaves similarly to argmax.
Hyperparameter Sensitivity Analysis. Our method uses softmax operation to compute the interpolation factor, and we analyze the effect of the softmax temperature hyperparameter T'. The temperature T directly influences the distribu- tion of the interpolation factor: a low T smooths the factor values, while a high pushes them toward extreme values of 0 1. Table sum- marizes how this behavior affects performance.
Table 7: Effect softmax temperature on perfor- mance across datasets. mIoU is used.
COCO
ADE20K
Cityscapes (zero-shot)
Total
(pre-training)
(incrementa))
0.0001
50.7
35.4
43.8
129.9
42.2
43,9
137.3
47.5
43.2
142.0
1.0
51.2
47.4
43.2
141.8
When T is small, the interpolation factor X be- comes overly smoothed, which prevents the ex- pert models for each dataset from being utilized. This leads to performance degradation on the incremental dataset. In contrast, when T is large, À converges to values close to or 1, resulting in the selective use of a single expert model. This degrades performance on the zero-shot dataset. These findings suggest that appropriately integrating multiple models is essential for effective generalization to zero-shot datascts, and that extreme interpolation factors hinder this process.

---

Decoder Interpolation. Unlike our method, which fine-tunes the entire decoder for each dataset, existing MoE-based continual learning methods [22, 45] primarily adopt Visual Prompt Tuning (VPT), where only small subset of parameters is trained for each incremental dataset. This approach differs from ours two key aspects: expert models consist of only partial decoder parameters, and single expert is selected at inference time instead of performing interpolation. To assess the effectivvness of our full decoder fine-tuning strategy, we replace with the VPT-based approach and compare their performance.
Specifically, we implement the prompt tuning method based on [45] as follows: (1) for each incremental dataset, we train only the decoder's object queries and positional embeddings and store them in a prompt pool; (2) during infer- ence, we compute interpolation factors for each dataset using the same procedure as our method; (3) we identify the dataset with the highest inter- polation factor; and (4) retrieve the correspond- ing object queries and positional embeddings from the prompt pool and apply them to the decoder for prediction.
Table 8: Performance comparison when the decoder interpolation our method replaced with visual prompt tuning-based approach. fc-elip and PQ are used.
COCO (pre-training)
Cityscapes (ineremental)
ADE20K
Method
(zero-shot)
Prompt Tuning
43.3
48.9
24.4
Decoder Interpolation
50.4
64.3
26.0
As shown in Table 8 and the experimental results, the prompt tuning variant consistently under- performs our method across pre-training, incremental, and zero-shot datasets. This suggests that full decoder fine-tuning enables more effective adaptation to new datasets compared to VPT, which is constrained by its limited number of trainable parameters. Moreover, interpolating multiple experts provides greater flexibility and representational power than selecting a single expert, further supporting the advantage of our approach.
7 Limitation
Our method generates a unique decoder weight for each input sample, which can limit its applicability when the inference batch size exceeds one-a common constraint in other MoB-based continual learning approaches [41, 45]. However, since only the decoder varies per input and the encoder is shared across samples, the encoder can process inputs in batches. The resulting embeddings are then decoded individually using their corresponding weights. This design reduces the batch size limitation by supporting batched encoder processing and per- per-sample decoding.
Conclusion
This paper identifies the performance limitations of existing Open-Vocabee Vocabulary Segmentation (OVS) methods on unseen data, an aspect that has been largely overlooked in prior work. To address this issue, we introduce a new learning scenario which newly collected datasets are incrementally used to further train the OVS model. Under this setting, show that conventional approaches such as retraining, fine-tuning, and continual learning are either impractical or difficult to apply effectively.
To overcome these challenges, we propose ConOVS, novel MoE-based continual learning method for OVS. In ConOVS, predictions are made by dynamically combining the decoders of expert models based on the probability that the input sample belongs to the distribution of each training dataset. We validate the effectiveness of our method through extensive evaluations across various sequential learning scenarios and compare it against existing approaches. Experimental results show that ConOVS consistently achieves superior performance on pre- tre-training, incremental, and zero-shot test datasets, demonstrating its ability to effectively expand the recognition capability of OVS models.
Broader Impacts. The proposed method can be applied to real-world applications such robotics, where new objects continuously appear in the environment. However, if the pre-training dataset is biased, the model may continue to produce skewed predictions even after additional training, as explicitly designed to preserve previously learned knowledge. It is therefore important to be aware of this characteristic of the proposed technique, as lack of such awareness may lead to unexpected model behavior.
10

---

Acknowledgement
We would like to thank Yeji Park, Beomyun Kwon, and Joonkyung Kim for the insightful discussions and valuable feedback during the development of this work. This work was partly supported by the Institute of Information Communications Technology Planning& & Evaluation (ITTP) grant funded (2025-25441311 Professional AI Talent Development
by the Korea (overnmen((MSIT) (No. Program for Multimodal AI Agents, Contribution: 50%) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-003504 Mitigating Hallucinations for Trustworthy Large Vision-I Language Model: Datasets, Evaluation, Learning, and Inference, Contribution: 50%).
References
[1] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9233-9242, 2020.
[2] Sungmin Cha, YoungJoon Yoo, Tacsup Moon, et al. Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. Advances in neural information processing systems, 34: 10919- 109900,,0,--00000 2021.
[3] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.
[4] Zhiyuan Chen and Bing Liu. Lifelong machine learning. Springer Nature, 2022.
[5] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open- vecabulayy object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16901-16911, 2024.
[6] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41 13-4123,, 2024.
[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiole. The cityscapes datasct for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 321 3213-3223, 2016.
[8] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
[9] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, al. Conceptgraphs: Open-vor vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021 5028. IEBE, 2024.
[10] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5356-5364, 2019.
[11] Martin Hahner, Dengxin Dai, Christos Sakaridis, Jan-Nico 7aech, and Luc Van Gool. Semantic under- standing of foggy scenes with purely synthetic data. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3675-3681. JEEE, 2019.
[12] Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Shruti Bhosale, Hsien-Hsin Lee, Carole-Jean Wu, and Benjamin Lee. Toward efficient inference for mixture of experts. Advances Neural Information Processing Systems, 37:84033-84059, 2024.
[13] Dongjun Hwang, Jung-Won Ha, Hyunjung Shim, and Junsuk Choc. Entropy regularization for weakly supervised object localization. Pattern Recognition Letters, 169:1-7, 2023.
[14] Dongjun Hwang, Hyoseo Kim, Doyeol Back, Hyunbin Kim, Inhye Kye, and Junsuk Choe. Curriculum learning with llass- composition for weakly supervised semantic segmentation. Pattern Recognition Letters, 188:171-110 177, 2025.
11

---

[15] Dongjun Hwang, Seong Joon Oh, and Junsuk Choe. Small object matters in weakly supervised object localization. Neurocomputing, page 130494, 2025.
[16] Gabriel Iharco, Marco Tulio Rihciro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint S..iv:2222200005555 2022.
[17] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23- 23-28,22 2020, Proceedings, Part XVI 16, pages 699-715. Springer, 2020.
[18] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[19] Rawal Khirodkar, Brandon Smith, Siddharthha Chandra, Amit Agrawal, and Antonio Criminisi. Sequential ensembling for semantic segmentation. arXiv preprint arXiv:2210.05387 2022.
[20] Beomyoung Kim, Joonsang Yu, and Sung Ju Hwang. Eelipse: Efficient continual learning in panoptic segmentation with visual prompt tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3346-3356, 2024.
[21] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barsska-a et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
[22] Minh Le, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Ngo, Nhat Ho, al. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, 37: 1199005519966 2024.
[23] Minhyeok Lee, Suhwan Cho. Jungho Lee, Sunghun Yang. Heeseung Choi, Ig-Jae Kim, and Sangyoun Lee. Effective sam combination for open- vocabulary semantic segmentation. Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[24] Kaican Li, Kai Chen, Haoyu Wang. Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In European Conference on Computer Vision, pages 406 423. Springer, 2022.
[25] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analyxis and machine intelligence, 40( 2):29333322997 2017.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014,
[27] Zhuang Liu, Hanzi Mao. Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrlll, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11 1976-11986, 2022.
[28] David Lopez-Paz Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[29] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, pages 23033 23044. PMLR, 2023.
[30] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
[31] Roozbeh Mottaghi. Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho. Seong-Whan Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
[32] Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, 160:306-336, 2023.
[33] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference computer vision, pages 49900499999. 2017.
12

---

[34] German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
[35] Reza Qorbani, Gianluca Villani, Theodoros Panagiotakopoulos, Marc Botet Colomer, Linus Härenstam- Nielsen, Mattia Segu. Pier Luigi Dovesi, Jussi Karlgren, Daniel Cremers, Federico Tombari, et Semantic library adaptation: Lora retrieval and fusion for open- semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9804-9811 2025.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8765. PMLR, 2021.
[37] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 1114, 2016, Proceedings, Part II 14, pages 102-118. Springer, 2016.
[38] David Rolnick, Arun Ahuja, Jonathan Schwarz. Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
[39] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty- aware evaluation for semantic nighttime image segmentation. Proceedings of the IEEE/CVF international conference on computer vision, pages 7374-7383, 2019.
[40] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via mudient--bseed localization. Interna- tional journal of computer vision, 128:336-359, 2020.
[41] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and 7solt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 1999-11919, 2023.
[42] Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip rnn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13171-13182, 2024.
[43] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mchrdad Farajtahar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel. and Hadi Pouransari. Sam-clip: Merging vision foun- dation models towards semantic and spatial understanding, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 3635-3647, 2024.
[44] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.
[45] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Syxtems, 35: 5682-5695, 2022.
[46] Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139-149,700 2022.
[47] Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, and Raquel Urtasun. Identifying unknown instances for autonomous driving. In Conference on Robot Learning, pages 384-393. PMLR, 2020.
[48] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-22 23998. PMLR, 2022.
[49] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with eext-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966 29662996.777 2023.
[50] Jianwei Yang. Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. Advances in Neural Information Processing Systems, 35:4203-44 4217, 2022.
13

---

[51] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[52] Qihang Yu, Ju He, Xueqing Deng. Xiaohui Shen, and Liang- Shang-Chichhe Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances Neural Information Processing Systems, 36, 2024.
[53] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-tast learning, Advances in neural information processing systems, 33:5824 5836, 2020.
[54] Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulan vocabulary sam: Segment and recognize twenty-thousand classes interactively. arXiv preprint arXiv:2401.0293 2024.
[55] Hao Zhang. Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020-1031, 2023.
[56] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision languae models. In Proceedings the IEEE/CVF international conference on computer vision, pages 19125 19136, 2023.
[57] Bolci Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-322 -21, 2019.
[58] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. European Conference on Computer Vision, pages 696-712. Springer, 2022.
[59] Chaoyang Zhu and Long Chen. survey on open- vocabulary detection and segmentation: Past, present, and future. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[60] Zhen Zhu, Weijie Lyu, Yao Xiao, and Derek Hoiem. Continual learning open-vocabulan classification with complementary memory systems, arXiv preprint or arXiv:2307.044 000.00007,, 2023.
[61] Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classification. In European Conference on Computer Vision, pages 269 225.. Springer, 2024.
[62] Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl. Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15116-15127, 2023.
[63] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li. Jianfeng Wang, Lijuan Wang. Jianfeng Gao. and Yong Jac Lec. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide a clear and accurate account of the paper's main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The limitations are clearly discussed in separate Limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide necessary details, including hyperparameters and training setups, to ensure reproducibility of the main results.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We disclose all necessary details for reproducibility, including code, and training scripts in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (ttpss//nips.cc/uublcc
Suides/CodeSubmissioooooie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.ccppublc/// http:/giiddee/oeSeueiiiiiioo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail all training and evaluation settings, including data splits, backbones, and hyperparameters, along with the rationale behind their choices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars since running multiple trials for every experimental setup would require substantial computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma еrror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the implementation details section, we provide the necessary information to reproduce our experiments. All experiments were conducted using two NVIDIA A5000 GPUs.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://eerris.ccccuuuiffffccccc
Answer: [Yes]
Justification:
We rigorously follow the NeurlPS Code of Ethics in all aspects of our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both potential positive and negative societal impacts are discussed in the Broader Impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [NA]
Justification: Our paper does not involve the release any models or datasets with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in this work are publicly available and properly credited, with licenses and usage terms respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paperswithcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: ducibility.
We ensured that the model, code are well documented for clarity and repro-
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
20

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work does not use LLMs in any important non-standard way.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ttps://earp..cccoootteeeee/222 for what should should not be described.
21


Reviews:
[{"review_id": "yzhUhYQtuq", "reviewer": "Reviewer_hsbu", "summary": "The paper proposes a continual learning framework for open-vocabulary segmentation (OVS), addressing the practical scenario where data arrives incrementally over time and the model needs to adapt to new datasets. The authors introduce a method that trains a separate mask decoder for each incoming dataset and combines these decoders in a Mixture-of-Experts (MoE) manner during inference. Compared to full model retraining, the proposed approach is more cost-effective. It also mitigates catastrophic forgetting more effectively than fine-tuning, and improves performance on both incremental and zero-shot test datasets—surpassing existing continual learning methods on OVS.", "questions": "These questions come from comments on weaknesses\n\n- Could this approach also be applied to OVD?\n\n- Why does continual learning yield better performance than retraining from scratch as shown in Table 1?\n\n- The paper assumes that each incremental dataset belongs to a different domain, which may not hold true in all scenarios. What happens if we do not know the domain of each incoming dataset? For example, what if the incoming data at a given time step is a mixture of multiple domains (e.g., ADE20K and Cityscapes)? How robust is the method in such situations?\n\n- If an incremental dataset is small or weakly labeled, how does that affect model performance? Will the proposed approach still perform well in such low-resource settings?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- While other OVS methods focus on training a segmentation model once and applying it everywhere, this paper addresses the scenario where data arrives incrementally over time and adapts the OVS model using the new incremental data. This paradigm has been shown to be effective for both incremental and zero-shot dataset performance. The observation aligns with prior work such as [1], where incorporating additional data during training also led to performance improvements in OVS.\n\n- The integration of continual learning with a Mixture-of-Experts (MoE) strategy for OVS is particularly novel and well-motivated. A standard OVS model typically consists of three components: a text encoder, an image encoder, and a mask decoder. This work proposes learning a separate mask decoder for each incoming dataset and combining their weights during inference. This design is shown to be highly effective, as demonstrated in Tables 1, 2, and 3.\n\n- The paper is clearly written and well-organized, making it easy to follow the motivation, methodology, and results.\n\n[1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\n\nWeaknesses:\n- The framework seems generalizable to many tasks, and the paper focuses only on OVS. However, there is no specific design for segmentation. A closely related line of work is Open-Vocabulary Detection (OVD). Could this approach also be applied\nto OVD?\n\n- As shown in Table 1, the continual learning model outperforms the retrained model. Why does continual learning yield better performance than retraining from scratch?\n\n- The paper assumes that each incremental dataset belongs to a different domain, which may not hold true in all scenarios. What happens if we do not know the domain of each incoming dataset? For example, what if the incoming data at a given time step is a mixture of multiple domains (e.g., ADE20K and Cityscapes)? How robust is the method in such situations?\n\n- If an incremental dataset is small or weakly labeled, how does that affect model performance? Will the proposed approach still perform well in such low-resource settings?", "was_revised": true, "final_justification": "The rebuttal has further clarified and provided more insights to my questions. It's a good paper and I'm fine to raise my rating.", "rebuttal": "Thank you for your thoughtful review and for highlighting the novelty and clarity of our approach. We address your questions regarding generalizability and robustness in detail below.\n\n---\n\n### **Q1) Could this approach also be applied to OVD?**\n\nIn general, our proposed method is not limited to open-vocabulary segmentation (OVS) but can also be extended to open-vocabulary detection (OVD) frameworks. \n\nOur approach is applicable to models with an encoder-decoder architecture that perform classification based on the similarity between image and text embeddings. This structural characteristic is shared by most OVD methods. For instance, YOLO-World \\[1\\] adopts a modular design consisting of an encoder and a prediction head, and omits the use of a conventional fc-based classifier. Given this structure, all components of our method can be directly incorporated without modification. \n\nTo be more specific, for each newly introduced training dataset, an MVN distribution can be formed using the encoder's embeddings. During inference, the interpolation weights can be dynamically adjusted based on the proximity between the input sample and each domain, enabling the model to incrementally extend its recognition capability.\n\nWe believe that applying our method to OVD models represents a promising direction for future work. Accordingly, we plan to include this discussion in the newly added *Future Work* section following Section 7 (*Limitation*).\n\n> \\[1\\] Cheng, Tianheng, et al. *YOLO-World: Real-time Open-Vocabulary Object Detection.* CVPR 2024\\.\n\n---\n\n### **Q2) Why does continual learning yield better performance than retraining from scratch, as shown in Table 1?**\n\nIn our experiments, the retraining method showed relatively lower performance because we conducted all methods under the same computational budget. Since retraining involves learning from a significantly larger amount of data compared to our method, it requires a longer training schedule to reach convergence. \n\nTo analyze this more precisely, we conducted additional experiments where the retraining method was trained for longer durations. As shown in the table below, when trained with a sufficiently long schedule (100k iterations), the retraining method outperforms our method on the pretraining and incremental datasets.\n\n| method | training iterations | coco (pretraining) | cityscapes (incremental) | ade20k (zero-shot) |\n| :---- | :---- | :---- | :---- | :---- |\n| retraining | 10k | 50.7 | 61.9 | 25.2 |\n| retraining | 20k | 50.7 | 62.5 | 25.4 |\n| retraining | 40k | **51.0** | 63.3 | 25.3 |\n| retraining | 100k | **51.0** | **64.4** | 25.5 |\n| ConOVS (Ours) | 10k | 50.4 | 64.2 | **26.0** |\n\nHowever, despite consuming significantly more computational resources, the retraining method does not yield substantially better performance than ConOVS. One possible explanation for the limited performance of retraining is task interference \\[2,3\\], which might occur when training on multiple domains simultaneously. In contrast, ConOVS avoids this issue by training domain-specific experts independently and combining them adaptively at inference time. As a result, our method achieves comparable performance than retraining, while requiring significantly less computational cost.\n\nAdditionally, we acknowledge that Table 1 in the original manuscript does not clearly convey how the training cost of retraining compares with other methods. To address this, we have included a comparison of training time and GPU memory usage across all methods. Please refer to our response to Reviewer yU8U’s Q1 for more details.\n\nWe recognize that this important explanation was not included in the original draft and have now added the relevant discussion to Section 6.1 (*Main Results*) and the caption of Table 1\\. We appreciate the constructive feedback.\n\n> \\[2\\] Yu, Tianhe, et al. *Gradient surgery for multi-task learning.*, NeurIPS 2020\\.   \n\\[3\\] Chen, Zhao, et al. *Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.*, ICML 2018\\.\n\n### **Q3) The method assumes each incremental dataset is from a distinct domain, which may not always be the case. How does it handle mixed or unknown domains during training?**\n\nAs suggested by the reviewer, we conducted an additional experiment to examine whether our method remains effective when a single incremental dataset consists of multiple domains. To this end, we combined Cityscapes and ADE20K into a single incremental dataset. Note that the performance on the zero-shot dataset was computed by averaging the results from four datasets: PC-59, PC-459, PAS-20, and PAS-21.\n\nAs shown in the table below, our method preserves the performance of the baseline fc-clip on the pre-training dataset, while improving performance on both the incremental and zero-shot datasets. This demonstrates that our method remains robust and effective even when the incremental dataset includes samples from diverse domains.\n\n| method | coco (pretraining) | cityscapes (incremental) | ade20k (incremental) | Average on 4 zero-shot datasets |\n| :---- | :---- | :---- | :---- | :---- |\n| fc-clip | 50.1 | 44.0 | 23.5 | 60.8 |\n| ConOVS (ours) | **50.1** | **60.4** | **31.5** | **61.3** |\n\nWe will include this analysis in the final version of the paper. We appreciate the helpful feedback.\n\n---\n\n### **Q4) How does the method perform when the incremental dataset is small or weakly labeled? Is it robust in low-resource settings?**\n\nTo examine whether the proposed method remains effective in low-resource settings, we conducted an additional experiment using a small incremental dataset. Specifically, we randomly sampled only 5% of the Cityscapes training dataset and used it as the incremental dataset.\n\nAs shown in the table below, our method successfully maintains the performance on the pre-training dataset while improving performance on both the incremental and zero-shot datasets, even when trained with the small incremental dataset. This demonstrates that the proposed method remains effective in low-resource settings where the number of incremental samples is limited.\n\n| Method | COCO (pre-training) | Cityscapes (incremental) | ADE20K (zero-shot) |\n| :---- | :---- | :---- | :---- |\n| fc-clip | 50.1 | 44.0 | 23.5 |\n| ConOVS (Ours) | 50.0 | **48.8** | **25.9** |\n\nWe also considered evaluating our approach under weakly labeled incremental settings. However, due to the lack of publicly available datasets that contain appropriate weak annotations, such as pseudo labels for panoptic segmentation, we were unable to conduct this experiment.\n\nNevertheless, we believe that our method can also be applied in weakly labeled settings. In fact, we consider our approach to be effective in overcoming the common issue of low-quality segmentation annotations found in weakly labeled datasets. This is because our technique constructs the MVN distribution using only image and text embeddings, without relying on segmentation annotations. Therefore, we believe there are no constraints in forming the MVN distribution, and we expect to accurately estimate the interpolation coefficients. While label noise may affect the overall performance, we believe our method can still contribute to effectively enhancing the model’s recognition ability even in weakly labeled environments.\n\nWe will include the experiment and analysis regarding small incremental datasets in Section 6 (*Experiments*), and the discussion on weakly labeled datasets will be added to the *Future Work* section. We appreciate your thoughtful feedback."}, {"review_id": "NwRnpdK8zM", "reviewer": "Reviewer_yU8U", "summary": "This paper proposes a continuous learning method, ConOVS, to address the challenges of Open Vocabulary Segmentation (OVS) models in sequential data collection scenarios. ConOVS based on the MoE framework, achieves dynamic adaptation through the following mechanisms:\n- Training phase: Fine-tune the decoder for each new dataset, generate independent experts, and calculate the multivariate normal distribution (MVN) of the image/text embeddings.\n- Inference stage: Generate interpolation factors based on the similarity between the input samples and the distribution of each dataset (through MVN probability estimation), and dynamically combine the weights of the expert decoder.\nThe effectiveness of ConOVS was verified in three scenarios (COCO pre-training + 20K incremental data from Cityscapes/ADE) and multiple zero-shot datasets (such as LVIS, BDD100K). The results show that it outperforms the baseline method on pre-trained, incremental and zero-shot datasets, and alleviates the problem of catastrophic forgetting.", "questions": "see weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- Experiment:\nThe segmentation performance was evaluated on over 10 datasets, covering multiple scenarios (single/multiple incremental datasets) and models (fc-clip, X-Decoder). The performance shows the enhance in performance.\n\n- In-depth analysis:\nThe ablation experiment verified the key designs (such as the MVN distribution being superior to K-means/KDE, and the image + text modal combination being superior to the single modal).\n\nWeakness：\n- Although ConOVS claims to reduce retraining costs, the paper lacks a demonstration on the training cost comparison betwing retraining / SFT and continuous learning. Maybe detailed training time / resource / computational cost analysis is needed.\n\n- The experimental setup relies primarily on only two or three incremental datasets (Cityscapes and ADE20K). The lack of evaluation under longer learning sequences (e.g., involving 5 to 10 incremental datasets) leaves open questions regarding the scalability and robustness of the method in handling high diversity and complex domain shifts over time.", "was_revised": true, "final_justification": "The author's response solves my concern. I keep my rating as 4: Borderline accept.", "rebuttal": "Thank you for highlighting the strengths of our work, particularly the thorough experimental validation and in-depth analysis. Below, we provide detailed responses to your concerns.\n\n---\n\n### **Q1) The paper lacks a comparison of training costs between retraining, SFT, and continual learning. A detailed analysis of time and resource usage is needed.**\n\nAs the reviewer mentioned, the original draft did not include an explicit comparison of training costs across methods, which we acknowledge as a critical omission. Although our intention was to demonstrate the efficiency of ConOVS by showing that it achieves better performance than retraining with similar computational cost, this point was not sufficiently supported without the training cost comparison. Thank you for highlighting this issue.\n\nTo address this, we measured the training time and GPU memory usage required by each method and summarized the results in the table below. Note that all values are reported as relative figures normalized to the computational cost of standard fine-tuning.\n\n| Method         | Scenario 1 (Cityscapes) |       | Scenario 2 (ADE20K) |       |\n|:--------------:|:-----------------------:|:-----:|:-------------------:|:-----:|\n|                | Training Time           | GPU Memory | Training Time     | GPU Memory |\n| Fine-tuning    | 1.00                    | 1.00       | 1.00              | 1.00       |\n| Retraining     | 1.25                    | 0.86       | 1.40              | 0.95       |\n| ER             | 1.25                    | 0.86       | 1.40              | 0.95       |\n| LwF            | 1.28                    | 1.07       | 1.21              | 1.13       |\n| EWC            | 1.07                    | 1.00       | 1.01              | 1.03       |\n| ECLIPSE        | 0.75                    | 0.57       | 0.74              | 0.52       |\n| ConOVS (Ours)  | 1.00                    | 1.00       | 1.00              | 1.00       |\n\n\nAccording to this comparison, the training time does not vary significantly across methods. This is expected, as we fixed the number of training iterations to ensure a fair comparison. One exception is ECLIPSE, which shows lower training cost. This is due to its use of Visual Prompt Tuning, which updates only a small subset of model parameters. While this leads to higher training efficiency, our results indicate that ConOVS outperforms ECLIPSE in terms of overall segmentation performance. We attribute this performance gap to the inherent limitations of updating only a fraction of the model parameters.\n\nWe will include this discussion and the updated table in Section 6 (*Experiments*) and Table 1\\. We appreciate your thoughtful feedback.\n\n---\n\n### **Q2) The use of only 2–3 incremental datasets limits evaluation of scalability and robustness under longer learning sequences and diverse domain shifts.**\n\nWe agree that evaluating our method with a larger number of incremental datasets is important for demonstrating generalizability. To address this, we designed a new scenario where the model is sequentially trained on five datasets: COCO, Cityscapes, ADE20K, BDD100K, and Mapillary Vistas. While we aimed to include more datasets, we were limited to those providing panoptic segmentation annotations.\n\nThe results of this extended experiment are shown in the table below. As can be seen, our method outperforms the baseline, fine-tuning, and retraining approaches. These results indicate that our method remains effective as the number of incremental datasets increases, demonstrating its scalability under complex domain shifts.\n\n|  | pre-training | incremental | |  |  | zero-shot |  |  |  |  |  | Average |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | COCO | Cityscapes | A-150 | BDD100K | Mapillary | LVIS | PC-59 | PC-459 | PAS-20 | PAS-21 | A-847 |  |\n| fc-clip | 50.1 | 44.0 | 23.5 | 19.0 | 26.0 | 20.5 | 53.0 | 16.9 | 93.1 | 80.2 | 13.8 | 40.0 |\n| fine-tuning | 31.7 | 55.5 | 28.7 | 25.8 | 36.3 | 17.7 | 49.8 | 16.3 | 90.0 | 73.9 | 14.0 | 40.0 |\n| retraining | 49.0 | 62.3 | 36.3 | 28.2 | 34.5 | 21.7 | **53.6** | **17.4** | **94.0** | **80.7** | 15.6 | 44.8 |\n| ConOVS (Ours) | **50.1** | **62.9** | **38.5** | **29.1** | **35.0** | **21.8** | **53.6** | 17.3 | 93.2 | 80.4 | **16.0** | **45.3** |\n\nWe will include this experiment and its discussion in Section 6.1 (*Main Results*). Thank you for the valuable feedback."}, {"review_id": "NPhrZzIDji", "reviewer": "Reviewer_5TFJ", "summary": "This paper presents ConOVS, a MoE-based framework for continual learning in the field of open-vocabulary segmentation. The paper first suggests baseline methods, being fine-tuning and retraining, to incorporate additional training data, which suffer from catastrophic forgetting and declines in performance for its original pre-trained dataset. In this regard, the authors propose ConOVS, which dynamically mixes the weight for the segmentation decoder during inference based on the encoder representation statistics. The authors demonstrate clear advantages compared to baselines, while also providing various experiments settings and ablations to verify the design choice.", "questions": "**[Q1]** Could the authors provide additional explanation or analysis on how the unseen classes are improving? \n- To the reviewers perspective, one possible explanation might be that the fine-tuned models on the incremental datasets might have improved in identifying segmentation masks, as it would be hard to expect the mask classification to have improved. One possible approach to examine this could be inspecting the RQ and SQ for the panoptic datasets, and observe which is improving or not.\n\n**[Q2]** Could the authors ablate the choice for only fine-tuning the decoder? Furthermore, could the proposed method be applicable to cost-based OVS methods that fine-tune the encoder[1,2,3], e.g. CAT-Seg[1]? \n- The fine-tuning of the encoders for cost-based methods[1,2,3] would fundamentally differ from X-Decoder as these models fine-tune the encoder solely on the segmentation dataset, whereas X-Decoder maintain the image-caption objective during its training.\n- Considering that these cost-based methods are able to avoid catastrophic forgetting while directly fine-tuning the encoders of CLIP, it would be interesting to observe how the effects of ConOVS to be complementary in terms of preventing forgetting issues.\n\n**[Q2]** Given that the retrained baseline is expected to underperform due to not having sufficient training iterations, how much could retraining improve when it has fully converged?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**[S1]** The paper presents a well-founded motivation for incremental learning on the open-vocabulary setup, and constructs solid baselines for comparison, with the proposed method showing solid gains over the baseline. \n\n**[S2]** The proposed solution is simple, yet sensible for solving the task. Furthermore, while it may be trivial for a MoE-based approach to prevent catastrophic forgetting, it is impressive to observe gains in unseen classes, where none of the \"experts\" should excel at.\n\n**[S3]** The paper is generally well-written with helpful figures and to understand the framework, as well as providing solid experiments to validate the choices for estimating the interpolation factor. \n\n**Weaknesses**\n\n**[W1]** Some of the results, especially regarding the gains for unseen classes, are not well justified. \n- While it is impressive to see the model improve on unseen classes, it is also counterintuitive to see the combined model to improve in such classes as none of the base models would be effective in identifying unseen classes. While the authors claim that the proposed method is generalizable, it is not explained where the gains are actually coming from.\n\n**[W2]** Some of the design choices are not well justified.\n- The method decides to freeze the encoder while only fine-tuning the decoder, without much justification or ablations. This can further relate to examining the applicability to streamline of works that focuses in fine-tuning the encoder of CLIP[1,2,3], which tend to show much better performance to its frozen counterparts, e.g. fc-clip.\n\n[1] Cho S, Shin H, Hong S, Arnab A, Seo PH, Kim S. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 4113-4123).\n\n[2] Xie B, Cao J, Xie J, Khan FS, Pang Y. Sed: A simple encoder-decoder for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2024 (pp. 3426-3436).\n\n[3] Lee M, Cho S, Lee J, Yang S, Choi H, Kim IJ, Lee S. Effective SAM Combination for Open-Vocabulary Semantic Segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 26081-26090).", "was_revised": true, "final_justification": "The authors have provided detailed response, which resolved my concerns. Therefore, I maintain my positive rating.", "rebuttal": "Thank you for your constructive review and for recognizing the motivation, simplicity, and effectiveness of our approach. We address your concerns in detail below.\n\n---\n\n### **Q1) The gains for unseen classes are not well justified. Further explanation or analysis is needed to clarify where these improvements come from.**\n\nWe agree with the reviewer that further analysis is needed to justify the performance gains on unseen classes.\n\nTo investigate the source of these improvements, we analyzed the performance on the zero-shot dataset ADE20K by comparing the PQ, SQ, and RQ scores of the baseline and our proposed method. Note that this experiment was conducted under Scenario 1, where the incremental dataset is Cityscapes.\n\nAs shown in the table below, applying ConOVS to the baseline model (fc-clip) improves both PQ and RQ. However, the most significant gain is observed in SQ, which measures the quality of the predicted segmentation masks. This suggests that the improvements in unseen classes primarily stem from enhanced segmentation quality rather than better mask classification.\n\n|  | PQ | SQ | RQ |\n| :---- | :---- | :---- | :---- |\n| fc-clip | 23.5 | 61.7 | 28.3 |\n| \\+ ConOVS (Ours) | **25.9 (+2.4)** | **73.1 (+11.4)** | **31.2 (+2.9)** |\n\nWe will include this analysis in Section 6.1 (*Main Results*). Thank you for the feedback.\n\n---\n\n### **Q2) The choice to freeze the encoder and only fine-tune the decoder is not well justified. An ablation on this design choice is needed.**\n\nWe agree with the reviewer that, given recent works that fine-tune the CLIP encoder \\[1,2\\], a more detailed ablation study on our design choice to freeze the encoder and only fine-tune the decoder is necessary.\n\nTo this end, we experimented with three encoder fine-tuning strategies: 1\\) fine-tuning only the last block of the encoder,  2\\) fine-tuning only the LayerNorm modules, and  3\\) replacing all MLP layers with LoRA modules and fine-tuning them. Note that, unlike prior attention-based methods designed for ViT-style CLIP encoders \\[1,2\\], our strategies are tailored for ConvNeXt-based encoders used in fc-clip.\n\nAs shown in the table below, all three encoder fine-tuning strategies yield lower performance on the incremental dataset compared to our design choice of fine-tuning only the decoder. We attribute this to the architectural difference: while previous methods \\[1,2\\] generate segmentation masks directly from the encoder and benefit from encoder fine-tuning, fc-clip generates masks in the decoder's mask head. This suggests that decoder fine-tuning is more effective for improving segmentation performance in our setup.\n\n|  | coco (pre-training) | cityscapes (incremental) | ade20k (zero-shot) |\n| :---- | :---- | :---- | :---- |\n| fc-clip | 50.1 | 44.0 | 23.5 |\n| Last Block | 50.0 | 50.4 | 25.8 |\n| LayerNorm | 49.9 | 44.0 | 25.8 |\n| LoRA | **50.7** | 47.9 | 25.9 |\n| Only decoder | 50.4 | **64.4** | **26.0** |\n\nWe will include this analysis in Section 6.2 (*Ablation Study*).\n\n> \\[1\\] Cho, Seokju, et al. *Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. CVPR* 2024\\.  \n\\[2\\] Lee, Minhyeok, et al. *Effective SAM Combination for Open-Vocabulary Semantic Segmentation.* CVPR 2025\\.\n\n---\n\n### **Q3) Furthermore, could the proposed method be applicable to cost-based OVS methods that fine-tune the encoder, e.g. CAT-Seg?**\n\nIn general, our method is also applicable to cost-based OVS approaches. Designed to merge independently trained models, it remains compatible with techniques that fine-tune the CLIP encoder. While cost-based methods typically generate segmentation maps by post-processing encoder features, our method does not rely on such steps, making it directly applicable without modification.\n\nHowever, when applying our method to cost-based OVS models, the encoding process must be performed twice. This is because our method computes the interpolation factor based on the proximity of the input sample, which requires an initial forward pass through the encoder. After determining the interpolation factor, a second forward pass must be performed using the merged encoder to generate the final feature representation. As a result, an increase in inference time is expected.\n\nWe believe that extending our method to encoder fine-tuning frameworks, such as cost-based OVS models, is an interesting direction for future work. We will include this discussion in the newly added *Future Works* section following Section 7\\.\n\n---\n\n### **Q4) Given that cost-based methods already mitigate forgetting by fine-tuning CLIP encoders, it would be interesting to see whether ConOVS provides complementary benefits.**\n\nOur method is complementary to cost-based OVS models in terms of preventing catastrophic forgetting. When our method is applied to cost-based OVS techniques, a cost-based expert is trained for each domain. The merging of these experts is then guided by interpolation factors computed using the original CLIP encoder, which remains accessible at all times. As a result, we expect that combining our method with cost-based OVS models can further enhance their inherent ability to mitigate catastrophic forgetting.\n\nWe consider empirically verifying this possibility an interesting direction for future work, and we plan to include this discussion in the *Future Work* section of the final version.\n\n---\n\n### **Q5) Given that the current retraining baseline underperforms due to limited training, how much improvement can be expected if it is allowed to fully converge?**\n\nWe agree that evaluating the retraining baseline with longer training iterations offers valuable insights into the continual learning scenario for OVS. To conduct this analysis, we performed an experiment using Cityscapes as the incremental dataset.\n\nAs shown in the table below, retraining the model with a longer training schedule (100k iterations) leads to better performance than our method on the pre-training and incremental datasets. \n\n| method | training iterations | coco (pretraining) | cityscapes (incremental) | ade20k (zero-shot) |\n| :---- | :---- | :---- | :---- | :---- |\n| retraining | 10k | 50.7 | 61.9 | 25.2 |\n| retraining | 20k | 50.7 | 62.5 | 25.4 |\n| retraining | 40k | **51.0** | 63.3 | 25.3 |\n| retraining | 100k | **51.0** | **64.4** | 25.5 |\n| ConOVS (Ours) | 10k | 50.4 | 64.2 | **26.0** |\n\nHowever, despite consuming significantly more computational resources, the retraining method does not yield substantially better performance than ConOVS. One possible explanation for the limited performance of retraining is task interference \\[3,4\\], which might occur when training on multiple domains simultaneously. In contrast, ConOVS avoids this issue by training domain-specific experts independently and combining them adaptively at inference time. As a result, our method achieves comparable performance than retraining, while requiring significantly less computational cost.\n\nWe will include this analysis in the final version of the paper. We appreciate the helpful feedback.\n\n> \\[3\\] Yu, Tianhe, et al. *Gradient surgery for multi-task learning.* NeurIPS 2020\\.  \n\\[4\\] Chen, Zhao, et al. *Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.* ICML 2018\\."}, {"review_id": "EqsIDI6AAM", "reviewer": "Reviewer_aa5k", "summary": "This paper aims to solve the problem of continual learning of Open-Vocabulary Semantic Segmentation (OVSS), a scenario where new datasets are added without access to previously used training data. To address this, the paper trains a mixture of experts, with each expert specifically trained on a distinct dataset. It then balances the influence of these experts by calculating the proximity between the embeddings of the image being inferred and the embeddings representing each dataset. For example, if an image's embedding is closer to a particular dataset, a higher weight is assigned to the expert trained on that dataset. The paper's experiments used Cityscapes and ADE20K as incremental datasets.", "questions": "Overall, while I am not well-versed in the literature of Open-Vocabulary Semantic Segmentation (OVSS) with continual learning and have low confidence in my decision, I believe the writing of Section 5.2 should be significantly improved, and more data should be tested for continual learning to convince the reader. Regarding SemLA, considering that the paper was released in March 2025, a discussion of it would be sufficient; it's not a significant reason for rejection. Overall, I am leaning towards rejection due to the experimental setup and writing clarity. As I'm unfamiliar with the topic, I will carefully listen to other reviewers' opinions and make a final decision.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. Well-written introduction part. \n\n---\n**Major Weakness**\n1. **Conceptual similarity to Qorbani et al., \"Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation\" (CVPR '25).**\nAside from the difference in the specific task, the conceptual similarity between this paper and SemLA is very high and warrants extensive discussion in the related works section. Both approaches calculate proximity to each training dataset to determine weights between datasets (similar to ConOVS). With these weights, they merge LoRA weights (analogous to Mixture-of-Experts in ConOVS). \n\n\n2. **The writing in Section 5.2, \"Inference Phase,\" is puzzling.** It would be much easier to understand if the shapes of the vectors were notated. Additionally, it is almost impossible for me to understand what is happening in this stage upon a first read. The paper states, \"evaluate the likelihoods of these embeddings under the MVN distributions for all datasets, and collect them into the vectors,\" but the method for doing so is unclear. What are the shapes of $l_{img}$ and $l_{text}$? After several re-readings, I could infer that the shape for the vectors might be $\\mathbb{R}^n$, with each element representing the likelihood of the image and text being close to the $i$-th dataset. Why is softmax conducted here? Why is a max operation performed? While I can infer the authors' intention, it takes a significant amount of time to reach that conclusion. Furthermore, if the purpose of using softmax is to create a probability distribution, then why do the authors perform an element-wise maximum that violates the assumption that their sum is 1? Overall, the writing in this section should be significantly improved for easier comprehension by readers.\n\n3. **Regarding the experimental setup, the number of incremental datasets (n) used is too low to convincingly demonstrate the method's effectiveness.** For instance, in Table 1, the number of incremental datasets is only one, which makes it difficult to assess the method's generalizability to a larger n. In contrast, SemLA demonstrated effectiveness by training with 10 datasets and testing with 20 benchmarks. Given this, there seems to be ample opportunity to integrate more datasets into the evaluation.\n\n---\n**Minor Weakness**\n- In Line 127, it would be beneficial if the \"characteristics of OVS tasks\" could be elaborated upon.\n\n- For Figure 2(b), it would be helpful to include citations for competitors in the caption.", "was_revised": true, "final_justification": "The authors have provided solid evidence to demonstrate generalizability with more datasets. All of my concerns have been resolved.", "rebuttal": "Thank you for your detailed review and constructive feedback. We appreciate your comments on the writing, experimental setup, and relation to SemLA, and address each point below.  \n\n---\n\n### **Q1) This paper and SemLA share a core idea: estimating dataset proximity to compute weights for merging LoRA modules, similar to ConOVS’s MoE approach.**\n\nSemLA \\[1\\] was released on arXiv in March 2025 and presented at CVPR in June 2025\\. According to the NeurIPS Call for Papers, it is officially considered a contemporaneous work.\n\nNonetheless, we agree with the reviewer that a discussion on the conceptual connection between SemLA and our method is necessary. As the reviewer pointed out, there is a conceptual similarity between the two approaches—both compute the proximity of an input image to each training dataset and use it to determine dynamic merging weights during inference. However, our contributions differ from those of SemLA in several important aspects:\n\n- Our paper is the first to identify and address the limitations of existing open-vocabulary segmentation (OVS) methods in continual learning (CL) scenarios, where new datasets are continuously collected over time. Highlighting the issues of OVS under CL is an important contribution of our work. To the best of our knowledge, this setting has not been considered in SemLA, which focuses on single-stage inference without addressing continual dataset shifts.\n\n- Compared to SemLA, our method adopts different design choices in three key aspects: (1) it uses both image and text embeddings to assess domain relevance, whereas SemLA relies solely on image embeddings; (2) it estimates domain proximity using multivariate normal (MVN) distributions, while SemLA computes L2 distances to dataset-specific centroids; and (3) it fine-tunes the decoder for each dataset, in contrast to SemLA, which applies LoRA modules to the CLIP image encoder. A detailed explanation of the first point is provided in our response to Q2 below.\n\nIn addition, we implemented SemLA within our experimental setup for a direct comparison. Specifically, we modified our framework to follow SemLA’s key design choices: (1) computing proximity using only image embeddings, (2) estimating domain proximity by calculating the L2 distance between the input embedding and dataset-specific centroids, and (3) replacing decoder fine-tuning with LoRA fine-tuning on the CLIP image encoder.\n\nAs shown in the table below, our method (ConOVS) outperforms this SemLA variant on the incremental dataset. However, since multiple design differences exist between the two methods, it is difficult to isolate the exact cause of the performance gap. A deeper analysis of this result was challenging to conduct within the limited rebuttal period, but we consider it an important direction for future work. Nonetheless, the observed improvement suggests that our design choices might be more suitable for continual open-vocabulary segmentation.\n\n| method | coco (pretraining) | cityscapes (incremental) | ade20k (zero-shot) |\n| :---- | :---: | :---: | :---: |\n| fc-clip | 50.1 | 44.0 | 23.5 |\n| SemLA | **50.9** | 61.1 | **26.0** |\n| ConOVS (ours) | 50.4 | **64.4** | **26.0** |\n\nWe will add the discussion on the relation between SemLA and our method to Section 2 (*Related Works*), and the experimental results to Section 6 (*Experiments*). We appreciate the insightful feedback.\n\n> \\[1\\] Qorbani, Reza, et al. *Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation.* CVPR 2025\\.\n\n---\n\n### **Q2) Section 5.2 lacks clarity in vector shapes and operations like softmax and max; the rationale should be explained more clearly.**\n\nWe agree with the reviewer that the description in Section 5.2 was difficult to follow. In this response, we first clarify the rationale behind our use of softmax and element-wise maximum operations, and then provide details on the improvements we made to Section 5.2 for better readability.\n\nWe apply the softmax operation to the log-likelihood vector to normalize the proximity scores of each domain to the \\[0, 1\\] range. This decision is based on a prior study \\[2\\], which found that when the interpolation factor exceeds 1, merging performance degrades. To normalize the log-likelihood values obtained from the MVN distribution, we considered three strategies, including min-max normalization, sigmoid, and softmax. As shown in the table below, softmax achieved the best performance, which led us to adopt it.\n\n| Operation | coco (pretraining) | cityscapes (incremental) | ade20k (zero-shot) | Average |\n| :---- | :---- | :---- | :---- | :---- |\n| minmax | 50.3 | 50.5 | **26.1** | 42.3 |\n| sigmoid | 50.3 | 50.3 | 26.0 | 42.2 |\n| softmax | **50.4** | **64.4** | 26.0 | **46.9** |\n\nWe use the element-wise maximum operation to combine information from both the image and text modalities. Initially, we considered whether to rely on only the image domain, only the text domain, or both. For combining both modalities, we evaluated three options: average, multiplication, and element-wise maximum. As shown in the table below, the results show that combining both modalities performs better than using a single modality, and among the combination methods, element-wise maximum achieved the best performance. Based on these findings, we selected element-wise maximum as our fusion strategy.\n\n|  | Operation | coco (pretraining) | cityscapes (incremental) | ade20k (zero-shot) | Average |\n| :---- | :---- | :---- | :---- | :---- | :---- |\n| image only | \\- | 51.5 | 43.4 | 25.8 | 40.2 |\n| text only | \\- | **51.9** | 60.7 | 25.9 | 46.2 |\n| image and text | average | 50.3 | 64.2 | 25.9 | 46.8 |\n| image and text | multiplication | 50.1 | 63.8 | 25.8 | 46.6 |\n| image and text | maximum | 50.4 | **64.4** | **26.0** | **46.9** |\n\nWe acknowledge that the use of element-wise maximum does not guarantee that the resulting proximity values sum to one. However, this aligns with prior practice in task vector-based merging methods \\[2,3\\], where interpolation factors are not required to sum to one. In our experiments, we observed no adverse effects resulting from this design choice.\n\nTo improve the clarity of Section 5.2, we not only added the above explanations regarding our design choices but also provided more concrete details, such as the dimensions of the vectors. For example, we recognized that the definitions of the likelihood vectors $l\\_{\\\\text{text}}$ and $l\\_{\\\\text{img}}$ were ambiguous. We revised the text to clarify that these vectors represent the proximity values between the input sample and each dataset, and we explicitly stated their dimensions to improve readability. \n\n> \\[2\\] Ilharco, Gabriel, et al. *Editing models with task arithmetic.* ICLR 2023\\.  \n\\[3\\] Yadav, Prateek, et al. *Ties-merging: Resolving interference when merging models.* NeurIPS 2023\\.\n\n---\n\n### **Q3) The number of incremental datasets is too small to convincingly demonstrate generalizability. More datasets should be included for a convincing evaluation.**\n\nWe agree that evaluating our method with a larger number of incremental datasets is important for demonstrating generalizability. To address this, we designed a new scenario in which the model is sequentially trained on five datasets: COCO, Cityscapes, ADE20K, BDD100K, and Mapillary Vistas. While we aimed to include more datasets, our work focuses on panoptic segmentation, for which publicly available datasets are considerably more limited than those available for semantic segmentation as used in SemLA. \n\nThe results of this extended experiment are shown in the table below. As can be seen, our method outperforms the baseline, fine-tuning, and retraining approaches. These results indicate that our method remains effective as the number of incremental datasets increases, demonstrating its scalability under complex domain shifts.\n\n|  | pre-training | incremental |  |  |  | zero-shot |  |  |  |  |  | Average |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | COCO | Cityscapes | A-150 | BDD100K | Mapillary | LVIS | PC-59 | PC-459 | PAS-20 | PAS-21 | A-847 |  |\n| fc-clip | 50.1 | 44.0 | 23.5 | 19.0 | 26.0 | 20.5 | 53.0 | 16.9 | 93.1 | 80.2 | 13.8 | 40.0 |\n| fine-tuning | 31.7 | 55.5 | 28.7 | 25.8 | 36.3 | 17.7 | 49.8 | 16.3 | 90.0 | 73.9 | 14.0 | 40.0 |\n| retraining | 49.0 | 62.3 | 36.3 | 28.2 | 34.5 | 21.7 | **53.6** | **17.4** | **94.0** | **80.7** | 15.6 | 44.8 |\n| ConOVS (Ours) | **50.1** | **62.9** | **38.5** | **29.1** | **35.0** | **21.8** | **53.6** | 17.3 | 93.2 | 80.4 | **16.0** | **45.3** |\n\nWe will include this experiment and its discussion in Section 6.1 (*Main Results*). Thank you for the valuable feedback.\n\n---\n\n### **Q4) In Line 127, it would be beneficial if the \"characteristics of OVS tasks\" could be elaborated upon.**\n\nWhat we intended by \"characteristics of OVS tasks\" is the fundamental difference between open-vocabulary segmentation (OVS) and closed-set segmentation. Specifically, OVS tasks involve a potentially infinite label space, whereas closed-set segmentation assumes a fixed, finite label set. Taking this into account, we will revise the sentence as follows:\n\n> *We believe this arises because existing CL methods assume a closed-set segmentation with a finite label space, whereas OVS involves a potentially infinite label space, which these methods do not account for.*\n\n---\n\n### **Q5) For Figure 2(b), it would be helpful to include citations for competitors in the caption.**\n\nWe agree with the reviewer and will update the caption of Figure 2(b) to include citations for the continual learning methods used. The revised caption is as follows:\n\n> *Figure 2: … (b) Comparison of the performance of OneFormer \\[12\\], the baseline (fc-clip \\[44\\]), retraining, fine-tuning, three existing continual learning methods \\[14,15,18\\], and ConOVS on the pre-training and incremental datasets.*"}]

Author rebuttal:

Meta-review: This work presents a continual learning framework for open-vocabulary segmentation, where data is presented in a sequential manner. Reviewers found that the proposed strategy is novel and well-motivated, which is empirically supported by extensive experiments across multiple datasets and models. Furthermore, the impact of the different components was properly evaluated on further ablation studies. Despite these strengths, reviewers also pointed to several weaknesses (e.g., several results or choices not properly justified, lack of detailed computational overhead analysis, as well as further clarification), which were successfully addressed by the authors during the rebuttal/discussion process. Considering all these points I recommend the acceptance of this work.
Decision: Accept (poster)