Title: OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open-Vocabulary

Segmentation

Continual Learning
Abstract: Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the problem of continual learning of Open-Vocabulary Semantic Segmentation (OVSS), a scenario where new datasets are added without access to previously used training data. To address this, the paper trains a mixture of experts, with each expert specifically trained on a distinct dataset. It then balances the influence of these experts by calculating the proximity between the embeddings of the image being inferred and the embeddings representing each dataset. For example, if an image's embedding is closer to a particular dataset, a higher weight is assigned to the expert trained on that dataset. The paper's experiments used Cityscapes and ADE20K as incremental datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. Well-written introduction part. \n\n---\n**Major Weakness**\n1. **Conceptual similarity to Qorbani et al., \"Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation\" (CVPR '25).**\nAside from the difference in the specific task, the conceptual similarity between this paper and SemLA is very high and warrants extensive discussion in the related works section. Both approaches calculate proximity to each training dataset to determine weights between datasets (similar to ConOVS). With these weights, they merge LoRA weights (analogous to Mixture-of-Experts in ConOVS). \n\n\n2. **The writing in Section 5.2, \"Inference Phase,\" is puzzling.** It would be much easier to understand if the shapes of the vectors were notated. Additionally, it is almost impossible for me to understand what is happening in this stage upon a first read. The paper states, \"evaluate the likelihoods of these embeddings under the MVN distributions for all datasets, and collect them into the vectors,\" but the method for doing so is unclear. What are the shapes of $l_{img}$ and $l_{text}$? After several re-readings, I could infer that the shape for the vectors might be $\\mathbb{R}^n$, with each element representing the likelihood of the image and text being close to the $i$-th dataset. Why is softmax conducted here? Why is a max operation performed? While I can infer the authors' intention, it takes a significant amount of time to reach that conclusion. Furthermore, if the purpose of using softmax is to create a probability distribution, then why do the authors perform an element-wise maximum that violates the assumption that their sum is 1? Overall, the writing in this section should be significantly improved for easier comprehension by readers.\n\n3. **Regarding the experimental setup, the number of incremental datasets (n) used is too low to convincingly demonstrate the method's effectiveness.** For instance, in Table 1, the number of incremental datasets is only one, which makes it difficult to assess the method's generalizability to a larger n. In contrast, SemLA demonstrated effectiveness by training with 10 datasets and testing with 20 benchmarks. Given this, there seems to be ample opportunity to integrate more datasets into the evaluation.\n\n---\n**Minor Weakness**\n- In Line 127, it would be beneficial if the \"characteristics of OVS tasks\" could be elaborated upon.\n\n- For Figure 2(b), it would be helpful to include citations for competitors in the caption.\n\nQUESTIONS:\nOverall, while I am not well-versed in the literature of Open-Vocabulary Semantic Segmentation (OVSS) with continual learning and have low confidence in my decision, I believe the writing of Section 5.2 should be significantly improved, and more data should be tested for continual learning to convince the reader. Regarding SemLA, considering that the paper was released in March 2025, a discussion of it would be sufficient; it's not a significant reason for rejection. Overall, I am leaning towards rejection due to the experimental setup and writing clarity. As I'm unfamiliar with the topic, I will carefully listen to other reviewers' opinions and make a final decision.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ConOVS, a MoE-based framework for continual learning in the field of open-vocabulary segmentation. The paper first suggests baseline methods, being fine-tuning and retraining, to incorporate additional training data, which suffer from catastrophic forgetting and declines in performance for its original pre-trained dataset. In this regard, the authors propose ConOVS, which dynamically mixes the weight for the segmentation decoder during inference based on the encoder representation statistics. The authors demonstrate clear advantages compared to baselines, while also providing various experiments settings and ablations to verify the design choice.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**[S1]** The paper presents a well-founded motivation for incremental learning on the open-vocabulary setup, and constructs solid baselines for comparison, with the proposed method showing solid gains over the baseline. \n\n**[S2]** The proposed solution is simple, yet sensible for solving the task. Furthermore, while it may be trivial for a MoE-based approach to prevent catastrophic forgetting, it is impressive to observe gains in unseen classes, where none of the \"experts\" should excel at.\n\n**[S3]** The paper is generally well-written with helpful figures and to understand the framework, as well as providing solid experiments to validate the choices for estimating the interpolation factor. \n\n**Weaknesses**\n\n**[W1]** Some of the results, especially regarding the gains for unseen classes, are not well justified. \n- While it is impressive to see the model improve on unseen classes, it is also counterintuitive to see the combined model to improve in such classes as none of the base models would be effective in identifying unseen classes. While the authors claim that the proposed method is generalizable, it is not explained where the gains are actually coming from.\n\n**[W2]** Some of the design choices are not well justified.\n- The method decides to freeze the encoder while only fine-tuning the decoder, without much justification or ablations. This can further relate to examining the applicability to streamline of works that focuses in fine-tuning the encoder of CLIP[1,2,3], which tend to show much better performance to its frozen counterparts, e.g. fc-clip.\n\n[1] Cho S, Shin H, Hong S, Arnab A, Seo PH, Kim S. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 4113-4123).\n\n[2] Xie B, Cao J, Xie J, Khan FS, Pang Y. Sed: A simple encoder-decoder for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2024 (pp. 3426-3436).\n\n[3] Lee M, Cho S, Lee J, Yang S, Choi H, Kim IJ, Lee S. Effective SAM Combination for Open-Vocabulary Semantic Segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 26081-26090).\n\nQUESTIONS:\n**[Q1]** Could the authors provide additional explanation or analysis on how the unseen classes are improving? \n- To the reviewers perspective, one possible explanation might be that the fine-tuned models on the incremental datasets might have improved in identifying segmentation masks, as it would be hard to expect the mask classification to have improved. One possible approach to examine this could be inspecting the RQ and SQ for the panoptic datasets, and observe which is improving or not.\n\n**[Q2]** Could the authors ablate the choice for only fine-tuning the decoder? Furthermore, could the proposed method be applicable to cost-based OVS methods that fine-tune the encoder[1,2,3], e.g. CAT-Seg[1]? \n- The fine-tuning of the encoders for cost-based methods[1,2,3] would fundamentally differ from X-Decoder as these models fine-tune the encoder solely on the segmentation dataset, whereas X-Decoder maintain the image-caption objective during its training.\n- Considering that these cost-based methods are able to avoid catastrophic forgetting while directly fine-tuning the encoders of CLIP, it would be interesting to observe how the effects of ConOVS to be complementary in terms of preventing forgetting issues.\n\n**[Q2]** Given that the retrained baseline is expected to underperform due to not having sufficient training iterations, how much could retraining improve when it has fully converged?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continuous learning method, ConOVS, to address the challenges of Open Vocabulary Segmentation (OVS) models in sequential data collection scenarios. ConOVS based on the MoE framework, achieves dynamic adaptation through the following mechanisms:\n- Training phase: Fine-tune the decoder for each new dataset, generate independent experts, and calculate the multivariate normal distribution (MVN) of the image/text embeddings.\n- Inference stage: Generate interpolation factors based on the similarity between the input samples and the distribution of each dataset (through MVN probability estimation), and dynamically combine the weights of the expert decoder.\nThe effectiveness of ConOVS was verified in three scenarios (COCO pre-training + 20K incremental data from Cityscapes/ADE) and multiple zero-shot datasets (such as LVIS, BDD100K). The results show that it outperforms the baseline method on pre-trained, incremental and zero-shot datasets, and alleviates the problem of catastrophic forgetting.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- Experiment:\nThe segmentation performance was evaluated on over 10 datasets, covering multiple scenarios (single/multiple incremental datasets) and models (fc-clip, X-Decoder). The performance shows the enhance in performance.\n\n- In-depth analysis:\nThe ablation experiment verified the key designs (such as the MVN distribution being superior to K-means/KDE, and the image + text modal combination being superior to the single modal).\n\nWeakness：\n- Although ConOVS claims to reduce retraining costs, the paper lacks a demonstration on the training cost comparison betwing retraining / SFT and continuous learning. Maybe detailed training time / resource / computational cost analysis is needed.\n\n- The experimental setup relies primarily on only two or three incremental datasets (Cityscapes and ADE20K). The lack of evaluation under longer learning sequences (e.g., involving 5 to 10 incremental datasets) leaves open questions regarding the scalability and robustness of the method in handling high diversity and complex domain shifts over time.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a continual learning framework for open-vocabulary segmentation (OVS), addressing the practical scenario where data arrives incrementally over time and the model needs to adapt to new datasets. The authors introduce a method that trains a separate mask decoder for each incoming dataset and combines these decoders in a Mixture-of-Experts (MoE) manner during inference. Compared to full model retraining, the proposed approach is more cost-effective. It also mitigates catastrophic forgetting more effectively than fine-tuning, and improves performance on both incremental and zero-shot test datasets—surpassing existing continual learning methods on OVS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- While other OVS methods focus on training a segmentation model once and applying it everywhere, this paper addresses the scenario where data arrives incrementally over time and adapts the OVS model using the new incremental data. This paradigm has been shown to be effective for both incremental and zero-shot dataset performance. The observation aligns with prior work such as [1], where incorporating additional data during training also led to performance improvements in OVS.\n\n- The integration of continual learning with a Mixture-of-Experts (MoE) strategy for OVS is particularly novel and well-motivated. A standard OVS model typically consists of three components: a text encoder, an image encoder, and a mask decoder. This work proposes learning a separate mask decoder for each incoming dataset and combining their weights during inference. This design is shown to be highly effective, as demonstrated in Tables 1, 2, and 3.\n\n- The paper is clearly written and well-organized, making it easy to follow the motivation, methodology, and results.\n\n[1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\n\nWeaknesses:\n- The framework seems generalizable to many tasks, and the paper focuses only on OVS. However, there is no specific design for segmentation. A closely related line of work is Open-Vocabulary Detection (OVD). Could this approach also be applied\nto OVD?\n\n- As shown in Table 1, the continual learning model outperforms the retrained model. Why does continual learning yield better performance than retraining from scratch?\n\n- The paper assumes that each incremental dataset belongs to a different domain, which may not hold true in all scenarios. What happens if we do not know the domain of each incoming dataset? For example, what if the incoming data at a given time step is a mixture of multiple domains (e.g., ADE20K and Cityscapes)? How robust is the method in such situations?\n\n- If an incremental dataset is small or weakly labeled, how does that affect model performance? Will the proposed approach still perform well in such low-resource settings?\n\nQUESTIONS:\nThese questions come from comments on weaknesses\n\n- Could this approach also be applied to OVD?\n\n- Why does continual learning yield better performance than retraining from scratch as shown in Table 1?\n\n- The paper assumes that each incremental dataset belongs to a different domain, which may not hold true in all scenarios. What happens if we do not know the domain of each incoming dataset? For example, what if the incoming data at a given time step is a mixture of multiple domains (e.g., ADE20K and Cityscapes)? How robust is the method in such situations?\n\n- If an incremental dataset is small or weakly labeled, how does that affect model performance? Will the proposed approach still perform well in such low-resource settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}