Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work extends a MaskFormer model to build LMSeg model, that learns to segment categories across multiple datasets using unified training without resorting to manual taxonomy. LMSeg is built on the fact that the text embeddings for semantically similar categories, as retrieved from CLIP text encoder, projects close to each other on euclidean space. Hence by learning to align the segment embeddings to the text embeddings, it is possible to learn to pool similar categories from multiple datasets without manual taxonomy mapping. Further, in order to dynamically guide the segment embeddings towards dataset-specific taxonomy, LMSeg proposes to use category-guided text-query cross-attention within the transformer decoder. This together with the dataset-aware augmentation strategy can train SoA multi-dataset model that outbeats single-dataset trained models (with similar total budget).
strength_and_weaknesses: Strengths:
1. Multi-dataset training for segmentation using a unified taxonomy is the first step towards open-set model. LMSeg automates this training without any -- additional parameters in the form of dataset-specific head, manual mapping and drop in per-dataset mIoU. Similar to other open-set vision model, LMSeg leverages language embeddings to build a unified model.
2. Moreover, the proposed LMSeg is simple, elegant and easy to implement straight out of MaskFormer model. 

Weaknesses:
1. The model is well engineered. However, I feel there is limited novelty. Multiple recent work has resorted to text-query alignment to build an open-set model for different problem settings (object detection, referring expression). Motivated by this development, LMSeg modifies and trains Maskformer like framework with similar losses. To me, the only novel contribution seems category-guided decoder design. 
2. Secondly, Table 2 & 6 suggests that the impact of LMSeg, CGD & DAA framework on semantic segmentation setting is very minimal. Hence the novel components only contributes to improvements in panoptic segmentation. Any thoughts on what causes this discrepancy ? 

How about evaluating LMSeg in an open-world setting ? Or atleast, cross-category evaluation (i.e., train model by leaving out test categories) or datasets relevant to training data (such as ADE-Full) ? These results should strengthen the empirical evaluation. As for now the multi-dataset evaluation has very limited use case.  
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. With given experimental settings it should be simple to reproduce the listed evaluation. 
summary_of_the_review: LMSeg work makes a good read with clean experimental evaluation with few insights drawn during ablation. However, due to limited technical novelty I would place current version at Borderline acceptance. Adding extra evaluation and drawing better insights, would clearly add more value.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve two problems of multi-dataset segmentation: (1), inconsistent taxonomy. (2), inflexible of one-hot taxonomy. They map the taxonomy into embedding space with pre-trained pre-trained text encoder. Then they proposed a category-guided decoding module is designed to dynamically guide predictions to each dataset’s taxonomy.
Extensive experiments demonstrate that the proposed method achieves significant improvements on four semantic and three panoptic segmentation datasets

strength_and_weaknesses: Overall, this paper is good. 

Strength:


1, The paper is well written and easy to follow. 

2, The idea of using unified text embedding to enhance the object queries is interesting and novel for DETR-like framework for multiple datasets segmentation tasks. Category-guided decoding (CGD) is simple yet effective. 

3, The extensive results show the effectiveness of proposed framework. 


Weakness:

1, The idea of using text embedding for multi-dataset segmentation is not novel. The idea is proposed in the work “The devil is in the labels: Semantic segmentation from”(half year ago on arxiv. ) This paper adds learnable embedding (From COOP-IJCV-2022) and extra attention module. The author should not claim this is very huge contribution in introduction part. 

2, Missing Experiment results M-Seg dataset. It would be better for benchmarking results on this dataset. 

3, Missing ablation studies on CGD design. 

4, Missing parameter and GFlops analysis on proposed CGD. 

clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: Overall, this paper is interesting to me. However, using text embedding for multi-dataset segmentation has been explored before.  Reminded by other reviewer opinions (Reviewer DHCs), I find the real technical novelty is very limited (Category-guided decoding (CGD) and Dataset-aware augmentation (DAA). I lower down my ratings.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents LMSeg, which aims to train a model for image segmentation on multi-datasets. The authors claim that they resolve two major challenges in this paper:
- 1 the domain gap or inconsistency between semantic segmentation and panoptic segmentation.
- 2 the separated label space.

Experiments show that the proposed method can achieve better results than the previous baseline.
strength_and_weaknesses: ### Strength

1. The paper is well written and easy to follow.

2. I recognize the significance of the problem targeted in this paper. It should be feasible to identify instances in images of datasets labelled for semantic segmentation.

### Weaknesses

**1. The novelty is limited.** The main claim of the authors is that they propose *text-query alignment to address the issue of taxonomy inconsistency*. However, introducing language embeddings to build a unified label space for image segmentation has already been proven to be effective [1, 2, 3]. The Category-guided decoding (CGD) here is just another variant of LSeg [1] that exploits the text-pixel affinity to bridge the domain gap across different label spaces. Dataset-aware augmentation (DAA) here is a non-learnable strategy for data augmentation, which seems to be better mentioned in the section of implementation details instead of methodology.

**2. The improvement is not strong enough.** The numbers shown in Table 2 are not impressive from my point of view. For example, the gain for the average mIOU across four datasets can only be shrunk to a negligible 0.2% for 640K. I focus more on the results trained with a longer schedule since MaskFormer is hard to optimize and originally it needs 300 epochs to train on COCO panoptic. The results look to be prettier when trained under a short schedule, however, this might be due to a better initialization as the encoder of the proposed method is first pre-trained by CLIP (as discussed in Table 4).

### Reference

[1] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. ICLR 2022

[2] Xu, Mengde, et al. "A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model." arXiv preprint arXiv:2112.14757 (2021).

[3] Rao, Yongming, et al. "Denseclip: Language-guided dense prediction with context-aware prompting." CVPR 2022.
clarity,_quality,_novelty_and_reproducibility: The illustration of this paper is clear and easy to understand. However, the originality of this paper is limited and the improvements are also not impressive to me. Although the authors attach their configs in the supplementary material, they do not promise to release the code in the paper so the reproducibility of this work cannot be guaranteed. 
summary_of_the_review: My concerns primarily lie in the novelty and efficacy of the proposed method. As discussed above, neither the story nor the performance of this paper impress me, I am thereby inclined to reject this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a language-guided framework for multi-dataset training. To solve the consistency of categories in each dataset, the paper encodes the categories in text embedding without manual reconciliation. Also, the paper adopts a dataset-aware augmentation strategy that assigns each dataset a specific image augmentation pipeline, which can suit the properties of images from different datasets.
The extensive experiments demonstrate that the proposed method achieves significant improvements on four semantic and three panoptic segmentation datasets.
strength_and_weaknesses: Pros:

1, The paper is well-written and easy to understand.
2, The proposed method is simple but efficient.
3, The experiments are sufficient.

Cons:

1, In the table2 of semantic segmentation, the proposed method seems to improve marginally if has a long training time. However, the table3 of panoptic segmentation improves largely.  Could the author have more analysis? is it related to the improvement of instance segmentation?

2, I note that the proposed method is benefited from both CLIP image encoder and text encoder. Thus I wonder if the main reason for the improvement of the proposed method is the usage of the CLIP model, especially the CLIP Image encoder.



clarity,_quality,_novelty_and_reproducibility: The paper is well written. And I think it could be reproduced easily.
summary_of_the_review: The proposed method gives new insight, using a text encoder to solve the inconsistency of the label system in the different datasets, in multi-dataset segmentation training. This is an interesting direction. In my view, the more experiments analysis should be added.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers are nominally split, so the average rating is borderline. But, reading the reviews, it seems clear that the main shared concern is the novelty. More specifically, as the authors acknowledge (both in the paper and in the discussion) the basic idea of using language embeddings for segmentation targets is of course not new; but there is also a clearly novel component here, namely the CGD (which allows multi-dataset application). So the question is how significant is this novelty. The reviewers are somewhat lukewarm about it, but acknowldedge it as a novel contribution. Furthermore, reviewer DHCs, who gave the lowest score, with limited novelty cited as the chief concertn, does agree after the discussion that CGD is novel.

So the final judgment seems to come down to the significance of the results (how "impressive" they are). There is a detailed discussion of these, with the authors explaining the fairness of comparison and the relevance of improvements observed with short training time. I agree that these results are meaningful. The main contribution of this paper is clearly not a new "SOTA" number but a new technical idea for dealing with multiple datasets.

I have read the paper and the reviews,rebuttals, and responses. I believe this submission is clearly above bar; as the field moves from fixed/arbitrary datasets to dealing with open-world data via data streams with images (and other data) from mixed origins, it will become increasingly important to explore tools like the system proposed here. 


justification_for_why_not_higher_score: The novelty is indeed limited, and the application is a bit niche at the moment. 
justification_for_why_not_lower_score: There is enough novelty/interesting findings here to justify acceptance.
Decision: Accept: poster