Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a transformer based approach for zero shot semantic segmentation. It makes the use of different loss functions like cross entropy loss for seen classes, regression loss between language and visual features to account for unseen classes, a pixel text score map to reduce the seen bias problem and a semantic consistency loss to transfer the relationship of word2vec features to the semantic prototypes of the embedding space. State of the art results are shown on Pascal VOC and Context datasets.
strength_and_weaknesses: The method obtains reasonable improvements over good baselines and state of the art results. 
The use of pixel text score map is interesting. 


No ablation experiments for Lsc and Lr, which are described as sub-sections of the technical contribution in the approach. 
The paper is not written well and has several errors.
clarity,_quality,_novelty_and_reproducibility: 


Introduction: 
WSSS are often based on easily obtaining annotations, such as scribbles => sentence seems incorrect

We argue that a shared shortcoming of previous ZS3 models falls in the reduced receptive field of CNNs and less uses attention mechanisms for extracting the global relations of visual features conditioned with language semantic information. => 'less uses' does not seem right

Current networkBaek et al. (2021) adopt traditional => space between network and Baek. Sentence also is not correct

Although generative methods achieve impressive performance in zero-shot semantic segmentation tasks.The methods are limited by a multi-stage training strategy, Remove although or replace '.' with ','

The lsc is proposed by Baek et al. (2021), which define the relation between prototypes as follows: => language does not seem correct

The overall loss is finally formulated as
L = Lce + Lr + λ1Lsc + λ2Laux, what is Laux in equation 2, I only see Lps in equation 2


summary_of_the_review: The paper has many errors and has a few missing ablation experiments. The method does not seem very novel as it mixes a combination of loss functions (mostly known) with transformers to improve results marginally over baselines. Overall the contributions are borderline and near the acceptance threshold if some concerns are alleviated. 

Updated rating after reading other reviews.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a discriminative zero-shot semantic segmentation approach that uses transformer image and language encoders to exploit the global relations of visual features and to reduce seen class bias. In this approach, the image encoder is trained with calculating pixel-text score maps which use dense language-guided semantic prototypes. The authors argue that CNN-based ZS3 models sometimes fail to extract language-guided activation fields because they work in reduced receptive fields and use fewer attention mechanisms to extract global relations. Experimental results on benchmark Pascal VOC and Pascal Context datasets show that this approach (i.e. SwinZS3) obtains state-of-the-art results.
strength_and_weaknesses: Strengths:
- The proposed transformer framework appears to be promising. The obtained limited experimental results (please see weaknesses) also support this situation.
- Nice to see the impact of auxiliary segmentation loss on SwinZS3 and DeepLab methods.
Weaknesses:
- The text flow feels irregular. For example, CLIP-based meyhods are mentioned regardless of the context at the end of the Introduction section. Some figures are not referred from the text (eg, Fig.1 and Fig.4). Grammatical problems also disrupts the flow. In addition, the text contains too many typos; e.g. "fro", "networkBaek", "generatining", etc.
- The authors follow the experimental settings provided by ZS3Net, and use 4 different splits for comparison with previous methods. However, there are 5 different splits according to the ZS3Net method and they share the results for 5 different splits in their paper. Hence, unseen-10 split results seem to be missing for both Pascal VOC and Pascal Context datasets.
- When the results shared in Table 2 are examined, it is observed that the CSRL method generally obtains better results for the seen classes. However, there is no clear experimental result to support the central claim that the seen/unseen bias is alleviated via minimizing the euclidean distance and using the pixel-text score maps.

*ZS3Net: Bucher, M., Vu, T. H., Cord, M., & Pérez, P. (2019). Zero-shot semantic segmentation. Advances in Neural Information Processing Systems
clarity,_quality,_novelty_and_reproducibility: 
The work has limited novelty: there are already uses of transformer models for semantic segmentation problems, the architecture change itself is not very interesting, and the other contributions are mainly adaptations of existing ZSL/semantic segmentation formulations. Due to the reasons mentioned above, the study includes various deficiencies in reproducibility.

Importantly: it is not clear *how* the hyper-parameters were chosen. While several hyper-parameters are shared, it is not clear how they are tuned and whether unseen class samples (test examples) have been used in the process.
summary_of_the_review: The arguments put forward by the authors are worth examining. (Limited) experiments also show that the method obtains successful results. However, the main and additional experiments are incomplete from my point of view. In addition, the study also needs serious improvements in writing, the overall manuscript quality does not meet the ICLR standards.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The article proposed a new network SwinZS3 that applies swin transformer to the zero-shot semantic segmentation domain. Experimental results on PASCAL VOC and PASCAL Context show that the method proposed in this paper outperforms existing methods.
strength_and_weaknesses: Strength

      -The proposed method achieved relatively good results on benchmarks.


Weakness


      -The presentation of some parts of the article is unclear. For example I'm confused about the structure of Language Encoder, but the article doesn't explain what it is throughout.
      -There are many problems with the layout and writing of the article. For example, there is incorrect capitalization in the first line of section 3.1 and missing spaces in line 4. There is confusion about the case of symbols in section 3.4.
     -Some recent related papers are not cited. Such as [1, 2, 3]
    [1] https://arxiv.org/abs/2112.14757
    [2] https://openaccess.thecvf.com/content/CVPR2022/papers/Ding_Decoupling_Zero-Shot_Semantic_Segmentation_CVPR_2022_paper.pdf
    [3] https://arxiv.org/abs/2202.11094
clarity,_quality,_novelty_and_reproducibility: The overall quality of the article is acceptable. The essay approach is slightly lacking in novelty. There are problems with the presentation and writing of the article. The reproducibility is okay if the code is released.
summary_of_the_review: The method proposed in the article is not novel enough. In addition, there are major problems with the writing and layout of the article. I deem that it's not a solid work. So I tend to give weak reject.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes to tackle the problem of zero-shot semantic segmentation (ZS3). To capture the global feature relations and semantic information, SWIN transformer is adopted as the backbone. To improve the decision boundary, pixel-text auxiliary segmentation loss is used. Good benchmark results are obtained in Pascal VOC and Context datasets. 
strength_and_weaknesses: Pros:
1. The topic of zero-shot semantic segmentation is interesting and important. 
2. It achieves good benchmark results on Pascal VOC and Context datasets. (however the comparisons might be unfair, which will be described in Cons.)


Cons:
1. The novelty of this paper is limited. It seems that the paper claims its novelty as using Swin Transformer, however, Swin Transformer has been widely used in CV community and simply adopts it as the backbone is not novel. In addition, the paper describes the cross-entropy loss, regression loss, semantic consistency loss in details. However, these techniques are proposed by JoEm (Baek et al. ICCV'2021). This paper simply uses them without any modifications. For L_aux, many papers have used similar losses. for example, Dong, Xiaoyi, et al. "MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining." arXiv preprint arXiv:2208.12262 (2022)..
 
2. The comparisons of the experiments (Table. 2) might be unfair. It uses stronger backbone (Swin Transformer) network compared to other methods. Please add experiments of DeepLabV3+ for all K in Table. 2. 

3. Please also discuss the relationship with CLIP-based zero-shot segmentation methods in the related work section. 

4. The paper writing needs significant improvement and careful revision. (1) The use of symbols is inconsistent. Does s in Eq.4 and Eq.6 the same? Some times N is used for the number of seen categories, sometimes K is used. (2) The introduction of "regression loss" is mostly unclear. where does semantic feature maps (s) come from? (3) The use of citation is wrong throughout the whole paper. (4) There are many typos. for example, extra "()" in Table. 2; "deeplabv3+" and "Deeplabv3+" in Table. 1. 

Questions:
The paper writing is not clear enough. 
1. In Table.1, Does the 1st row (deeplabv3+) mean JoEm? 

clarity,_quality,_novelty_and_reproducibility: The clarity, quality and originality of the paper is relatively poor. 
For detailed explanation, please see Cons. above. 
summary_of_the_review: Due to lack of novelty, unfair experimental comparisons, and poor paper writing, the reviewer suggests rejecting this paper in its current form. The paper needs major revision before resubmitting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a transformer-based approach for zero-shot semantic segmentation. To improve the decision boundary, pixel-text auxiliary segmentation loss is used and was shown to improve the performance. 

Strengths
-------------


- Reviewer sxBw, xJ8c,S84PJ,r nqEw: The method obtains reasonable improvements over good baselines and state-of-the-art results. Reviewer xJ8c: Nice to see the impact of auxiliary segmentation loss on SwinZS3 and DeepLab methods. 
- Reviewer xJ8c: The proposed transformer framework appears to be promising. 
- Reviewer nqEw: The topic of zero-shot semantic segmentation is interesting and important.

Weaknesses
-------------

- Reviewer sxBw: No ablation experiments for Lsc and Lr, which are described as sub-sections of the technical contribution in the approach. 
- Reviewer sxBw, xJ8c, S84P, nqEw : The paper is not written well and has several grammatical errors. Detailed comments by reviewers clarify the details. 
- Reviewer xJ8c: there is no clear experimental result to support the central claim that the seen/unseen bias is alleviated via minimizing the euclidean distance and using the pixel-text score maps.
- Reviewer xJ8c:  missing experiments for both Pascal VOC and Pascal Context datasets.
-Reviewer nqEw : The novelty of this paper is limited. It seems that the paper claims its novelty as using Swin Transformer and compared to MaskCLIP paper (Dong, Xiaoyi, et al. "). 
- The comparisons of the experiments (Table. 2) might be unfair. It uses a stronger backbone (Swin Transformer) network compared to other methods. Please add experiments of DeepLabV3+ for all K in Table. 2.
- Missing discussion and experimental depth. 


In conclusion, all reviewers are leaning towards not accepting the paper with its current version. 


justification_for_why_not_higher_score: Four knowledgeable reviewers recommend rejection. There is a rebuttal, and during the discussion, the reviewers reached a consensus that the paper has merits but is not ready for publication as there are several concerns in the paper's writings,  claims that lack sufficient support and design choices that lack sufficient justification, and missing experiments. The paper does not give a clear and strong intuition why the method work. No basis for overturning the reviews. 
justification_for_why_not_lower_score: N/A.
Decision: Reject