Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on text-supervised semantic segmentation by studying a multi-view consistency learning framework. Specifically, a text-to-views consistency and a cross-view segmentation consistency training strategy are proposed and added upon existing work Group VIT. The proposed method tries to construct more training constrains based on correspondence among multiple augmented views to obtain more improvements. Experiments are conducted on multiple datasets for zero-shot segmentation, e.g. PASCAL and COCO., showing improvements upon some existing baselines. 

strength_and_weaknesses: 
Strength:

+ The motivation of introducing multi-view consistency constrain to text-supervised semantic segmentation is reasonable.

+ Experimental reuslts on multiple datasets show some benifits compared with existing methods. 


Concerns: 

- The general idea of introducing multi-view consistency constrains to vision-language pretraining models is not new. Some existing works, e.g. DeCLIP [a] have explored adding such multi-view self-supervision constrains and validated its effectiveness when learning vision-language models. The differences here are mainly about applying multi-view constrains on segment level rather than whole image when building upon the framework of Group VIT.

[a] Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm, ICLR 2022

-  Cross-view constrains are applied on segments of two augmented views. Since there is no guarantee for location (segments) correspondence between these two augmented views, the proposed strategy in Figure 5 of constructing positive and negative pairs for the corss-view segments seems not reasonable. 

- The performance of the most important baseline GroupViT seems not consistent with the original paper. For example, in Table 1 Comparison with zero-shot baselines on PASCAL VOC 2012, it reports 51.2 mIoU for GroupViT and 52.4 mIoU for the proposed method ViewCo, while in the original paper [b] GroupViT obtains 52.3 mIoU, which is much higher than 51.2 reported in this submission and is only 0.1 lower than the proposed method. Besdies, on PASCAL Context, GroupViT reports 22.4 mIoU, the improvements from ViewCo seems also marginal (23.0 mIoU v.s. 22.4 mIoU). 

[b] GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The novelty is limited due to the concerns raised in the Strength And Weaknesses section. The code is not provided for reproducibility.
summary_of_the_review: My current rating is based on concerns for limited technical contributions, unreasonable positive/negative pair selection for cross views, quite limited improvments upon baseline GroupViT. Please find more details in the Strength And Weaknesses section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper improves Group VIT, a text-supervised segmentation model, by introducing region-wise cross-view consistent regularization. The author benchmarked for zero-shot segmentation capability across multiple datasets and showed improvements over several baselines. 

My primary concern with this work is selecting the positive segmentation across views, where the author assumes the segments with the same id form positive pairs. I think this assumption is wrong, and I'll expand the discussion to the Weaknesses section. 

This work's contribution is also limited since most of its success is built upon the Group VIT model. The quantitative improvements from the newly introduced regularization are not substantial, around 1-2% over Group VIT.   






strength_and_weaknesses: ### Strength:

(1) This work improves Group VIT by enforcing view consistency

(2) Author conducts substantial experiments across multiple datasets and shows consistent quantitative improvement over the baseline. 

### Weaknesses:

(1)  I think the assumption of cross-view segment consistency by segment id is wrong. As observed in other VIT frameworks, which use learnable tokens for instance segmentation or objective detection, the tokens usually have spatial rather than semantic biases. Based on that, the segment id primarily indicates the segmentation's relative spatial location to the image, and matching group ids does not make sense (we can simply horizontally and vertically flip the images to break the assignment).  

Instead of directly matching segment ids, I think reasonable alternatives are: (a) Compute bipartite matching based on group features similarity and (b) Spatially warp one view to another and compute the segmentation overlapping to match the segments. Then ignore the segment with zero overlapped pixels.   
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the motivation is clear. However, the novelty is limited, and the major assumption doesn't hold. 
summary_of_the_review: Given my concerns over the assumption of the objectives and the limited improvement over the baseline, I tend to reject this submission. 

I'll reconsider my rate if the author can justify why using segment id for cross-view consistency makes sense or provide experimental results to show that segment ids indicate semantic categories rather than spatial bases.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies text-supervised semantic segmentation. The authors propose multi-View Consistent learning (ViewCo) to introduce the correspondence among multiple augmented views of the same image. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strength
+ Compared to the baselines and other methods, the proposed methods could bring some improvements in accuracy.
+ Some ablation experiments are introduced to prove the effectiveness of the proposed method.

Weaknesses
-  The definition of positive pairs and negative pairs in the cross-view consistency module is missing.

clarity,_quality,_novelty_and_reproducibility: There are many hyperparameters and implemental details in the new system, so it could be difficult to reproduce the experimental results without the author's source code. 
summary_of_the_review: Overall, it is a solid paper and gives a reasonable idea to improve text-supervised semantic segmentation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to improve text-supervised segmentation model i.e., GroupVIT by enforcing multi-view semantic consistency. Specifically, a text-to-views consistency module is included to encourage different views to be aligned with the same text input. In addition, there is a cross-view consistency module which enforces features from the different views of the same input image to be closer. The authors conduct extensive experiments on the semantic segmentation benchmarks and show SOTA results.
strength_and_weaknesses: Strength

-The idea of multi-view consistency is new in visual-language learning. This paper proposes a new method to implement the idea. 

-This paper outperforms GroupVIT consistently on multiple benchmarks.  

Weakness

-A potential related work is missing. The DenseCL [A] paper studies multi-view consistency for self-supervised contrastive learning. The author should discuss the difference.

[A] Dense Contrastive Learning for Self-Supervised Visual Pre-Training. Want et al., CVPR'21
clarity,_quality,_novelty_and_reproducibility: Quality: The paper has a good quality and is evaluated extensively.

Clarity: The paper is written well and easy to understand.

Originality: The proposed method is new.

summary_of_the_review: Overall, I think this is a good paper with sufficient novelty and good results. I would recommend to accept it.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I think the paper is above car for acceptance. TTIC is despite two of the reviewers giving it very low scores. In the ensuing discussion there was a lot of new information and clarifications provided by the authors, and the modifications on the paper have significantly improved it. I think the key idea of using multi view consistency in these settings is sufficient novel, and the results convincing enough to make this a worthwhile part of the conference. I very strongly encourage the authors to take all the comments by the expert reviewers to heart, and to incorporate as much material from the discussion into the revised paper as possible.
justification_for_why_not_higher_score: The paper, and in particular the initial submission, has done shortcomings indicated in the reviews. Some of these have been addressed in the modified paper, but the final revision is yet to be made, so I would be a bit conservative about promoting the paper. That said, I probably wouldn't mind if this were a spotlight.
justification_for_why_not_lower_score: Enough novelty in the main idea to justify acceptance.
Decision: Accept: poster