Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a cross-modal alignment model (CoupAlign) for referring image segmentation.  The authors propose to use coupled sentence-mask alignments with word-pixel alignment to enforce the model learned more accurate and consistent segmentation masks. Word-pixel alignment serves as an early fusion module that fuses the features between each block of swing transformer feature and BERT feature. The Sentence mask alignment learns to weight the mask using the sentence embedding to localize the referred object. The authors benchmark the proposed method on 2 refer segmentation benchmarks and achieve state-of-the-art performance. 
strengths_and_weaknesses: [Strength]

- WPA module serves as an early fusion module that can balance the fusion and compute. 

-  The sentence mask alignment (SMA) module is a general module that can scale up to weakly supervised learning such as GLID. 

- The proposed model achieves state-of-the-art performance on refer segmentation benchmark.  

- The visualization of word-pixel alignment shows the effectiveness of the module and aux loss. 

[Weakness]

- It would be good to conduct an ablation study on how many WPA module is needed in the model. Does the early fusion (first few blocks of the model) need the WPA module or the more WAP module the better?  

- It seems the WPA module use cross-attention similar to [1], so it would be good to compare the difference between different cross-attention operations. Why not use multi-head attention on the WAP module?

[1] Lu, J., Yang, J., Batra, D. and Parikh, D., 2016. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29.


questions: - Is it possible to extend the current framework to incorporate weakly supervised data such as image caption pairs? 

- What is the role of WPA module and how many do we need it. What is the good balance between fusion and computing? 

- For cross attention, why not use multi-head attention? is there any specific reason? 

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In order to employ the semantic consistency of pixels within the same object, this paper proposes a CoupAlign mechanism to couple a work-pixel alignment (WPA) and a sentence-mask alignment (SMA). WPA fuses the linguistic and pixel-level features within the intermediate layers of the feature encoders. SMA weights the generated masks to localize the referred object. In addition, the authors provide an auxiliary contrastive loss for facilitating the segmentation accuracy. The experiments show that the proposed CoupAlign mechanism achieves state-of-the-art referring image segmentation performance.
strengths_and_weaknesses: [Strengths] 
+ The idea of implementing the CoupAlign mechanism is interesting.
+ The manuscript is well organized, and most paragraphs are easy to follow.
+ The references are adequate.

[Weaknesses]
- The experiment misses evaluating standard datasets, ReferIt and RefCOCO+, for comparison with other RIS methods. Such an incomplete comparison makes the evaluating experiments weak.
- Missing some function descriptions harms the readers to reimplement the proposed model. For example, what are the functions of SwinStage and BERTStage in (3)? It is unclear why these functions relate to two different stages. Also, what is the function of CrossAttn in (8) and MaskGenerator in (9)? Please explain these functions' structures mentioned above, including the inputs and outputs, to clarify the model design.
- The shapes of the projection matrices mentioned in section 3.2 are not correct.

questions: Though the proposed CoupAlign mechanism coupling WPA and SMA is interesting, the manuscript lacks some clear terminology definitions, as mentioned in [Weaknesses], and the experiments are not supported by sufficient standard evaluating datasets. I would like to see more information in the authors’ response.
limitations:  The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new model for referring image segmentation (RIS). The model takes in an image and a natural language description as input, and outputs the segmentation mask corresponding to the language input. The proposed approach features 1) a hierarchical fusion architecture in its encoder to gradually fuse information from the visual and language inputs (dubbed as Word-Pixel Alignment or WPA), and 2) a module to assemble segmentation masks from a set of candidate masks and the language representation (dubbed as Sentence-Mask Alignment or SMA). In terms of experiments, the authors train and test their models for RIS on two datasets -- RefCOCO and G-Ref, and achieved competitive results compared to previous SOTA methods. The authors also carried out ablation studies on various design choices and provided some qualitative visualizations to better understand the model's behavior.  

strengths_and_weaknesses: Strengths
+ The task of segmenting images with arbitrary language input is interesting and could be a prominent task in near future. 
+ For presentation, 1) the illustration in Fig. 2 is comprehensive and helpful and 2) the visualization in Fig. 4 is very helpful for readers to better understand the model behavior. 
+ Overall, the proposed model is intuitive and achieves strong performance when compared to other SOTA methods on popular datasets.

Weaknesses
- Notations are confusing. For example, there are d_k, d_s, d_q, are they equal to each other? In Eq 11, L_g is with shape 1 x D while Q_o is with N x d_q, does this imply D == d_q? Also, in Eq 12, Y_1 is with shape H/4 x W/4 x d_s while Q_o is with size N x d_q, how can you multiply these two (unless d_q == d_s, which I did not see stated anywhere)?   
- Since both WPA and Cross Attention are fusing the visual and language information, why using different attention computation for these two? The design seems arbitrary and I did not see any justification for the difference. 
- Fig 3, are these examples representative or through cherry picking? Since I did not see why the proposed method would have a qualitatively different behavior compared to previous models (e.g. LAVT) by adding more intermediate level fusion. If this is not representative, I think it's better to not include these as they can be misleading (i.e. to make readers think the proposed model is qualitatively better/different). 
- L281-6, since this is where the main technical novelty lie, it would be better to have more detailed ablations on questions like how many layers of fusion is needed? Or which fusion layer contributes the most? 

############# POST REBUTTAL NOTE #############
The authors have properly addressed my comments above with the added experiments and paper revisions. With this, I will raise up my rating to accept the paper. 









questions: - In Eq 5, V'_i is "assembled" from the language feature while L'_i is assembled from the visual feature? Should these two be swapped? 
- In Eq 14, is y^-_k sampled from the same image or from some other images? 
- L287, "when SMA is removed", how do you remove SMA while still generating the segmentation conditioned on the language input? This requires more elaboration. 

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes CoupAlign for referring image segmentation, which couples sentence-mask alignment with word-pixel alignment to enforce the mask-aware constraint. Experiments validate the effectiveness of the method.
strengths_and_weaknesses: Strengths:

- The paper is mostly clear and easy to follow.
- The writing of this paper is good and the figures are appealing.

Weaknesses

- Novelty. The basic idea of the proposed CoupAlign framework is to couple sentence-mask alignment with word-pixel alignment for consistent and accurate segmentation results. However, the proposed Word-Pixel Alignment (WPA) module is just the same as the Language-Aware Deep Fusion proposed in GLIP [1]. The Auxiliary Loss can be regarded as a simple supervised pixel-level contrastive learning loss, which has been demonstrated by recent works [2]. The proposed Sentence-Mask Alignment (SMA) module can be a contribution, but I think it is just a cross-modal version weighted-summation of queries from the DETR-decoder-like mask generator. In conclusion, the novelty of this paper is limited.

[1] Grounded Language-Image Pre-training, CVPR, 2022.

[2] Exploring Cross-Image Pixel Contrast for Semantic Segmentation, ICCV, 2021.

questions: - The precision@0.5/0.7/0.9 values of the 3-rd and 4-th row are the same. Is this a typo error? Or it is just a coincidence.
- The attention weights on different words are not analyzed in the WPA module. It is better to show if the model correctly attends to the referring motion or appearance words.
- In table 4, I think there is one experiment missing, i.e., w/ Bi-WPA, w/o Uni-WPA, w/o SMA, w/o Aux Loss.
- From Figure 2 and Table 4, I think the baseline without SMA is that directly summarizes all mask query embeddings without weights generated from the sentence features. What if evaluating the importance of each mask embedding from themselves? That is, generate the weights from the mask embeddings (e.g., using MLP like the equation (3) in [3]) without the sentence embeddings.
- Could you please show the diversity of masks generate from mask queries by statistics?
- The computational cost or runtime should be discussed.
- Experiments on RefCOCO+ [4] are missing. Why?
- Why the improvement in the test split of G-Ref is so small (i.e., 0.13%) but significant on Val split (i.e., 1.6%). Does it mean the proposed methods overfit the train and Val splits?
- The improvements of WPA when inserting it into different stages of the encoders should be discussed. And what if only using the words features from the last stage of the language encoder instead of from the intermediate layers.

[3] SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation, Arxiv, 2021.
[4] Modeling context in referring expressions, ECCV, 2016.

limitations: The limitations and potential negative societal impact are discussed in the supplementary files.
review_title: null

Meta-review: metareview: The paper was reviewed by four reviewers and received all positive scores at the end: 2 x Borderline Accepts and 2 x Weak Accepts. Most initial concerns with the paper were with exposition and experimental validation. These concerns, however, were addressed convincingly during the rebuttal period with additional experiments and ablations, as well as direct edits to the manuscript itself. In the current form the paper would be a valuable contribution to NeurIPS program. 
confidence: Certain
Decision: Accept