Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a simple one-stage multi-task framework for visual grounding tasks resorting to a transformer architecture. The proposed method achieves state-of-the-art performance on both referring expression comprehension and segmentation tasks.
 
main_review: This paper proposes a simple one-stage multi-task framework for visual grounding tasks resorting to a transformer architecture. The proposed method achieves state-of-the-art performance on both referring expression comprehension and segmentation tasks.

Strengths:
- The paper does a through literature review in multiple research directions. Built upon the transformer models, the paper proposes an effective framework for REC and RES.
- The authors did experiments on multiple datasets and studied various backbones, which could provide insights for future work.

Weaknesses:
- This paper is an application of the Transformer model to a new task, which is ok, but incremental and less exciting. 
- Using multitask learning for REC and RES is not new. There are already some multitask learning methods for REC and RES, for example, Luo et al., "Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation", CVPR 2020. 
- The paper claims “As with other transformer-based architectures, we show that pre-training can further improve the performance and both vanilla and pre-trained models outperform state-of-the-art on both tasks by significant margins”. But for all the pretrained methods, the model is pre-trained for only 6 epoches, making the conclusion not reliable. I understand that this methodological concern can be addressed with more computing resources. But if that is the case, then I would suggest limiting the scope of the experiments --- less but longer training on fewer datasets. 

--------------

The authors' response resolved my concern on training so I raised the score. However, method-wise, this work is sort of applying the multitask learning idea in Luo et al. on top of the Transformer models (either is not new on REC and RES tasks). This work is a combination of different tricks that were proposed before though it achieves the SOTA performance (no new conceptual thinking on REC and RES). So I find it less exciting and insightful.



limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The work explores the tasks of referring expression comprehension (REC) and segmentation (RES) as well as the more general visual grounding. This proposed approach falls into a group of "one-stage" methods, in that it does not rely on an initial step of obtaining region proposals. The method has an end-to-end Transformer-based design, and it jointly optimizes both REC and RES objectives in a multi-task learning framework. The authors also argue for the use of contextualized query encoding. The experiments are conducted on 4 standard tasks/benchmarks. 
main_review: =Originality
- While the overall architecture is not identical to any prior work, the key proposed ideas have been explored previously: multi-task REC/RES training has been proposed in [36], while [a] has explored the role of linguistic context from the full caption.
[a] Yang, Sibei, Guanbin Li, and Yizhou Yu. "Propagating Over Phrase Relations for One-Stage Visual Grounding." European Conference on Computer Vision. Springer, Cham, 2020.

=Quality
- Overall, the paper seems compelling, with a clear delivery, broad set of experiments on 4 tasks and performance improvement across the board. 
- At the same time, the ablation study is somewhat limited, especially in terms of specific architectural design choices. E.g. the proposed decoding mechanism (self-attention graph) is not ablated (by e.g. comparing it to a transformer design).
- Moreover, I would like to see an ablation study performed on REC or RES task in addition to the Flickr30kEntities dataset. 
- I would also like to see more analysis on the source of improvement. E.g., if we consider the task of RES, the vanilla model (w/o joint REC training) already achieves 58.39 (from Tab 4 left), improving over the best prior work (54.21), so what is the source of this improvement? Is it perhaps the effect of a different backbone and/or something else? Generally, what explains the better/more fine-grained quality of the predicted segmentation masks? (Which does not appear to be the focus of this work?)
- The experimental setting on the Flickr30kEntities dataset is different from majority of prior work, in that the authors incorporate the full caption in addition to the individual phrases. While I do believe that this is a meaningful thing to do, I would like to see a clear comparison between prior works that do *not* use the caption and a version of the method w/o a caption, and prior works that *use* the caption vs. the full proposed model. If I read the Table 4 (right) correctly, w/o Context Encoder is comparable in performance to prior works (in Table 3). Also, the authors mention that some prior work also assumes having access to the full caption [1,9] (L282) but those are not reported in Table 3.
- Overall, the improvement from the joint training is larger for the RES task than for the REC task (~1%). The benefit of RES to REC is less obvious.

=Clarity
- The authors mention generating "queries" throughout the paper (L9, ), but it is not obvious whether they are referring to actual generated text, predicted textual embeddings or whether it refers to "queries" as often used in Transformer terminology (query/key/value). This should be clarified early on.
- Fig 1 is rather confusing! It is hard to parse and relate to the main text. E.g. the names used in Fig 1 and in the main text (L138-14) are not consistent. I also missed the fact that (a) and (b) were separate figures as they appear to be one figure, while in fact (b) is a "zoomed in" version.  I do not follow what the "Encoder outputs" refers to. The legend colors are also somewhat ambiguous. Overall, Fig 1 does not help in following along with the method description.
- I do not entirely understand Sec. "Encoding Referring Queries", i.e. the contextualized encoding of input queries. (a) What exactly happens in the tasks of REC/RES where no external caption is available, i.e. would not this be redundant to the query itself? Why should it help then? (b) How does this apply to input image queries (in image-text retrieval)?
-  I also do not understand how the REC/RES task is framed when there are multiple referring expressions associated with the same image, would those be inferred jointly? Would both of them be used as context for each other? If that is true, how is that comparable to prior works?

=Significance
- The work follows several recent (concurrent) works in an attempt to perform visual grounding in an end-to-end manner in a Transformer-based design. I believe such efforts are of interest to the community, and unifying multiple tasks (REC/RES etc) seems like an additional benefit.

=Related Work
- It should be acknowledged that [29] and [56] also perform both REC and RES tasks.
- The paper misses to cite another one-stage approach, [a]. This work also explores the role of linguistic context from the full caption!
[a] Yang, Sibei, Guanbin Li, and Yizhou Yu. "Propagating Over Phrase Relations for One-Stage Visual Grounding." European Conference on Computer Vision. Springer, Cham, 2020.

=Other
- L19: The connection between visual grounding and object detection is not that direct, this misrepresents things a little. I.e., in grounding a text phrase is given as *an input*, while in detection we would need to *output* the label (or text phrase, in a generalized case).
- L155 perceptions => perceptrons
- L138 "four stages" is confusing, as you are arguing that your approach is one-stage.
- L156: unclear why transformers are permutation invariant? it is common to include the positional encoding in the transformer models
- From Table 1 we see the biggest gains on the RefCOCOg dataset, any intuition for why that is?

====== UPDATE

I appreciate the additional ablations/comparisons provided by the authors, as they shed more light on the proposed approach. I also found most of the clarifications helpful. The authors have provided additional details re comparison to [a], and promised to include a comprehensive comparison in the paper.

While I do not strongly share the novelty concerns expressed by Reviewers 7Fb1 and e6Cm, I agree that the proposed method builds upon several ideas expressed in prior work. (And who does not?..)

I still see value in the proposed method, as explained in the "Significance" section above: this one of the first attempts to establish an end-to-end visual grounding pipeline. While it was not required, the authors have compared to several concurrent works in their supplemental material, and showed superior/competitive results.

Thus, I still recommend acceptance and raise my score to 7.
limitations_and_societal_impact: The paper does not present any discussion on potential negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a Transformer-based multi-task framework for visual grounding tasks (referring expression comprehension/segmentation). The core part of this model (comparing with DETR) is the Query Encoder which builds the multimodal query embedding for each referring phrase to decode the grounding results. Extensive experiments and ablations illustrate that the proposed method achieves state-of-the-art performance.
main_review: Strengths
1) REC/RES tasks can be viewed as a general phrase grounding task. Utilizing DETR architecture to formulate phrase grounding task as a set prediction problem is intuitive and straightforward.

2) Extensive experiments show the proposed model achieves state-of-the-art performances on both REC and RES tasks. Training of this model is simpler and faster by removing the Hungarian matching loss.

Weaknesses
1) For the RES task, authors discuss the limitations of previous methods from the perspective of results, but do not clearly explain why the proposed model can achieve huge performance gains. Considering that the proposed model mainly adds a Query Encoder based on DETR to construct multimodal query embeddings for the referring phrases, could the powerful capacity of this model be attributed to the strong multimodal modeling ability of Transformers? I hope authors can further elaborate their contributions and technical novelty comparing with DETR. In addition, what the effects of focal loss and Dice loss which are not adopted by previous methods?

2) Authors argue that an explicit constraint tends to downplay the quality of the final predicted mask since the feature map from the REC branch can blur out fine-grained region shape information needed by the RES branch. I think there could be some quantitative analysis to show that applying the explicit constraints of MCN on the proposed model will downgrade the RES performance to support the above argument.

3) Line 325-326 “Query encoders are also quite …”. But the model component mentioned in Table 4 (right) is “Query Decoder”.

4) How to initialize the designed learnable modal label embedding E_label mentioned in line 160? Could authors give a more detailed explanation?


---- Update ----

After reading the comments of other reviewers, I agree with Reviewer 7Fb1 that this paper is an intuitive and straightforward application of DETR for the general phrase grounding task, but lacks some novelty comparing with other variants of DETR. The removal of Hungarian matching loss reduces the complexity of the proposed model, but I don’t recognize it as a contribution since the one-to-one correspondence between phrase and referred object naturally makes the Hungarian matching loss unnecessary. In addition, other losses used in this paper are more like performance tricks rather than reasonable design for the phrase grounding task. Therefore, I lean towards rejection.

limitations_and_societal_impact: Yes. see main review.
review_title: null

Meta-review: The paper proposes a Transformer-based model for the tasks of referring expression comprehension and segmentation. The reviewers agreed that the techniques used by the paper are not entirely novel and have been individually explored before (especially compared to models like DETR and its variants), even though their combination leads to reasonable performance gains on both tasks. I encourage the authors to add in the ablation studies and analyses from their responses, plus provide better comparison (both conceptually and empirically) with respect to existing methods in their revision.
Decision: Accept (Poster)