Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to integrate visual and linguistic features in both top-down and bottom-up modulation of the visual input. This is done by fusing two modalities while doing convolution and deconvolution operations over the visual input. Experiments on image segmentation from referring expressions in standard datasets show that the proposed approach achieves state-of-the-art or competitive results. Ablation studies show that both top-down and bottom-up is essential. I believe the novelty and contribution are rather thin because many ways of the modeling language are not explored at all. 

Below I list suggestions (S) and questions (q) for authors:

S1 Second paragraph of introductions: please add a figure to explain the concepts of top-down, bottom-up processing, high-level, low-level effects etc.

S2 Section 2.2.: Please cite the below papers [1,2] for referring expression comprehension. For Section 2.4 please add [3]

S3 Figure1: following this figure is not intuitive. I recommend adding two arrows for top-down and bottom-up processing and adding more space between two branches. 

S4 Section 4.2: It is not clear how each of these ablations was performed. For instance, I'm not 100% sure whether two modalities are fused at different levels of top-down or bottom-up processing.

[1] Nagaraja et. al. "Modeling context between objects for referring expression understanding." 
[2] Cirik et. al "Using syntax to ground referring expressions in natural images."
[3] Chen et. al. "Touchdown: Natural language navigation and spatial reasoning in visual street environments." 

S4: Section 4.3: the claim of modeling the long-range dependencies is a bit speculative. I would rephrase that.

S5: Figure2: Failure cases are more informative than successful ones. Please either bring the figure from A.1 or add a comparison with a model from the literature where the other model is successful where yours is not to do a contrastive analysis on how your model can be improved.

Q1: Section3: What's the effect of the number of layers for the model? Why stop at 3? Do you have results for the number of layers 0,1,2?

Q2: Is there a way to interpret the interaction between language and visual input?

Q3: Have you experimented with different ways of fusing or processing language input? Examples: gating the language representation, attention over tokens, using different fusion methods, bi-directional LSTM, BERT-like contextual representations, adding inductive bias with parse tree for referring expressions, alignment between feature maps and word tokens or phrases?

review_title: good experimental setup, lacks the depth of novelty and analyses

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper concerns the problem of image segmentation from referring expressions. Given an image and a query phrase about a particular object in the image, the goal is to locate the target object as a mask at the pixel level. The basic framework is U-Net, which consists of two branches: an image encoder and segmentation map decoder (connected at the bottom in a U-shape). The paper proposes to use language to modulate the image encoding and decoding process intensively, by applying auxiliary convolutional connections between the two branches and further condition the convolution kernel on the language embedding. Overall, the paper is easy to follow and has done a good literature review.

The major concern of the paper lies in the empirical results. Despite that the introduction of top-down and bottom-up language modulation significantly boost the baseline performance (Tab. 1), the full model struggles to match existing works on certain metrics such as UNC testA/testB, UNC+ testB, and ReferIt, which put a question mark on the effectiveness of the work. The results on the validation set are promising but not as good on the test set, which indicates a possible over-tuning of the model.

A minor comment on the model part. In the text above Eq. 1, the paper mentions "[...] ,we split the textual representation [...]". However, what is the rationale for splitting the representation since each split does not attach to any particular abstract of the image feature (low-level, mid-level, and high-level)?

Besides, some numbers from Tab. 1 do not match those from Tab. 2. For instance, the IoU on LSCM and Step-ConvRNN. Please double check.

============== Post-Rebuttal ==============

The authors' responses to point 1 & 2 do not sound (reflecting a question to another paper does not solve the problem). The authors mentioned "We made this decision based on Mei et al (2018) which proposed our baseline model (the top down approach)", where the reference of Mei et al (2018) cannot be found in the paper, as a critical baseline. This raises a flag on the novelty of the work and completeness of the related work. Therefore, I am lowering my rating to 4.
review_title: Interesting idea; empirical results are ok

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: This article proposes a novel approach integrating language throughout the visual pathway for segmenting objects according to referring expressions. 

The article is well written, and poses an important question about how best to integrate linguistic and visual information. The limitations of the currently dominant top-down approach are well argued. The answer proposed by the authors is to integrate linguistic information throughout the visual hierarchy. The task of segmenting by referring expression is important and well chosen. 

The proposed model is sound, and well described in the article, and the experimental results demonstrate that the model outperforms clearly the state-of-the art in all metrics. The qualitative examples provided are quite impressive and demonstrate the success of the approach. 

In sum, I feel this is a well written paper addressing a very timely and important problem in computer vision and AI research and should be of broad interest in the community. 
review_title: Very interesting paper

final_score: rating: 2: Strong rejection
review_content: This paper presents a model for image segmentation from referring expressions which integrates linguistic representations of the referring expressions both at low-level and high-level stages of visual processing. They argue that this model is both more cognitively plausible and more successful than models which only use linguistic representations to modulate attention over high-level visual features.  

I vote for rejection, mainly on grounds of significance and quality, expanded below. To change my vote I would require a substantial improvement on one or both of the quality/significance issues listed below, either presenting the model with a clear conceptual motivation, or doing satisfactory model analysis to understand the contribution of the paper.  

Pros: The presented model shows some moderate quantitative improvement over other recent work.

Cons: Poor conceptual motivation and error analysis, with little evident understanding of the actual effect of the linguistic representations within the model.  

Quality/Significance

1. The paper does not provide a clear motivation for their model. Here are some arguments I looked for but did not find:
  1. Cognitively: There are some references to relevant cognitive science papers, but there seems to be little concrete inspiration taken from this or other cognitive work in the particular model design.
  2. A priori based on the task: What would we expect to gain from using language in early-stage visual representations? What sort of correlations might exist between particular types of linguistic input and low-level visual representations? This might be another way to motivate the model, but I can't find any such discussion in the introduction or anywhere else.
2. The evaluations don't convince me that this paper has made a significant conceptual contribution.
  1. The quantitative results don't seem to constitute an enormous improvement over past work. The variability in table 2 across evaluation sets makes me doubt the statistical significance of the claims. No statistical significance tests (across resamples of test data or across random training restarts) are provided.
  2. There is no satisfactory analysis of the actual cause of the model's success. What are the contents of the linguistic representations, and how exactly do they modulate low-level visual features? For reference, Hu et al. (2020, Figure 4) [1] and Hui et al. (2020, Figure 5) [2] both do some of what I'm looking for here, showing the influence of language on the behavior of the model. While the more complex representations used in this model make it more difficult to provide e.g. an easy heatmap, we absolutely need to see an error analysis that helps us believe your claim that language ought to play a role in low-level visual processing.  

Originality

I don't closely follow the relevant literature and can't speak confidently on the originality of the model. I did have trouble understanding the innovation over Step-ConvRNN, however -- these models seemed within tweaking-distance of one another based on the presentation in this paper.  

[1]: https://openaccess.thecvf.com/content_CVPR_2020/papers/Hu_Bi-Directional_Relationship_Inferring_Network_for_Referring_Image_Segmentation_CVPR_2020_paper.pdf#page=5
[2]: https://arxiv.org/pdf/2010.00515.pdf#page=14

## Post-rebuttal update

I have read the other reviews and the authors' rebuttals, and do not wish to change my review.

I strongly believe that numerical task improvements are not in themselves a conceptual contribution. I look forward to the results of the analyses the authors mention in response to R3-Q2, to better understand what exact interaction between language and low-level visual input is being modeled.

Along with R4 I remain unconvinced of the strength of the empirical results. The authors' response is not helpful here. I can't understand where the numbers (mean 60.74 IoU, std 0.06) come from -- taking stats across table 2 and table 1, I get very different results, so I must be misunderstanding where they come from.

Significance tests would not take too much time -- it's not absolutely critical that you retrain the models for this. You can use data resampling methods instead. For example, on each individual dataset, run bootstrap tests comparing the predictions of your model and others on random resamples of the evaluation data and corresponding predictions.
Pooling IoU results across datasets within model and then comparing between models can yield misleading results and should be avoided. 
review_title: Possibly worthwhile idea, but poorly motivated and evaluated

Meta-review: The paper proposes to improve image segmentation from referring expression  by integrating visual and language features using an UNet architecture and experimenting with top-down, bottom-up, and combined (dual) modulation.  

Review Summary: The submission received divergent reviews with scores spanning from 2 (R2) to 5 (R3,R4) to 10 (R1).  The author response failed to address the reviewer concerns with some reviewer (R4) lowering their score tto 4 after the rebuttal.  It also became clear that some relevant work (Mei et al, 2018) was used for the baseline but not cited.  The author response also did not recognize the importance of significance tests.

As there is considerable work in the area of image segmentation from referring expression, and the proposed model is very similar to the LingUNet model of Misra 2018, the originality and significance of the work is fairly low.  The main contributions appears to be experimental comparisons of the three types of modulation (top-down, bottom-up, dual).

Pros:
- Investigation of a important problem of grounding language to visual regions
- Experimental study of whether dual modulation improves image segmentation from referring expression

Cons
- Relatively minor novelty with limited analyses (R3,R2)
- Missing citations (see R3's comments).  Relevant work (Mei et al, 2018) which was the basis for the top-down baseline model, was used but not cited or properly compared against
- Relatively weak experimental results (R2,R4).  As R4 noted, while validation results are good, test results are weak compared to existing work, indicating potential overtuning.  
- No qualitative comparisons against baselines.  
- Cognitive claims not backed up and limited discussion/analysis (R2)

Recommendation:
The AC concurs with R2, R3, and R4 that the work is limited in novelty and not ready for publication at ICLR.   Despite R1's high score, referring expression for image segmentation is a well studied task, and it is unclear what are the key innovations of the proposed model over LingUNet.  Due to the limited novelty, relatively weak test results, as well as other flaws pointed out by the reviewers, the AC recommends rejection.

Decision: Reject