Language Controls More Than Top-Down Attention: Modulating Bottom-Up Visual Processing with Referring Expressions
Authors: Ozan Arkan Can
Ilker Kesen
Deniz Yuret
Venue:  (2021)
Keywords: Referring Expression Understanding
Language-Vision Problems
Grounded Language Understanding

Abstract: How to best integrate linguistic and perceptual processing in multimodal tasks is an important open problem. In this work we argue that the common technique of using language to direct visual attention over high-level visual features may not be optimal. Using language throughout the bottom-up visual pathway, going from pixels to high-level features, may be necessary. Our experiments on several English referring expression datasets show significant improvements when language is used to control the filters for bottom-up visual processing in addition to top-down attention.

Under review as a conference paper at ICLR 2021
LANGUAGE CONTROLS MORE THAN TOP-DowN AT- TENTION: MODULATING BOTTOM-UP VISUAL PRO- CESSING WITH REFERRING EXPRESSIONS
Anonymous authors Paper under double-blind review
ABSTRACT
How to best integrate linguistic and perceptual processing in multimodal tasks is an important open problem. In this work we argue that the common technique of using language to direct visual attention over high-level visual features may not be optimal. Using language throughout the bottom-up visual pathway, going from pixels to high-level features, may be necessary. Our experiments several English referring expression datasets show significant improvements when lan- guage is used to control the filters for bottom-up visual processing in addition top-down attention.
INTRODUCTION
As human beings, we can easily understand the surrounding environment with our visual system and interact with each other using language. Since the work of [Winograd (1972). developing system that understands human language in a situated environment is one of the long-standing goals of artificial intelligence. Recent successes of deep learning studies in both language and vision domains have increased the interest in tasks that combine language and vision (Antol et al., 2015 Xu al. 2015 Krishna al. 2016 Suhr et al.) 2017 Anderson al. 2018b Hudson & Manning 2019). However, how to best integrate linguistic and perceptual processing is still an important open problem. In this work we investigate whether language should be used to control the filters for bottom-up visual processing as well as top-down attention.
In the human visual system, attention is driven by both "top-down" cognitive processes (e.g. focus- ing on target's color or location) and "bottom-up" salient, behaviourally relevant stimuli (e.g. fast moving objects) (Corbetta & Shulman 2002) Connor et al.) [2004] Theeuwes) (210)). Studies on em- bodied language explore the link between linguistic and perceptual representations (Pulvermüller)) 1999 Vigliocoo et al. 2004; Gallese & Lakoff 2005) and it is often assumed that language has high-level effect perception and drives the "top-down" visual attention (Bloom) 2002 Jack-
endoff & Jackendoff] [002] Dessalegn & Landau| (2008). However, recent studies from cogmitive science point out that language comprehension also affects low-level visual processing (Meteyard et al. 2007 Boutonnet & Lupyan, 2015). Motivated by this, we propose a mode that can modulate either or both of "bottom--pp" and "top-down" " visual processing with language conditional filters.
Current deep learning systems for language-vision tasks typically start with low-level image pro- cessing that is not conditioned on language, then connect the language representation with high level visual features to control the visual focus. To integrate both modalities, concatenation Ma--
linowski al. 2013). clement-wise multiplication (Malinowski et al. 2015 Lu al. 2016 Kim et al. 2016) or attention from language to vision (Xu et al.] 2015][Xu & Saenko 2016 Yang et al. 2016 al. 2017 Anderson al. 2018a| Zellers et al. (2019) may be used. Specifically they do not condition low-level visual features on language. One exception is De Vries et al. (2017) which proposes conditioning the ResNet (He et al., 2016) image processing network with language conditioned batch normalization parameters every stage. Our model differs from these architec- tures by having explicit "bottom-up" and "top-down" branches and allowing us to experiment with modulating one both branches with language generated kernels.
We will release our code and pre-trained models along with reproducible environment after the blind review process.

---

Under review as a conference paper at ICLR 2021
We evaluate our proposed model on the task of image segmentation from referring expressions where given an image and a natural language description, the model returns a segmentation mask that marks the object(s) described. We can contrast this with purely image based object detection (Gir- shick
2015; Ren et al. 2017) and semantic segmentation (Long et al. 2015 Ronneberger et al., Chen al., 2017) tasks which are limited predefined semantic classes. Our task gives users more flexibility to interact with the system by allowing them to describe objects of interest in free form language. The language input may contain various visual attributes (e.g., color, shape), spatial information (e.g., "on the right", "in front of"), actions (e.g.. "running". "sitting") and inter- actions/relations between different objects (e.g., "arm of the chair that the cat is sitting in"). This makes the task both more challenging and suitable for comparing different strategies of language control.
2015
The perceptual module of our model is based on the U-Net image segmentation architecture (Ron- neberger et al. 2013). This architecture has clearly separated bottom-up and top-down branches which allows to easily vary what parts are conditioned on language. The bottom-up branch starts from low level visual features and applies a sequence contracting filters that result in successively higher level feature maps with lower spatial resolution. Following this is top-down branch which takes the final low resolution feature map and applies a sequence of expanding filters that even- tually result in segmentation mask the original image resolution. Information flows between branches through skip connections between contracting and expanding filters at the same level. We experiment with conditioning one or both of these branches with language.
To make visual processing conditional on language, we add language-conditional filters at each level of the architecture, similar to Misra et al. (2018). Our baseline only applies language- conditional filters on the top-down branch. Modulating only the top-down/expanding branch with language means the high level features extracted by the bottom- nattom-up/contracting branch cannot be language-condee coniitional. Our model expands on this baseline by modulating both branches with language-condittonal filters. Empirically, we find that adding language modulation to the bottom- up/contracting branch has significant positive improvement on the baseline model. Our proposed model achieves state-of the art performance on three different English referring expression datasets.
2 RELATED WORK
In this section, we review related work in several related areas: Semantic segmentation classifies the object category of each pixel in an image without language input. Referring expression comprehen- sion locates a bounding box for the object(s) described in the language input. Image segmentation from referring expressions generates segmentation mask for the object(s) described in the language input. We also cover work on language-conditional (dynamic) filters and studies that use them to modulate deep-learning models with language.
2.1 SEMANTIC SEGMENTATION
Primitive semantic segmentation models are based on Fully Convolutional Networks (FCN) (Long et al. 2015). Deeplab (Chen et al. 2017) and U-Net (Ronneberger et al. 2015) are the most notable ofute-of-the-ard art semantic segmentation models related to our work. DeepLab replaces regular convolutions with atrous (dilated) convolutions in the last residual block of ResNets (He et al. 20166) and implements Atrous Spatial Pyramid Pooling (ASPP) which fuses multi-scale visual information. The U-Net architecture (Ronneberger et al.] 2015) improves over the standard FCN by connecting contracting (bottom-up) and expanding (top-down) paths at the same resolution: the output of the encoder layer at each level is passed to the decoder at the same level.
2.2 REFERRING EXPRESSION COMPREHENSION
Early models for this task were typically built using a hybrid LSTM-CNN architecture (Hu et al.
2016b. Mao et al. 2016). Newer models (Hu et al. 2017 Yu et al.
2016 2018 8 Wang et al. 2019 2017
use an Region- based CNN (R-CNN) variant (Girshick et al., 2014
Region-based
Ren et al. 2017 He et al.
as sub-component to generate object proposals. Nagaraja al. (2016) proposes solution based
on multiple instance learning. Cirik et al. (2018) implements a model based on Neural Module Net-
works (NMN) by using syntax information. Among the literature, Compositional Modular Network
2

---

Under review as a conference paper at ICLR 2021
(CMN) (Hu et al. ule Tree Networks (NMTree) (Liu et al. 2019) are the most notable state-of-the-- methods, and all of them are based on NMN (Andreas et al. 2016).
2017), Modular Attention Network (MAttNet) (Yu et al. 2018 and Neural Mod-
2.3 IMAGE SEGMENTATION FROM REFERRING EXPRESSIONS
Notable models for this task include Recurrent Multimodal Interaction (RMI) model (Liu et al.
2017),
Recurrent Refinement Networks (RRN) Lii et al.] 2018), Dynamic Multimodal Network (DMN) (Margffoy- Tuay et al. 2018), Convolutional RNN with See-through-Text Embedding Pixel-
wise heatmaps (Step-ConvRNN or BOIRRN--STEM (Chen et al., 2019a). Caption-aware Consis-
tent Segmentation Model (CAC) (Chen et al. 2019b), Bi-directional Relationship Inferring Network (BRINet et al. (2020) and Linguistic Structure guided Context Modelling (LSCM) module Hui et al. (2020). RRN which has a structure similar U-Net, is built top of a Convolutional LSTM (SonvLSTM) (SHI et al. 2015) network. Unlike our model, ConvLSTM filters are not generated from language representation and the multi-modal representation is used only in the initial time step. filters for language representation of each word. It per- forms convolution operation on visual representation with language-condee conditioonl filters to generate multi-modal representation for each word. Like RMI, word-level multi- multi-modal representations are fed as input to a multi-modal RRN to obtain multi-modal representation for image/language pairs. Step-ConvRNN starts with visual-textual co-embedding and uses ConvRNN to iteratively re- fine a heatmap for image segmentation. Step-ConvRNN uses a bottom-up and top-down approach similar to this work, however, our model uses spatial language generated kernels within a simpler architecture. CAC also generates x language-conditional dynamic filters. Unlike our model, CAC applies these dynamic filters to single resolution single feature map and additionally gen- crates Docation-speciff dynamic filters (e.g. left, bottom) to capture relations between the objects exist at the different parts of the image. BRINet implements two different attention mechanisms: Language-guidee visual attention and vision-gen sirion-giiied linguistic attention. LSCM implements a depen- dency parsing guided bottom-up attention mechanism to predict masks.
DMN generates x language-condition
2.4 LANGUAGE-CONDITION
FILTERS
To control a deep learning model with language, early work such Modulated ResNet (MOD- ERN) (De Vries et al. 2017) and Feature-wise Linear Modulation (FiLM) (Perez et al. 2018) used conditional batch normalization layers with only sunguage-condition
coefficients rather than cus- tomized filters. Finn et al. (2016) generates conditiond dynamic filters. Li ct al. (017)) is the first work which generates dynamic manguage-condition solution method which has a group convolutional layer whose filters are generated from the question input. Gavrilyuk et al. (018 introduces new task called actor and action segmentation and to solve this task, proposes an architecture which uses dynamic filters for multiple resolutions. Similar our work, |Misra et al. (2018) adds language conditional filters to a U-Net based architecture for the task of mapping instructions to actions in virtual environments. also uses an architecture based on U-Net and Misra et al.) 2018 to solve navigation and spatial reasoning problem. Those models only modulate top-down visual processing with language.
filters. Gao et al. (2018) proposes VQA
Referring expression models that incorporate language-condicee conditional filters into the architecture in- clude (Chen et al.) 2019b] Margffoy- et al.] (2018). Margffoy- Tuay et al. (2018) generates language-conditional filters for words individually rather than whole sentence. [hen et al. (2019b) generates x 1 language-conditional filters from expressions. To make x language-conditional filters spatially aware, different filters are generated for different image regions (c.g. top, left, right, bottom).
Our main contribution in this work is an explicit evaluation of language conditional filters for bottom-up visual processing in comparison to only using language for top-down attention control.
3 MODEL
Figure shows overview of our proposed architecture. For given referring expression S and an input image I, the task is predicting a segmentation mask M that covers the object(s) referred to. First, the model extracts 64 64 1024 tensor of low-level features using backbone convolutional
3

---

Under review as a conference paper at ICLR 2021
neural network and encodes the referring expression S to vector representation using long short- term memory (LSTM) network (Hochreiter Schmidhuber 1997). Starting with the visual feature tensor, the model generates feature maps contracting and expanding path where the final map represents the segmentation mask, similar to U-Net (Ronneherger et al.) 2015). 3x3 convolutional filters generated from the language representation (language kernels) used modulate both the contracting and the expanding paths, Our experiments show that modulating both paths improves the performance dramatically.
Refarring expression S
ret vosse that the right two other simifar red yase
[--
Pypa Asage
Segnentatoor Mask
Referring expresston representation
DeepLab Resnut -01-22
Layer 4
Features
Kiu
Kau
Ka
K1d
K
Language
F1
Festures
CONV
Features
HI
Convolved
CONN
Convolved
CONY
Featunes
Peatures Convolved
H2
CONY
Convolved
CONV
F13
Convolved
CONN
Convolved
Figure 1: Overview of our model.
3.1 LOW-LEVEL IMAGE FEATURES
Given an input image I, we extract visual features from the fourth layer of the DeepLab ResNet101-
v2 network (Chen al. 2017) pre-trained the Pascal VOC dataset (Everingham al.. 2010). We set W =H 512 as the image size for our experiments. Thus, the output of the fourth convolutional layer of DeepLab ResNec1011-2 produces feature map with the size of (64, 64) and 1024 channels for this setup. We concatenate 8-D location features to this feature map following previous work (Hu al. 2016b; Liu al. 2017 Ye et al. 2019 Chen al., 2019a). The final representation, Io, has 1032 channels, and the spatial dimensions are (64, 64).
3.2 LANGUAGE REPRESENTATION
Consider a referring expression S [Wi, ..., Wn] where w; represents the i'th word. In this work, each word is represented with a 300-dimensional GloVe embedding (Pennington et al.) 2014 i.c. w, E 300 We map the referring expression S to hidden states using a long short-term memory network (Hochreiter & Schmidhuber 1997) h, LSTM((;-1. W;). We use the final hidden state of the LSTM as the textual representation, r h We set the size of hidden states to 256, i.e. hy 2256

---

Under review as a conference paper at ICLR 2021
3.3 SEGMENTATION MODEL
After generating image (1)) and language (r) representations, our model generates segmentation Mask M. We take the U-Net (Ronneberger et al. 2013) image segmentation model as the visual processing backbone. Our model extends the U-Net by conditioning both contracting and expanding branches on language using spatial language kernels.
Our model applies convolutional modules to the image representation fo. Each module, F;, takes the concatenation of the previously generated feature map (Down; 1) and its convolved version with 3 x language kernel Ka and produces an output feature map (Down,). Each F; has 2D convo- lution layer followed by batch normalization (loffe & Szegedy 2015) and ReLU activation function (Maas al. (003)). The convolution layers have 5 x 5 filters with stride 2 and padding halving the spatial resolution, and they all have the same number of output channels.
Following Mísra et al. (2018), we split the textual representation
equal parts (1,) generate Iinguage-conditions filters (language kernels). We use each & to generate language-conditional kernel (Kaa):
Kid = AFFINE, (DROPOUT(())
(1)
Each AFFINE; is affine transformation followed by normalizing and reshaping to convert the output to a convolutional filter. DROPOUT the dropout regularization (Srivastava et al. 2014).
After obtaining the kernel, convolve it over the feature map obtained from the previous module to relate expressions to image features:
id = CONVOLVE Down;-1)
(2)
Then, the concatenation of the resulting text-modulated features (Gsa) and the previously generated features (Down;-1) 1) is fed into module F; for the next step.
In the expanding branch, generate m feature maps starting from the final output of the contracting branch as follows:
Gju- = CONVOLVE(Kj.)) [,)
(3)
U Upm = Hm
(4)
Up, Hy @ Up3-1)
(5)
Similar to the bottom-up up phase, Gju the modulated feature map with language-conditiong kernels generated as follows:
Kju- AFFINE, (DROPOUTR))))
where AFFINE, is again an affine transformation followed by normalizing and reshaping. Here, convolve the kernel (Kju) upsampling module Hm gets the concatenation (+) of the text related features and the feature map (Upz) generated from the previous module. Only the first module operates on just convolved fea- tures. Each H2 consists of 2D deconvolution layer followed by a batch normalization and ReLU activation function. The deconvolution layers have doubling the spatial resolution, and they all have the same number of output channels.
the feature maps from the contracting branch (Down,). Each
filters with stride
and padding
After generating the final feature map UP1, we apply stack of layers (D1, D2. Dm)) map Upi to the exact image size. Similar to upsampling modules, each D₃ is 2D deconvolution layer followed by batch normalization and the ReLU activation. The deconvolutional layer has 5 filters with stride = 2 and podding = 2 to double the spatial sizes of the input. Each D; preserves the number channels except for the last one which maps the features to a single channel for the mask prediction. There is batch norm operation and the ReLU activation for the final module, instead we apply a sigmoid function to turn the final features into probabilities (P E #HW).
3.4
LEARNING
Given the probabilities (P R'x for each pixel belonging to the target object(s), and the ground-truth mask E IR HWW the main training objective is the wixe Binary- Caany--ross-EEr Enttroy (BCE) loss:
5

---

Under review as a conference paper at ICLR 2021
Table 1: Ablation results and comparison with the previous works the val set of UNC dataset with prec@X X and IoU metrics.
Method
prec
prec@00.6
prec@007
pree
prec@00
loU
Top-Down Modulation w/ FiLM layers
60.97
53.19
43.71
31.62
10.57
54.21
Top-Down Modulation w/ 1xl filters
63.00
54.20
44.71
32.01
10.74
55.04
Top-Down Modulation w/ 3x3 filters
64.29
55.26
46.29
32.96
11.78
56.13
Bottom-Up Modulation (disconnected) w/ 3x3 filters
69.92
62.84
52.16
32.70
7.18
58.77
Bottom-Up Up Modulation
filters
72.13
65.92
17.49
Dual Modulation w/ 1x1 filters
71.76
65.77
58.19
44.80
17.05
60.75
Dual Modulation w/ 3x3 filters (full model)
73.53
67.53
60.00
46,96
18,80
61.95
LSCM (Hui et al.
2020
70.84
63.82
53.67
38.69
12.06
61.54
BRINet
2020
71.83
65.05
55.64
39.36
11.21
61.35
Mep-CooivRINN lien
al. 2019a
70.15
63.37
53.15
36.53
10.45
59.13
CMSA (e ct 2019
66.44
59.70
50.77
35.52
10.96
58.32
RRN (Lret al. 2018
52.5
42.4
28.13
8.51
55.33
DMN [Marettoy-
2018
65.83
57.82
46.80
27.64
5.12
54.83
RMI
Liu ela.. 2017
42.99
33.24
22.75
12.11
2.23
45.18
H W
J
ΣΣ i
Giylog (Po)) + (1-Gi))log(1 P.j) P
HW
4 EXPERIMENTS
In this section we first give the details of the datasets and our experimental configurations (Section 4.T. detailed analysis of the contribution of our idea and the different parts of the architecture given in Section 4.2 Then present our main results and compare our model with the state-of- the-art (Section
Finally, Section 4.4 shows some qualitative results.
4.1 DATASETS AND EXPERIMENT SETUP
Datasets: We evaluate our model on and Referlt (130.5k expressions, 19.9k images), UNC (142k expressions, 20k images), UNC+ (141.5k expressions, 20k images) eYu 2016) and Google-Ref (G-Ref) (104.5k expressions, 26.7k images) (Mao et al. 2016 (Kazemzadeh et al.) 2014) datasets. Unlike UNC, location-specific expressions are excluded UNC+ through enforcing annotators to describe objects by their appearance. ReferIt, UNC, UNC+ datasets are collected through two- player game (Kazemzadeh et al., 2014) and have short expressions (avg. words). G-Ref have longer and richer expressions, since its expressions are collected from Amazon Mechanical Turk instead of a two-player game. ReferIt images are collected from IAPR Tc-12 dataset (Escalante) al. 2010) and the others images present in MS COCO dataset (Lin al. 2014).
Evaluation Metrics: Following the previous work (Liu et al. 2017 Margffoy- Tuay et al. 2018 Ye al. 22199; Chen al., 2019a),, we use overall intersection-over-une as evaluation metrics. Given the predicted segmentation mask and the ground truth, the loU metric is the ratio between the intersection and the union of the two. The overall loU calculates the total intersection over total union score. The second metric, precision@.X. calculates the percentage of test examples that have IoU score higher than the threshold X. In experiments, X {0.5, 0.6, 0.7, 0.8, 0.9).
(IoU) and precision X
Implementation Details: As (Liu et al.) 2017, Margffoy-Tuss et al., 2018 Ye et al. 2019 Chen ct al. 2019a),, we limit the maximum length of expressions 20. In all convolutional layers, we set the filter size, stride, and number of filters (ch) as (5,5), 2, and 96, respectively. The depth is 4 in the U-Net part of the network. We set the dropout probability to 0.2 throughout the network. We use Adam optimizer [Kigmma & Ba 2014) with default parameters. We freeze the DeepLab ResNet101-v2 weights. There are 60 examples in each minibatch. We train our model for 15 epochs on Tesla V100 GPU and each epoch takes most two hours depending on the dataset.

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison with the previous works four datasets. Evaluation metric is the overall loU and higher better. Bold scores indicate the state-of-the-art performances. n_" indicates that the model has not been evaluated the dataset. "n/a" indicates that splits not same.
UNC testA
UNC+ testA
G-Ref val
Referlt test
Method
testB
val
test
CNN+LSTM (Hu el al. 2016a
28. 14
48.03
RMI (Liu et al. 2017
45,
45,69
45.57
29.86
30.48
29.5
34.52
58.73
DMN (Mrgttoy Tuay al.
2018
49.78
54.83
45.13
38.88
44.22
32.29
36.76
52.81
DynamicFiliers
al. 2017
54.30
KWA (Shi al. 2018
36.92
59.09
RRN (T.
55.33
57.26
53.93
39.75
42.15 15
36.11
36.45
63.63
CMSA Ye al.
2019
58.32
60.61
55.09
43.76
47.6
37.89
39.98
63.80
CAC (Chen et al.
2019b
58,90
61.77
53.81
44.32
Step-ConyRNI
ben al al. 2019a
60.04
63.46
57.97
48.19
52.33
40.41
46.4
64.13
BRINet (Hu ct af.
2020
61.35
63.37
59.57
48.57
52.87
42.13
48.04
63.46
LSCM Huu et al. 2020
61.47
64.99
59.55
49.34
53.12
43.50
48.05
66.57
MAttNet et al. 2018
56.51
62.37
51.70
46.67
52.39
40.08
n/a
NMTree (La al.
2019
56.59
63.02
52.06
47.40
53.01
41.56
n/a
Our Model
61.95
63.85
58.14
50.42
54.16
42.15
49.76
64.63
4.2 ABLATION RESULTS
We performed ablation studies to better understand the contributions of the different parts of model. Table [ shows the performances of the different architectures on the UNC validation split with prec @ X and overall IoU metrics. Unless otherwise specified, 3 language-conditional filters are used in our models.
Modulating both top-down and bottom-up visual processing: We implemented three models, Top-down Modulation, Bottom-Up Modulation and Dual Modulation, to show the effect of mod- ulating language in expanding and contracting visual branches. Since language information leaks through cross-connections between visual branches, we also experimented with bottom-up modu- lation model which has no connection between visual branches. Bottom-up Modulation outperforms Top-down Modulation with 2:4.6 IoU improvement. Modulating language in both visual branches yields the best results by improving Bottom-up Modulation model with 1.2 IoU score.
Language-conditione Spatial Filters: When compare the performances of Top-Down Modula- tion w/ 1x1 filters and Top-Down Modulation models, we see that the usage of language-conditional spatial filters brings additional improvement in our full model, the performance of the model decreases significantly. We performed the same experiment G-Ref dataset and observed 1.33 ToU difference again.
the base model. Similarly, if we use 1 1 filters
FiLM layers vs. Language-conditional Filters: Another method for modulating language using conditional batch normalization De Vries et al. (2017) its successor, FiLM layers. Thus, also replaced language-conditional filters with FiLM layers in Top-Down Modulation w/ 1x1 model and observed =00.8 IoU improvement. Morever, since can take advantage of language-conditional spatial filters, Top-Down Modulation w/ 3x3 model baseline outperforms its FiLM variation with ~ IoU improvement.
4.3 QUANTITATIVE RESULTS
Table shows the comparison of our model with the previous work. Our model outperforms all previous models all datasets. When compare our model with the previous state-of-the-an model. Step-Conv RNN, the most significant improvement is on the G-Ref dataset.
We also compare our model with MAuNet and NMTree which are referring expression comprehen- sion models. Since they present segmentation results after they predict bounding boxes for objects, they comparable with our work. Our model is significantly better than MAttNet. NMTree which depends on an explicit object proposal network that is trained on more COCO images. This result shows the ability of model to detect object regions and relate them with expressions.

---

Under review as a conference paper at ICLR 2021
Expression:
Jonger banana
shorter banana
the heart
Expression:
bug un lef)
second bug from
secund bug from right bag: un right
Expression:
sandwich
sandwieh in front
white howl
sandwich sext to white bowl
Expression:
sitting
Expression:
standing
red
Figure 2: Some correct predictions of model UNC validation set. First column shows the input images and others show the predictions for the given referring expressions.
Table presents the comparison our model with the state-of-ther thet--r-t- in terms of prec@.X X scores. The difference between our model and the state-of-ie sotte--o--hh-aaee art increases when the threshold increases. This indicates that our model is better at both finding and segmenting the referred objects.
4.4 QUALITATIVE RESULTS
In this section, visualize some of the segmentation predictions of our model to gain better insights about the trained model.
Figure shows that our model can learn a variety of language and visual reasoning patterns. For example, the first two examples of the first row show that our model learns relate superlative adjectives (e.g., longer, shorter) with visual comparison. Examples include spatial prepositions (e.g., on right, on left, next to, behind, over, bottom) demonstrate the spatial reasoning ability of the model. We also that the model can learn domain-speciff nomenclature (catcher, batter) that is present in the dataset. Lastly, we can see that the model can distinguish the different actions (e.g.. standing, squatting, sitting).
of the cases that model segments correctly. These examples demonstrate
Figure shows some of the incorrect segmentation predictions from our model on the UNC vali- dation dataset. In the figure, each group shows one of the observed patterns within the examples. One them (a) is that our model tends to combine similar objects or their parts when they are hard to distinguish. Another reason for the errors is that some of the expressions are ambiguous (b), where there are multiple objects that could be the correspondence of the expression. And the model segments both possible objects. Some of the examples (d) are hard to segment completely due to the lack of light or objects that mask the referred objects. Finally, some of the annotations contain incorrect incomplete ground-truth mask (c).
8

---

Under review as a conference paper at ICLR 2021
Original Image
Ground Truth
Prediction
Original Image
Ground Truth
Prediction
guy behind right of guy with cow
women
back white shirt behind fence
coffee mug
left white chair
half filled wine glass
upper most left side apple
d)
front guy red and white shirt
Figure 3: Some incorrect predictions our model on UNC validation set. Each group (a-d) shows one pattern we observed within the predictions. In each group, the first column shows the original image, the second one is the ground truth mask and the third one is the prediction of our model.
5 CONCLUSION
We showed that modulating not only top-down but also bottom-up visual processing with language input improves the performance significantly. Our experiments showed that the proposed model achieves state-of-the-art results on 4 different benchmarks and performs significantly (*< 6 IoU) better than a baseline which uses language only to direct top-down attention. Our future work will focus using it as sub-component solve far more language-vision task like mapping natural language instructions to sequences of actions.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings the IEEE conference computer vision and pattern recognition, pp- 6077-6086, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauff Ian Reid, Stephen Gould, and Anton van den Hengel. Aision-and-languaal navigation: Interpreting visually-grounded navigation instructions in real environments. In The IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2018b.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruy Batra, C. Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Paul Bloom. How children learn the meanings of words. MIT press, 2002.

---

Under review as a conference paper at ICLR 2021
Bastien Boutonnet and Gary Lupyan. Words jump-start vision: A label advantage in object recog nition. Journal of Neuroscience, 35(25)99329 9335, 2015.
Ding-Jie Jie Chen, Songhao Jia, Yi-Chen Lo, Twann-Tzone Chen, and Tyng-Luh Liu. See-through- grouping for referring image segmentation. Proceedings of the IEEE International Conference on Computer Vision, pp. 7454-7463, 2019a.
Liang- Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on patlern analysis and machine intelligence, 40(4): 834-848, 848, 2017.
Y.-W. Chen, Y.-H. Tsai, T. Wang, Y.-Y. Lin, and M.-H. Yang. Referring expression object segmen- tation with caption-an aware consistency. In British Machine Vision Conference (BMVC), 2019b.
Volkan Cirik, Taylor Berg-Kirkpatrickk and Louis-Philippe Morency. Using syntax to ground refer- ring expressions in natural images. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Charles E Connor, Howard E Egeth, and Steven Yantis. Visual attention: bottom-up versus top- down. Current biology, (4(19):850-RRS8 R852, 2004.
Maurizio Corbetta and Gordon L. Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience, 3(3):201-215, 2002.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Banchiamlack Dessalegn and Barbara Landau. More than meets the eye: The role of language in binding and maintaining feature conjunctions. Psychological science, 19(2):189-195, 2008.
Hugo Jair Escalante, Carlos A Hernández, Jesus A Gonzalez, Aurelio López-López, Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor, and Michael Grubinger. The segmented and annotated iapr tc-12 12 benchmark. Computer vision and image understanding, 114 (4):419-422 2010.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through vidco prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Vittorio Gallese and George Lakoff. The brain's 's concepts: The role of the sensory motor system in conceptual knowledge. Cognitive neuropsychology. 22(3-4):455-475 2005.
Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang- Duestion-guidee hybrid convolution for visual question answering. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pp. 469-485, 2018.
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video seg- mentation from a sentence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5958-5966, 2018
Ross Girshick. Fast r-cnn. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi: 10. 1.111999cc..2215.11 URL 169
2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. 2014 JEEE Conference Computer Vision and Pattern Recognition, Jun 2014. doi: 10.1109/cvp:2011..81. URL http:/ / dxx doi. oaaiio.g11....
1109/CVPR 2014.87
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cmn. 2017 IEEE Inter- national Conference on Computer Vision (ICCV). Oct 2017. doi: 10.11.99/ew.2017..322. URL http://dxx doi. wtt.:oogg/1......1 ICCV. 2017.322
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expres- sions. Lecture Notes in Computer Science, pp. 108-124, 2016a. ISSN 1611-3349. doi: 10. 10.1007/ 778-3-319-46448-0.22 URL WwRLhtpp:///
3 319 44448-448888 c_
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016b.
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling rela- tionships in referential expressions with compositional modular networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1 10.1109/cvp22017.470.
URL http:://xx www.:o.......oo.
CVPR. 2017.470
Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-directional relationship inferring network for referring image segmentation. In Proceedings of the IEEE/CVF Conference
Computer Vision and Pattern Recognition, pp. 4424 4433, 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han. Linguistic structure guided context modeling for referring image segmentation. 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 500.03667 2015.
Ray Jackendoff and Ray S Jackendoff. Foundations of language: Brain, meaning, grammar, evolu- lion. Oxford University Press, USA, 2002.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Jwa-Hwa Kim, Sang- Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Zhang. Multimodal residual learning for visual qa. In Advances in neural informa- fion processing systems, pp. 361-369, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69880 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
URL atttps://arxxt wttpv/..wwr.bbbbbbb.ba/... 07332
Ruiyu Li, Kaican Li. Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi. Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5745-5753, 2018.

---

Under review as a conference paper at ICLR 2021
Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6495-6503, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chenxi Liu, Zhe L. Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan L. Yuille. Recurrent mul- timodal interaction for referring image segmentation. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1280-1289, 2017.
Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In The IEEE International Conference on Computer Viston (ICCV), October 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances in neural information processing systems, pp. 289- 297, 2016.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 375-333--3 2017.
Andrew Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlincarities improve neural net- work acoustic models. In Proc. ieml, volume 30, pp. 2013.
Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based ap- proach answering questions about images. In Proceedings the IEEE international conference on computer vision, pp. 1-9, 2015.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10.1 10.11099cxp:20166.. URL
http::/dx. doi. org/ 10.
Edgar Margffoy-Tuay Juan C Pérez, Emilio Botero, and Pablo Arbeláez. Dynamic multimodal instance segmentation guided by natural language queries. In Proceedings of the European Con- ference on Computer Vision (ECCV), pp. 630-645, 2018.
Lotte Meteyard, Bahador Bahrami, and Gabriella Vigliocco. Motion detection and motion verbs: Language affects low-level visual perception. Psychological Science, 181111:1007-011, 2007.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions actions in 3d environments with visual goal prediction. arXiv preprint 1809 2018
Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis. Modeling context between objects for referring expression understanding. Lecture Notes in Computer Science pp. 792-807, 2016. ISSN 1611-3349. doi: 10.1007/978-33311-4699999 48. URL http::/dxx..../// doi wwwp::doooooooooooo 3017/978-33319-6655
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
12

---

Under review as a conference paper at ICLR 2021
Friedemann Pulvermüller. Words in the brain's language. Behavioral and brain sciences, 22(2): 253-279, 1999.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 197 Jun 2017. ISSN 2160-9292. doi: 10.1 10.1109/pami2221..557777
URL http: www..iooi..ooooooooooooo www.:oi...ggggg. 1119/TPAM....MM 201 6. 2577031
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241- Springer, 2015.
Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-aware network for refer- ring expression image segmentation. In The European Conference on Computer Vision (ECCV), September 2018.
Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang- chun WOO. Convolutional Istm network: machine learning approach for preeip- itation nowcasting. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and
R. Garnett (eds.), Advances Neural Information Processing Systems 28, pp. 802- 810. Curran Associates, Inc., 2015. URL http: http://papers.nnt wttp:ac.ppppccc.pppppeeep.pppppe
5955 www---oooolaiinallllo -eteriinn---------tttt--------- oat.aaae.......aatoooo-oa-aaaaaa www.aha----eaannngaaaar-------e- for
precipitation-nox
pdf
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15()):199--1988 2014.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pp. 217-223, Vancouver, Canada, July 2017. Association for Computational Lingustttssi URL https: htt:::/wwwwww. wwwp:w.wwwebb.o.. www.weeww.............eeeeeeeeee
Jan Theeuwes. Top-down and bottom-up control of visual selection. Acta psychologica, 135(2): 77-99, 2010.
Gabriella Vigliocco, David P Vinson, William Lewis, and Merrill F Garrett. Representing the mean- ings of object and action words: The featural and unitary semantic space hypothesis. Cognitive psychology, 48(4):422-488, 2004.
Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, and Anton van den Hengel. Neighbour- hood watch: Referring expression comprehension via language-guided guided graph attention networks. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10. 10.1109/cvpr:2119.0000 9.00006. BRLhhtp::// //xx
110911111///100 2019. 00200
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial atten- tion for visual question answering. In European Conference on Computer Vision, pp. 451-466. Springer, 2016.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-2057, 2015.
Zichao Yang. Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21-29, 2016.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cpp:20199.00775 URL http::tpp//// //tp:///xx doi.org/ 10. 1109/VVR. 2019.01075
13

---

Under review as a conference paper at ICLR 2021
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. Lecture Notes in Computer Science, pp. 09-85,22 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. Mattnet: Modular attention network for referring expression comprehension. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1 00142.
doi. dow. 10. 1ww1111191V//. 2018 00142
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720-6731, 2019.
14

---

Under review as a conference paper at ICLR 2021
ground truth image
UNK
man
man wearing
man wearing with black
man wearing black and red color
Figure 4: Incremental segmentation result of our model on UNC test split instance.
A APPENDIX
A. INCREMENTAL SEGMENTATION
We also analyzed the behaviour of our model with respect incrementally given language input in Figure 4 In the imitial step, our model only sees unknown word token. In the second step, our model sees only the first word of the expression. every step, our model starts see a new word in addition to the previous ones. TigureAshone that our model can capture ambiguities in input image and expressions pairs. For unknown token input, our model captures all salient objects since there no restriction. When the word is fed, the model discards unrelated objects like umbrella and wheel. Additionally, when our model starts to see color words for coat, it initially focuses on both men, since both coats has black color. When it sees the final expression, it shifts its focus to the correct object.
15


Reviews:
[{"review_id": "vr4cjmbkui", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to integrate visual and linguistic features in both top-down and bottom-up modulation of the visual input. This is done by fusing two modalities while doing convolution and deconvolution operations over the visual input. Experiments on image segmentation from referring expressions in standard datasets show that the proposed approach achieves state-of-the-art or competitive results. Ablation studies show that both top-down and bottom-up is essential. I believe the novelty and contribution are rather thin because many ways of the modeling language are not explored at all. \n\nBelow I list suggestions (S) and questions (q) for authors:\n\nS1 Second paragraph of introductions: please add a figure to explain the concepts of top-down, bottom-up processing, high-level, low-level effects etc.\n\nS2 Section 2.2.: Please cite the below papers [1,2] for referring expression comprehension. For Section 2.4 please add [3]\n\nS3 Figure1: following this figure is not intuitive. I recommend adding two arrows for top-down and bottom-up processing and adding more space between two branches. \n\nS4 Section 4.2: It is not clear how each of these ablations was performed. For instance, I'm not 100% sure whether two modalities are fused at different levels of top-down or bottom-up processing.\n\n[1] Nagaraja et. al. \"Modeling context between objects for referring expression understanding.\" \n[2] Cirik et. al \"Using syntax to ground referring expressions in natural images.\"\n[3] Chen et. al. \"Touchdown: Natural language navigation and spatial reasoning in visual street environments.\" \n\nS4: Section 4.3: the claim of modeling the long-range dependencies is a bit speculative. I would rephrase that.\n\nS5: Figure2: Failure cases are more informative than successful ones. Please either bring the figure from A.1 or add a comparison with a model from the literature where the other model is successful where yours is not to do a contrastive analysis on how your model can be improved.\n\nQ1: Section3: What's the effect of the number of layers for the model? Why stop at 3? Do you have results for the number of layers 0,1,2?\n\nQ2: Is there a way to interpret the interaction between language and visual input?\n\nQ3: Have you experimented with different ways of fusing or processing language input? Examples: gating the language representation, attention over tokens, using different fusion methods, bi-directional LSTM, BERT-like contextual representations, adding inductive bias with parse tree for referring expressions, alignment between feature maps and word tokens or phrases?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bxS1-55Cv35", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model for image segmentation from referring expressions which integrates linguistic representations of the referring expressions both at low-level and high-level stages of visual processing. They argue that this model is both more cognitively plausible and more successful than models which only use linguistic representations to modulate attention over high-level visual features.  \n\nI vote for rejection, mainly on grounds of significance and quality, expanded below. To change my vote I would require a substantial improvement on one or both of the quality/significance issues listed below, either presenting the model with a clear conceptual motivation, or doing satisfactory model analysis to understand the contribution of the paper.  \n\nPros: The presented model shows some moderate quantitative improvement over other recent work.\n\nCons: Poor conceptual motivation and error analysis, with little evident understanding of the actual effect of the linguistic representations within the model.  \n\nQuality/Significance\n\n1. The paper does not provide a clear motivation for their model. Here are some arguments I looked for but did not find:\n  1. Cognitively: There are some references to relevant cognitive science papers, but there seems to be little concrete inspiration taken from this or other cognitive work in the particular model design.\n  2. A priori based on the task: What would we expect to gain from using language in early-stage visual representations? What sort of correlations might exist between particular types of linguistic input and low-level visual representations? This might be another way to motivate the model, but I can't find any such discussion in the introduction or anywhere else.\n2. The evaluations don't convince me that this paper has made a significant conceptual contribution.\n  1. The quantitative results don't seem to constitute an enormous improvement over past work. The variability in table 2 across evaluation sets makes me doubt the statistical significance of the claims. No statistical significance tests (across resamples of test data or across random training restarts) are provided.\n  2. There is no satisfactory analysis of the actual cause of the model's success. What are the contents of the linguistic representations, and how exactly do they modulate low-level visual features? For reference, Hu et al. (2020, Figure 4) [1] and Hui et al. (2020, Figure 5) [2] both do some of what I'm looking for here, showing the influence of language on the behavior of the model. While the more complex representations used in this model make it more difficult to provide e.g. an easy heatmap, we absolutely need to see an error analysis that helps us believe your claim that language ought to play a role in low-level visual processing.  \n\nOriginality\n\nI don't closely follow the relevant literature and can't speak confidently on the originality of the model. I did have trouble understanding the innovation over Step-ConvRNN, however -- these models seemed within tweaking-distance of one another based on the presentation in this paper.  \n\n[1]: https://openaccess.thecvf.com/content_CVPR_2020/papers/Hu_Bi-Directional_Relationship_Inferring_Network_for_Referring_Image_Segmentation_CVPR_2020_paper.pdf#page=5\n[2]: https://arxiv.org/pdf/2010.00515.pdf#page=14\n\n## Post-rebuttal update\n\nI have read the other reviews and the authors' rebuttals, and do not wish to change my review.\n\nI strongly believe that numerical task improvements are not in themselves a conceptual contribution. I look forward to the results of the analyses the authors mention in response to R3-Q2, to better understand what exact interaction between language and low-level visual input is being modeled.\n\nAlong with R4 I remain unconvinced of the strength of the empirical results. The authors' response is not helpful here. I can't understand where the numbers (mean 60.74 IoU, std 0.06) come from -- taking stats across table 2 and table 1, I get very different results, so I must be misunderstanding where they come from.\n\nSignificance tests would not take too much time -- it's not absolutely critical that you retrain the models for this. You can use data resampling methods instead. For example, on each individual dataset, run bootstrap tests comparing the predictions of your model and others on random resamples of the evaluation data and corresponding predictions.\nPooling IoU results across datasets within model and then comparing between models can yield misleading results and should be avoided. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YpGeltReJ7D", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper concerns the problem of image segmentation from referring expressions. Given an image and a query phrase about a particular object in the image, the goal is to locate the target object as a mask at the pixel level. The basic framework is U-Net, which consists of two branches: an image encoder and segmentation map decoder (connected at the bottom in a U-shape). The paper proposes to use language to modulate the image encoding and decoding process intensively, by applying auxiliary convolutional connections between the two branches and further condition the convolution kernel on the language embedding. Overall, the paper is easy to follow and has done a good literature review.\n\nThe major concern of the paper lies in the empirical results. Despite that the introduction of top-down and bottom-up language modulation significantly boost the baseline performance (Tab. 1), the full model struggles to match existing works on certain metrics such as UNC testA/testB, UNC+ testB, and ReferIt, which put a question mark on the effectiveness of the work. The results on the validation set are promising but not as good on the test set, which indicates a possible over-tuning of the model.\n\nA minor comment on the model part. In the text above Eq. 1, the paper mentions \"[...] ,we split the textual representation [...]\". However, what is the rationale for splitting the representation since each split does not attach to any particular abstract of the image feature (low-level, mid-level, and high-level)?\n\nBesides, some numbers from Tab. 1 do not match those from Tab. 2. For instance, the IoU on LSCM and Step-ConvRNN. Please double check.\n\n============== Post-Rebuttal ==============\n\nThe authors' responses to point 1 & 2 do not sound (reflecting a question to another paper does not solve the problem). The authors mentioned \"We made this decision based on Mei et al (2018) which proposed our baseline model (the top down approach)\", where the reference of Mei et al (2018) cannot be found in the paper, as a critical baseline. This raises a flag on the novelty of the work and completeness of the related work. Therefore, I am lowering my rating to 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CiQc9XzOPRR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article proposes a novel approach integrating language throughout the visual pathway for segmenting objects according to referring expressions. \n\nThe article is well written, and poses an important question about how best to integrate linguistic and visual information. The limitations of the currently dominant top-down approach are well argued. The answer proposed by the authors is to integrate linguistic information throughout the visual hierarchy. The task of segmenting by referring expression is important and well chosen. \n\nThe proposed model is sound, and well described in the article, and the experimental results demonstrate that the model outperforms clearly the state-of-the art in all metrics. The qualitative examples provided are quite impressive and demonstrate the success of the approach. \n\nIn sum, I feel this is a well written paper addressing a very timely and important problem in computer vision and AI research and should be of broad interest in the community. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to improve image segmentation from referring expression  by integrating visual and language features using an UNet architecture and experimenting with top-down, bottom-up, and combined (dual) modulation.  

Review Summary: The submission received divergent reviews with scores spanning from 2 (R2) to 5 (R3,R4) to 10 (R1).  The author response failed to address the reviewer concerns with some reviewer (R4) lowering their score tto 4 after the rebuttal.  It also became clear that some relevant work (Mei et al, 2018) was used for the baseline but not cited.  The author response also did not recognize the importance of significance tests.

As there is considerable work in the area of image segmentation from referring expression, and the proposed model is very similar to the LingUNet model of Misra 2018, the originality and significance of the work is fairly low.  The main contributions appears to be experimental comparisons of the three types of modulation (top-down, bottom-up, dual).

Pros:
- Investigation of a important problem of grounding language to visual regions
- Experimental study of whether dual modulation improves image segmentation from referring expression

Cons
- Relatively minor novelty with limited analyses (R3,R2)
- Missing citations (see R3's comments).  Relevant work (Mei et al, 2018) which was the basis for the top-down baseline model, was used but not cited or properly compared against
- Relatively weak experimental results (R2,R4).  As R4 noted, while validation results are good, test results are weak compared to existing work, indicating potential overtuning.  
- No qualitative comparisons against baselines.  
- Cognitive claims not backed up and limited discussion/analysis (R2)

Recommendation:
The AC concurs with R2, R3, and R4 that the work is limited in novelty and not ready for publication at ICLR.   Despite R1's high score, referring expression for image segmentation is a well studied task, and it is unclear what are the key innovations of the proposed model over LingUNet.  Due to the limited novelty, relatively weak test results, as well as other flaws pointed out by the reviewers, the AC recommends rejection.

Decision: Reject