Language Controls More Than Top-Down Attention: Modulating Bottom-Up Visual Processing with Referring Expressions

1 INTRODUCTION . As human beings , we can easily understand the surrounding environment with our visual system and interact with each other using language . Since the work of Winograd ( 1972 ) , developing a system that understands human language in a situated environment is one of the long-standing goals of artificial intelligence . Recent successes of deep learning studies in both language and vision domains have increased the interest in tasks that combine language and vision ( Antol et al. , 2015 ; Xu et al. , 2015 ; Krishna et al. , 2016 ; Suhr et al. , 2017 ; Anderson et al. , 2018b ; Hudson & Manning , 2019 ) . However , how to best integrate linguistic and perceptual processing is still an important open problem . In this work we investigate whether language should be used to control the filters for bottom-up visual processing as well as top-down attention . In the human visual system , attention is driven by both “ top-down ” cognitive processes ( e.g . focusing on target ’ s color or location ) and “ bottom-up ” salient , behaviourally relevant stimuli ( e.g . fast moving objects ) ( Corbetta & Shulman , 2002 ; Connor et al. , 2004 ; Theeuwes , 2010 ) . Studies on embodied language explore the link between linguistic and perceptual representations ( Pulvermüller , 1999 ; Vigliocco et al. , 2004 ; Gallese & Lakoff , 2005 ) and it is often assumed that language has a high-level effect on perception and drives the “ top-down ” visual attention ( Bloom , 2002 ; Jackendoff & Jackendoff , 2002 ; Dessalegn & Landau , 2008 ) . However , recent studies from cognitive science point out that language comprehension also affects low-level visual processing ( Meteyard et al. , 2007 ; Boutonnet & Lupyan , 2015 ) . Motivated by this , we propose a model1 that can modulate either or both of “ bottom-up ” and “ top-down ” visual processing with language conditional filters . Current deep learning systems for language-vision tasks typically start with low-level image processing that is not conditioned on language , then connect the language representation with high level visual features to control the visual focus . To integrate both modalities , concatenation ( Malinowski et al. , 2015 ) , element-wise multiplication ( Malinowski et al. , 2015 ; Lu et al. , 2016 ; Kim et al. , 2016 ) or attention from language to vision ( Xu et al. , 2015 ; Xu & Saenko , 2016 ; Yang et al. , 2016 ; Lu et al. , 2017 ; Anderson et al. , 2018a ; Zellers et al. , 2019 ) may be used . Specifically they do not condition low-level visual features on language . One exception is De Vries et al . ( 2017 ) which proposes conditioning the ResNet ( He et al. , 2016 ) image processing network with language conditioned batch normalization parameters at every stage . Our model differs from these architectures by having explicit “ bottom-up ” and “ top-down ” branches and allowing us to experiment with modulating one or both branches with language generated kernels . 1We will release our code and pre-trained models along with a reproducible environment after the blind review process . We evaluate our proposed model on the task of image segmentation from referring expressions where given an image and a natural language description , the model returns a segmentation mask that marks the object ( s ) described . We can contrast this with purely image based object detection ( Girshick , 2015 ; Ren et al. , 2017 ) and semantic segmentation ( Long et al. , 2015 ; Ronneberger et al. , 2015 ; Chen et al. , 2017 ) tasks which are limited to predefined semantic classes . Our task gives users more flexibility to interact with the system by allowing them to describe objects of interest in free form language . The language input may contain various visual attributes ( e.g. , color , shape ) , spatial information ( e.g. , “ on the right ” , “ in front of ” ) , actions ( e.g. , “ running ” , “ sitting ” ) and interactions/relations between different objects ( e.g. , “ arm of the chair that the cat is sitting in ” ) . This makes the task both more challenging and suitable for comparing different strategies of language control . The perceptual module of our model is based on the U-Net image segmentation architecture ( Ronneberger et al. , 2015 ) . This architecture has clearly separated bottom-up and top-down branches which allows us to easily vary what parts are conditioned on language . The bottom-up branch starts from low level visual features and applies a sequence of contracting filters that result in successively higher level feature maps with lower spatial resolution . Following this is a top-down branch which takes the final low resolution feature map and applies a sequence of expanding filters that eventually result in a segmentation mask at the original image resolution . Information flows between branches through skip connections between contracting and expanding filters at the same level . We experiment with conditioning one or both of these branches with language . To make visual processing conditional on language , we add language-conditional filters at each level of the architecture , similar to Misra et al . ( 2018 ) . Our baseline only applies languageconditional filters on the top-down branch . Modulating only the top-down/expanding branch with language means the high level features extracted by the bottom-up/contracting branch can not be language-conditional . Our model expands on this baseline by modulating both branches with language-conditional filters . Empirically , we find that adding language modulation to the bottomup/contracting branch has a significant positive improvement on the baseline model . Our proposed model achieves state-of-the art performance on three different English referring expression datasets . 2 RELATED WORK . In this section , we review related work in several related areas : Semantic segmentation classifies the object category of each pixel in an image without language input . Referring expression comprehension locates a bounding box for the object ( s ) described in the language input . Image segmentation from referring expressions generates a segmentation mask for the object ( s ) described in the language input . We also cover work on language-conditional ( dynamic ) filters and studies that use them to modulate deep-learning models with language . 2.1 SEMANTIC SEGMENTATION . Primitive semantic segmentation models are based on Fully Convolutional Networks ( FCN ) ( Long et al. , 2015 ) . DeepLab ( Chen et al. , 2017 ) and U-Net ( Ronneberger et al. , 2015 ) are the most notable state-of-the-art semantic segmentation models related to our work . DeepLab replaces regular convolutions with atrous ( dilated ) convolutions in the last residual block of ResNets ( He et al. , 2016 ) and implements Atrous Spatial Pyramid Pooling ( ASPP ) which fuses multi-scale visual information . The U-Net architecture ( Ronneberger et al. , 2015 ) improves over the standard FCN by connecting contracting ( bottom-up ) and expanding ( top-down ) paths at the same resolution : the output of the encoder layer at each level is passed to the decoder at the same level . 2.2 REFERRING EXPRESSION COMPREHENSION . Early models for this task were typically built using a hybrid LSTM-CNN architecture ( Hu et al. , 2016b ; Mao et al. , 2016 ) . Newer models ( Hu et al. , 2017 ; Yu et al. , 2016 ; 2018 ; Wang et al. , 2019 ) use an Region-based CNN ( R-CNN ) variant ( Girshick et al. , 2014 ; Ren et al. , 2017 ; He et al. , 2017 ) as a sub-component to generate object proposals . Nagaraja et al . ( 2016 ) proposes a solution based on multiple instance learning . Cirik et al . ( 2018 ) implements a model based on Neural Module Networks ( NMN ) by using syntax information . Among the literature , Compositional Modular Network ( CMN ) ( Hu et al. , 2017 ) , Modular Attention Network ( MAttNet ) ( Yu et al. , 2018 ) and Neural Module Tree Networks ( NMTree ) ( Liu et al. , 2019 ) are the most notable state-of-the-art methods , and all of them are based on NMN ( Andreas et al. , 2016 ) . 2.3 IMAGE SEGMENTATION FROM REFERRING EXPRESSIONS . Notable models for this task include Recurrent Multimodal Interaction ( RMI ) model ( Liu et al. , 2017 ) , Recurrent Refinement Networks ( RRN ) ( Li et al. , 2018 ) , Dynamic Multimodal Network ( DMN ) ( Margffoy-Tuay et al. , 2018 ) , Convolutional RNN with See-through-Text Embedding Pixelwise heatmaps ( Step-ConvRNN or ConvRNN-STEM ) ( Chen et al. , 2019a ) , Caption-aware Consistent Segmentation Model ( CAC ) ( Chen et al. , 2019b ) , Bi-directional Relationship Inferring Network ( BRINet ) Hu et al . ( 2020 ) and Linguistic Structure guided Context Modelling ( LSCM ) module Hui et al . ( 2020 ) . RRN which has a structure similar to U-Net , is built on top of a Convolutional LSTM ( ConvLSTM ) ( SHI et al. , 2015 ) network . Unlike our model , ConvLSTM filters are not generated from language representation and the multi-modal representation is used only in the initial time step . DMN generates 1 x 1 language-conditional filters for language representation of each word . It performs convolution operation on visual representation with language-conditional filters to generate multi-modal representation for each word . Like RMI , word-level multi-modal representations are fed as input to a multi-modal RRN to obtain multi-modal representation for image/language pairs . Step-ConvRNN starts with a visual-textual co-embedding and uses a ConvRNN to iteratively refine a heatmap for image segmentation . Step-ConvRNN uses a bottom-up and top-down approach similar to this work , however , our model uses spatial language generated kernels within a simpler architecture . CAC also generates 1 x 1 language-conditional dynamic filters . Unlike our model , CAC applies these dynamic filters to single resolution / single feature map and additionally generates location-specific dynamic filters ( e.g . left , bottom ) to capture relations between the objects exist at the different parts of the image . BRINet implements two different attention mechanisms : language-guided visual attention and vision-guided linguistic attention . LSCM implements a dependency parsing guided bottom-up attention mechanism to predict masks . 2.4 LANGUAGE-CONDITIONAL FILTERS . To control a deep learning model with language , early work such as Modulated ResNet ( MODERN ) ( De Vries et al. , 2017 ) and Feature-wise Linear Modulation ( FiLM ) ( Perez et al. , 2018 ) used conditional batch normalization layers with only language-conditioned coefficients rather than customized filters . Finn et al . ( 2016 ) generates action-conditioned dynamic filters . Li et al . ( 2017 ) is the first work which generates dynamic language-conditional filters . Gao et al . ( 2018 ) proposes a VQA solution method which has a group convolutional layer whose filters are generated from the question input . Gavrilyuk et al . ( 2018 ) introduces a new task called as actor and action segmentation and to solve this task , proposes an architecture which uses dynamic filters for multiple resolutions . Similar to our work , Misra et al . ( 2018 ) adds language conditional filters to a U-Net based architecture for the task of mapping instructions to actions in virtual environments. ? also uses an architecture based on U-Net and Misra et al . ( 2018 ) to solve a navigation and spatial reasoning problem . Those models only modulate top-down visual processing with language . Referring expression models that incorporate language-conditional filters into the architecture include ( Chen et al. , 2019b ; Margffoy-Tuay et al. , 2018 ) . Margffoy-Tuay et al . ( 2018 ) generates language-conditional filters for words individually rather than whole sentence . Chen et al . ( 2019b ) generates 1 x 1 language-conditional filters from expressions . To make 1 x 1 language-conditional filters spatially aware , different filters are generated for different image regions ( e.g . top , left , right , bottom ) . Our main contribution in this work is an explicit evaluation of language conditional filters for bottom-up visual processing in comparison to only using language for top-down attention control .

Summary: This paper concerns the problem of image segmentation from referring expressions. Given an image and a query phrase about a particular object in the image, the goal is to locate the target object as a mask at the pixel level. The basic framework is U-Net, which consists of two branches: an image encoder and segmentation map decoder (connected at the bottom in a U-shape). The paper proposes to use language to modulate the image encoding and decoding process intensively, by applying auxiliary convolutional connections between the two branches and further condition the convolution kernel on the language embedding. Overall, the paper is easy to follow and has done a good literature review.