A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
Authors: Noriyuki Kojima
Hadar Averbuch-Elor
Yoav Artzi
Venue: Accepted by TMLR ()

Abstract: Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models is complex, even if it is generally expected to take place if the task is addressed in a way that is conductive to generalization. We propose a framework to jointly study task performance and phrase grounding, and propose three benchmarks to study the relation between the two. Our results show that contemporary models demonstrate inconsistency between their ability to ground phrases and solve tasks. We show how this can be addressed through brute-force training on ground phrasing annotations, and analyze the dynamics it creates. Code and data are available at https://github.com/lil-lab/phrase_grounding.

Published in Transactions on Machine Learning Research (01/2024)
A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
Noriyuki Kojima
nkojima@kotobb..com
Deportment of Computer Science and Cornell Tech, Cornell University Kotoba Technologies, Inc.
Hadar Averbuch-Elor
School Electrical Engineering, Tel-Aviv University
Yoav Artzi
youv@cs.cornell.ct
Department of Computer Science and Cornell Tech, Cornell University
Reviewed on OpenReview: /ttps://openweee opw.reveee.r.......om net/ forum? 5G3PI1NEdw
Abstract
Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models complex, even if it is generally expected to take place if the task is addressed in a way that is conductive to generalization. We propose framework to jointly study task performance and phrase grounding, and propose three benchmarks to study the relation between the two. Our results show that contemporary models demonstrate inconsistency between their ability to ground phrases and solve lasks. We show how this can be addressed through brute-force training on ground phrasing annotations, and analyze the dynamics it creates. Code and data are available at
wtwp:/hh....mmlllaaaaaaaaaaaaaaa http://hhaaeeeeeeeee http-//laabbpaarssgoroonode
1 Introduction
Key to reasoning about natural language in visual contexts is grounding (i.c., resolving) words and phrases to image regions. For example, consider reasoning about the spatial description in Figure I where the task is to locate a hidden teddybear named Touchdown (Chen al. 2019a). The compositional description outlines a reasoning process that includes creating correspondences between the language (e.g., light- colored building, arched doorways, etc.) and image regions. To correctly interpret the language for such lask, the model is expected to follow such reasoning process, which is core to almost all vision and language tasks, including visual question answering (Antol et al. 2015 Goyal et al. 2017b Suhr et al, 2017 2019), caption generation [Kiros et al., 2014 Xu et al. 2015 resolution and generation of referring expressions [Kazemzadeh et al. 2014 Mao et al. 2016a), and vision and language navigation (Misra et al, However, existing work shows that high-performance models do not necessarily follow the expected reasoning process, often instead relying on reasoning shortcuts
2017 Anderson et al.] 2018b; Blukis et al.
2018
that hide their limitations Agrawal et al. 2017 Cirik al. 2018 Kojima et al. 2020 inter alia).
In this paper, we propose to examine the reasoning process of vision and language models by focusing on their phrase grounding abilities. Our goal is to characterize how well models that are trained to solve language and vision tasks associate phrases from their input language to regions in the input image, and how well their ability to create such associations for a specific input correspond to their task success.
Ideally, models that can successfully complete the task they are trained for, should also be able ground phrases from the text (i.e,, align them to regions in the input image), and failures in grounding phrases

---

Published in Transactions on Machine Learning Research (01/2024)
light-colored building with he several red doors and arched doorways There is a American flag]
bydn the last door Louchdown is sitting on the stars of the flag
We jointly study the task performance and phrase grounding of the vision and language models. Above, we illustrate our approach in one of our benchmarks: TOUCHDOWN SDR. Top Left: The task is to locate the hidden Touchdown in urban panorama using text description. Top Right: The TOUCHDOWN SDR dataset is expanded with 167k manual annotations of bounding boxes to support the study. Bottom: ViLT simultaneously locates Touchdown and demonstrates reasoning processes through phrase grounding.
Figure 1:
should reflect in the overall task performance, showing task reasoning corresponds to correctly resolving phrases. For example, in the spatial description resolution (SDR) task in Figure 1 successfully identifying the location of the teddybear described by the input language should correspond to successfully grounding the phrases in the text (e.g,, arched doorways, an american flag). We propose quantify this by correlating task and phrase grounding performance.
This requires test data annotations of both task and phrase grounding labels. We create three benchmarking resources, by extending and re-purposing existing datasets, focusing on two tasks. The first task is spatial description resolution (SDR; Chen et al. 2019a), where the goal is to locate hidden object following a language description. We use the TOUCHDOWN benchmark Chen et al.] (2019a), and extend by annotating the correspondences between phrases in the imput language and regions in the image context, providing 167k manual bounding box annotations. We also study reference games Clark & Wilkes-Gibe Gibbs (1986), where the objective to identify a referent out of a set given its description. We construct reference games using two existing datasets: Flickr30k Entities (Plummer et al. 2015 and KILOGRAM (Ji et al. 2022).
We experiment with two modeling approaches that represent broader trends in model development: ViLT (Kim et al. 2021) and MDETR (Kamath et al. 2021). We use our models to jointly address the target tasks and output phrase groundings. Despite relatively strong task performance, our extensive ex- periments demonstrate inconsistencies between task success and the ability of the models to ground input phrascs to regions in the input image. We further conduct probing experiments the internal activations of the models, confirming their limited ability to resolve phrases correctly. While ideally such grounding behavior would arise naturally from task training, we show how augmenting the training data with phrase information can alleviate much of this problem. This indicates that the source of the problem we observe is the learning process, rather than the model architectures. Furthermore, in most cases, we observe that relatively small amount of phrase grounding annotations closes much of the gap and dramatically improves the correlation of task reasoning and phrase grounding.
provide deeper look into
Code, data, and models are available at: http:://gtttttoooo///a//aaaaaaaa
2 Related Work
Vision and Language Reasoning Models Vision and language reasoning widely studied through tasks like visual question answering, spatial reasoning, referring expression resolution, instruction following,
among others (Antol et al. 2015 Goyal et al. 2017b Chen et al.. 2019a Cirik et al. 2020 Mao al. 2016a
2

---

Published in Transactions on Machine Learning Research (01/2024)
Anderson et al.)
2018b
Misra et al.
2017
Hawkins et al. 2020 Ji al.) 2022 Alper et al. 2023 Alper &
Evrrbuch-Flor 2023 inter alia). Many models are designed and specialized for each task ( (Anderson et al.
2018a
Misra et al. 2017 Shi et al. 2019 inter alia). More recently, focus has shifted to vision and language
pre- with transformers to enable a wide range of tasks to be performed with a single model (Lu et al.
2019
Tan & Bansal] 2019; Li et al. 2019 Chen et al., 2020; Zhou et all; 2020a] [Lu et al. 2020: Li et al,
2020
Fichenberg et al. 2021 Kim et al., 2021 Alayrac et al. 2022. inter alia). Both with specialized and
more recent general models, reasoning procedures are often opaque, and they, generally, have been shown to
frequently shorteut reasoning steps and exhibit undesirable behaviors like failing to generalize relying
spurious correlations (Agrawal et al. 2017 Cirik et al. 2018 Jain et al. 2019 Agrawal et al. 2016 Goyal
et al. 2017a Kojima et al. 2020]. The impact of such issues goes beyond benchmarking, because they
illustrate deficiencies in models' abilities to acquire the expected reasoning process and generalize properly.
Phrase Grounding The process of associating text phrases with corresponding image regions is known as phrase grounding and is critical to numerous vision and language tasks (Chen et al.) 2017 Gupta et al, 2020j. Several resources have been proposed to study this process, including datasots annotating various phrase types such as Flickr30K Entities (Plummer et al.) 2015) and Visual Genome (Krishna el al. 2016) and datasets focusing on specific objects, such as people in Who's Waldo (Cui et al.] 2021). This process is also studied as reference expression resolution, through resources like ReferltGame (Kazemzadeh et al. 2014 Mao et al., 2016b) and PhraseCut (Wu et al.]
2020)
Although numerous models have been proposed
Kim el. al. 2018 Engilberge el. al. 2018 Deng 2021 Li et al. 2021). connecting them to tasks that require comprehensive understanding of text remains challenging. This issue stems from the fact that phrase grounding resources often lack task annotations beyond phrase grounding itself, and thus not suitable for a joint study. We design our data and methods to tightly connect phrase grounding and end tasks.
for purase grounding (Hu et al.; 2016 Wang et al.
2016
et al. 2018 Yu et al. 2018 Liu et al. 2019a Ding et al.
Bridging Tasks and Phrase Grounding Several past studies explore the connection between phrase grounding and vision and language tasks. Early work in visual question answering focuses on attention mechanisms,, using resources such as VQA-HAT (Das et al., (017] and VQS (Gan et al., 2017] to guide model attention through numan-annotan 2019). However, these works primarily aim to direct the model's attention to image regions useful for question answering, rather than accurately ground phrases. Attention-focusel techniques were also studied as task priors {Le et al. 2023)). ). More recently, several models are pre-trained on large-scale phrase grounding
masks (Qiao et al., 2018 Zhang et al.) 2019 Selvaraju et all,
annotations for downstream vision and language tasks (Li et al. 2022 Zhang et al. 2022 Dou et al.] 2022; Yang et al. 2022). We experiment with one such model, MDETR (Kamath et al. 2021 J. Kamath et al.
(2023) investigates phrase grounding in landem with an image-lask classification lask, aiming to reveal the blind spots of recognition models. Phrase grounding was also studied in the context of captioning
(Pont-Tuuse) navigation Ku ct al. 2020 Our work distinguishes itself from previous work by drawing stronger connections between phrase grounding and vision and language tasks, creating dedicated resources to study the problem, quantifying the correlation between the two, and investigating how to make the ability to ground phrases and strong correlations emerge in models by training variation of models.
ct al. 2020 Zhou et al. 2020b) and vision-land-langge
3
Task Performance and Phrase Grounding Correspondence
Our aim to gange and study the alignment between task success and phrase grounding. We focus on tasks that require various language-conditioned visual reasoning abilities, such as recognition and spatial reasoning. For example, the TOUCHDOWN SDR task (Fiuure 1 requires resolving the sentence given an image to identify a location in the image (i.c., where the bear Touchdown is hidden). Formally, an annotated example of such a task includes an input text I, an input image 1, and a ground-truth task annotation y (e.g., pixel coordinates of Touchdown).
We expect that correctly solving an example of this type of task includes grounding the different phrases the text to regions in the image (e.g., of objects), and that incorrect resolution of such elements will likely lead to failure on the overall task itself. Success on the task together with this type of correspondence
3

---

Published in Transactions on Machine Learning Research (01/2024)
Task
Grounding
Dataset
Task
# Images
Image
Size
Texts
Input Length
# Phrases
# Boxes
TOUCHDOWN
SDR
25,391
800 x 3712
9,308
137.47
145,839
167,979
KILLGRRAM
RG
9,432
200 200
9,432
75.95
34,809
65,799
Flickr30k Entities
RG
31,780
384 384
158,121
64.30
448,806
275,775
A summary of dataset statistios. SDR stands for the spatial descriptive resolution task and RG stands for reference games, Input length is measured as the mean number of characters in the captions.
Table 1:
would mean that the model demonstrates the type of the reasoning over text and images as expected from the inputs. This is not only the kind of reasoning we expect models to display, but is also related to their ability to generalize well, indicating the model does not take reasoning shortcuts through unexpected artifacts and
biases in the data (Agrawal et al. 2017; Cirik et al.] 2018 Jain al., 2019: Agrawal et al. 2016 Goyal ett al. 2017a Kojima et al. 2020). Poor correspondence between task success and phrase grounding would indicate the model does not acquire the reasoning the benchmark aims to study, even if task performance itself is relatively high.
Formally, the text description ã contains phrases (pI....,PMl PMr), each phrase p is a sub-sequence tokens (Zi,..., (.y....;t)) from the lext 5 = (xi,,..., TT). Grounding a phrase p requires mapping it an image region r corresponding to it. For example, Figure shows phrase grounding annotations (top right) and predictions (numbered images). (Section 4 describes three benchmarks with both task- and phrasc-level annotations.
The models study map the inpuls T and I to the output i, and are concurrently queried for mapping phrases p to their corresponding image regions A. We evaluate the models through task-specific performance measures, phrase grounding, and the correlation between the two. Phrase grounding evaluation is donc by comparing the predicted image region to the ground-truth image region r for each phrase. We calculate mean-IoU (Everingham et al. 2015) and re by calculating overlaps of and The correspondence between task performance and phrase grounding is the dataset-level correlation between task performance and phrase grounding. This quantifies the consistency of model's success or failure across these metrics calculated for each example. We use the Pearson correlation coefficient. (Benesty et al. 2009) for tasks with continuous predictions (e.g., distance to gold location in TOUCHDOWN SDR) and point biserial correlation coefficien Gupta 1960 when the task output is discrete.
4 Benchmarks with Phrase Grounding
We use three benchmarks in our study by expanding re-purposing existing datasets. Table summarizes the basic statistics of three datasets. The datasets represent a diversity of visual stimuli: street panoramas in TOUCHDOWN SDR (Section 4.1], abstract shapes (Section 4.2 and entity-focuser photos (eccion 4.3)
4.1 Touchdown SDR
TOUCHDOWN (Chen et al. 2019a is large collection of urban visual scenes captured in Street panora- mas of New York City. The images are relatively complex, capturing broad views of cluttered street scenes. TOUCHDOWN provides annotations for navigation and spatial descriptive resolution (SDR) tasks. We focus on the SDR task, and collect additional phrase grounding annotations. The data includes 9,326 unique natural language descriptions for finding a hidden teddy bear in the image. Each instruction is paired with up to three panoramas, and 25,575 unique panoramas are annotated with the pixel coordinates of the teddy bear. We preprocess the spherical panorama images into perspective images and remove images where the pixel coordinates are not within the frame, keeping 25,391 images and 9,308 descriptions.
KK stand for top-k, if model makes multiple predictions ranked by the confidence.
We average mean-TaU across all phrases in each example. Point biserial correlation is metric used calculate the correlation between continuous discrete metrics.
4

---

Published in Transactions on Machine Learning Research (01/2024)
Spatial Description Resolution The SDR task is to find the pixel coordinates of hidden teddy bear following natural language description. The instructions outline the reasoning steps necessary to find a teddy bear by providing compositional references to objects in the image. Prior qualitative analysis shows that each SDR instruction is cstimated to contain over three references to unique entities and over three spatial relations on average (Chen el al.) 2019a, Table 3). Such characteristios make TOUCHDOWN SDR an ideal testbed to study the relations of the model's 's task and phrase grounding performance. SDR is evaluated via accuracy, consistency, and pixel distance error. A prediction considered accurate if the coordinates are within a 40-pixel slack radius of the ground-truth coordinates. Consistency is evaluated similarly, but a unique text description is considered correct only if all the examples it appears in are correct. We follow the train/ http://veelloen///////e/ /est splits of Chen et al. (2019a) and use the TOUCHDOWN images that were scrubbed
of personal identifiable information (PII) by Mehla et al. (2220 T
Crowdsourcing Phrase Grounding Annotations We manually annotate 145k phrases in the instruc- tions with 167k bounding boxes on the images. Phrases are extracted from the instructions using spaCy2 noun chunker (Honnibal & Montani 2017). Qualitatively, we observe that the extracted phrases often con- tain references to visual entities (c.g., the building)) spatial regions (c.g., the top of the door), and pronouns referring to visual entities (e.g., it). We implement the annolation task on Amazon Mechanical Turk. Given an image, the complete text description, and a set of extracted phrases, workers are asked to draw bounding boxes on all the image regions referred to by each phrase or draw no bounding boxes if the phrase cannot be grounded. 121 workers participated in our data collection at a total cost of $16,645. Appendix A provides annotation details and data statistics.
4.2 KiloGram
The KILOGGAMM dataset (Ji et al. 2022) a resource for investigating abstract visual reasoning. con- tains 1,016 abstract synthetic images of Tangram puzzles, each with 10 11 natural language annotations describing the whole shape (e.g., a dog) and its segmented parts (e.g., head). We use 942 images from the traain/development/// /test splits defined by Ji et al. (2022). These images are colored differently based on the annotations, resulting in 9,432 distinct images in the dataset. We synthesize text descriptions for each annotation by combining phrases describing the whole shape and segmented parts using a template. The correspondence between the segmented parts of the puzzle and the phrases collected by [i et al. (2022). This correspondence is used to automatically generate bounding boxes for cach phrase describing a segmented part. After augmentation, we obtain 9,432 unique image-text pairs with 65k bounding boxes for 34k phrases.
Reference Games We study reference games in KiloGram, following Ji et al. 2022 and past cognitive
science work (Clark& & Wilkes- Gibbs 1986 Fox Tree 1999 Hawkins et al. 2020). The goal is to predict the correct image referred to by the text description from a set of candidate images. Ji et al. (2022 reports the phrases describing the segmented parts play a critical rolc in disambiguating the abstract shapes of the Tangram puzzles. This makes KILOGRAMM reference games an ideal task to sl.dy the relations of the model's understanding of individual phrases and its task performance. Following [ii et al. (2022), we use ten
candidate images in the reference games. We follow the data splits and evaluation of Ji al. (0022). We
illustrate the KILOGRAM reference games in Figure? 2
4.3 Flickr30k Entities
Flickr30k Entities (Plummer et al., 2015) is phrase grounding dataset of 31k unique natural images an- notated with up to five captions each, totaling 158,000 captions. The captions have 448k phrases grouped into 244k coreference chains (i.c., phrases referring to the same visual entities), and each coreference chain is manually annotated with 276k bounding boxes.
Reference Games We formulate a reference games using this data, creating sets of images from the dataset as contexts, and using the caption of the larget image as ils description. is generally challenging
'The images are available via Street Learn: htttps//ssites.ee www.le..ooooooooooooooo..o coom/vieesstreetee haw.r//cccoooon
5

---

Published in Transactions on Machine Learning Research (01/2024)
Task a rear body rear half
a front
Task
Several
American
head
onlookers
Prediction
body half
Prediction bike riders
paraphernalia
This looks like wriggling worm with rear body half, rear, front body half, and head.
Several bike riders in a parade, wearing American
paraphernalia with onlookers nearby.
Figure Illsstation of reference games with phrase grounding in the KILlGRAM (left) and Flickr30k Entities (right) benchmarks. Given the input text (framed at the bottom) and the context (a single column of images), the task is to select the referenced image. We replicate each context multiple times to illus- trate phrase grounding for multiple phrases from the input text (depicted below each column, except the left column in each example). Red bounding boxes show ground truth predictions for both the task (left column) and phrase grounding (remaining columns). Blue bounding boxes show model predictions MDETR for KILOGRAM and ViLT for Flickr30k Entities. In addition, green masks show pixel-wise segmentation predictions made by ViLT for Flickr30k Entities.
to construct contexts that require to reason about the details of the input caption, because random images share little with each other (e.g., if the caption reads "The yellow dog walks on the beach with tennis ball in its mouth.", both of the correct and distractor images should at least contain a dog). We construct reference games by selecting five distractor images with the highost CLIP similarity scores (Radford et al. 2021 to the caption. We adopt the train/ train/deveie hete//dvvllleen. //est splits of Plummer al. (2015) dividing images and captions and constructing one static reference game per caption for evaluation. We illustrate the Flickr30k Entities
reference games in Figure 2
5 Methods
We investigate two contemporary vision and language models on our benchmarks. VILT-ALIGNER is designed phrase grounding extension of state-of-the-art vision and language transformer. MDETR is

---

details Published in Transactions on Machine Learning Research (01/2024)
modulated detector introduced by Kamath et al. (2021) that extends the object detector to address both phrase grounding and vision and language tasks. We discuss model designs and learning strategies.
5.1 Modeling
We cast all three tasks as predicting (z, y) coordinates (i.e., pointing) within the input images. In ToUcH- DOWN SDR, this amount to identifying the coordinates of the hidden bear location. The goal in the KILO- GRAM and Flickr30k Entities reference games images into single image, and treat the task as pointing to the target image. Roughly speaking, this mimics how humans would physically point to the correct image among candidate image set. We study two models:
select image out of set. We concatenate all the
ViLT-Aligner We extend ViLT (Kim et al. 2021 to produce a segmentation probability map over the entire image conditioned on a query string that is part of the input text. We vary the query string to switch between the task (i.c., SDR or reference resolution) and phrase grounding. Phrase grounding is resolved as a conventional segmentation task conditioned on the target phrase. The task themselves (SDR or reference resolution) are solved by taking the x- and y-coordinates with the highest value in the map as the prediction. In TOUCHDOWN SDR, the query string for SDR task identified from the input string heuristically (i.c., touchdown or bear). In the case of the reference games, we use the entire input string the query. Appendix B provides more details of the model architecture.
MDETR MDETR (Kamath et al. 2021 extends the DETR (Carion et al. 2020 object detector to perform phrase grounding in tandem with downstream vision and language tasks. MDETR is currently one of the best models in several phrase grounding benchmarks and demonstrates strong performances in language-vision reasoning tasks. It directly predicts coordinates using a task-specific MLP head and a query vector, which we provide to MDETR's Transformer decoder. The MLP head is applied to the final decoder representation of the query vector to obtaín prediction of coordinates. This makes the MDETR architecture directly applicable to both our tasks and phrase grounding. In practice, we use different MLP heads for our tasks and phrase grounding. We use version of MDETR, where the image encoder is the ere-traieed ResNet-101 (He et al. 2016) and the text encoder is the pre-trained RoBERTa-bass (Liu et al. 2019b We refer the readers for to Kamath et al. 2021 for the full details of the model architecture.
5.2 Training
Phrase Grounding Pre-training (Kamath al. 2021
pre-train MDETR using phrase grounding data
Flickr30k Young et al. 2014), MSCOCO (Lin 2014b/. Flickr30k Entities (Plummer et al. 2015 and GQA (Hudson & Manning. 2019 train bal- ct al.
aggregated from Visual Genome (Krishna et al. 2016),
anced set. We assess the impact of such pre-training by ablating it, and adapting it to ViLT-Aligner. In MDETR, enabling phrase grounding pre-training is as simple as loading pre-trained checkpoints from Ka- math et al. (2021), while disabling it means learning models from scratch, except for pre-trained ResNet-101 and RoBERTa weights. We perform phrase grounding pre-training on ViLT-ALIGNER using the same data. The original data is in the form of bounding boxes. We generate gold segmentation maps by assigning a value of one to the enclosed region and zero otherwise, then normalize the pixel values in the map such that they up to 1.0. Each example in the datasct is composed of text, image, annotated phrases, and gold segmentation maps. During pre-training, ViLT-ALIGNES predicts segmentation map for each annotated phrase by taking an image and a text as inputs. We minimize the KL--divergence between predicted and gold segmentation maps as our pre-training objective. We provide implementation and of pre-training in Appendix C
Fine-tuning with Task Data We fine-tune models on each task data by minimizing the task objective Ltask- As described in [Section 5.1, the task is implemented as predicting the x- and y-coordinates of the input image. In VIL.--LIIGER, we produce the gold sogmentation probability map as Gaussian with its center at the ground-truth coordinates. The task loss is the KL divergence between the gold-standard and predicted segmentation probability maps. In MDETR, the task loss is the I.1 loss between the predicted
7

---

Published in Transactions on Machine Learning Research (01/2024)
and ground-truth coordinates. We also conduct experiment with optional fine-tuning on dataset-specific phrase grounding annotations. In this case, we minimize the weighted sum of task and phrase grounding losses. ViLT-ALIGNER uses the same Kl-divergence loss as in phrase grounding pre-training, In MDETR, we follow the phrase grounding loss in Kamath et al. (2021), which is the weighted sum of three different losses: bounding box detection losses (I.1 and GloU), soft-loken prediction loss, and contrastive alignment
loss. For further details, please refer to Kamath et al. (2021) We provide additional details in Appendix D.
6 Experimental Setup
Fine-tuning We follow the data splits used in previous works, as described in Section 4 We fine-tune our models on ToUCHDOWN SDR, KILOGRAM, and Flicker30k Entities using the training set for 100, 20, and 20 epochs, respoctively. We use an AdamW optimizer (Loshchilov & Hutter 2017) with a lincar learning rale schedule and warm-up steps equivalent to 1% of the total training epochs. We maintain the exponential moving average of our models with a decay rate of 0.9998. For training on KILOGRAM and Flicker30k Entities, we augment reference games by shuffling and choosing a different set of sufficiently challenging distractor images. In KILOGRAM, adhere to the guidelines presented in (Ji et al. identify challenging images. For Flicker30k Entities, we select from the top-20 images with the highest CLIP similarity scores to the caption. We select checkpoints by maximizing task metrics on the validation set (the development set for Flickr30k Entitics). We perform fine-tuning for TOUCHDOWN SDR with cight NVIDIA A6000 GPUs, while for KILOGRAM and Flickr30k Entities, we use eight. NVIDIA GeForce RTX 2080 GPUs. Further details are available Appendix D)
2022 to
Probing Experiments Our experiments focus on models demonstrating explicit phrase grounding ca- pabilitics. A related question important for our conclusions is: can models internally comprehend phrase grounding but fail to display it explicitly? We study the internal activations of ViLT-ALIGNER P which has been fine-tuned solely on task annotations, omitting phrase grounding annotations during fine-tuning, by training linear probe. We extract spatial feature maps containing both image and phrase query information from the ALIGNER after each deconvolution layer. These maps are interpolated to match the size of the last spatial feal ure map of the ALIGNER, then concatenated along the channel dimension. The linear probe has an input dimension that corresponds to the number channels in the concatenated map, and the output dimension is set to one. We implement the probe operation as 1D-convolution to the concatenated feature map, with bilinear interpolation and softmax operations to obtain the final segmentation map. We train this linear probe on the dataset-specific phrase grounding annotations.
Experimenting with Limited Phrase Grounding Annotation We study the impact of the amount of phrase grounding annotations. We initialize VILT-ALIGNER with phrase grounding pre-training and investi- gate fine-tuning with varying amounts of randomly-sampled phrases from each dataset-specific annotations We use task annotation data for all examples.
Correlation Measures We use Pearson correlation coefficient (Benesty et al. 2009 for TOUCHDOWN SDR, and point biserial correlation coefficient for KILLGRAM and Flickr30k Entities. Because of the different choices, correlation coefficients over TOUCIDOWN SDR are not directly comparable to those in reference games in KILOGRAM and Flickr30k Entities. Section 3 provides further details.
Evaluation We evaluate the performance of fine-tuned models for task, phrase grounding, and task- grounding correlations, as described in [Section 3] To evaluate phrase grounding using ViLT-ALIGNER, we convert the probabilistic segmentation map into a binary segmentation map through normalization (dividing
"We do not probe MDETR because of the complexity of the architecture. This makes identifying reasonable probe structure challenge that is beyond the scope of this work.
WW unable to conduct similar study with MDETR due to computational limitations. MDETR highly sensitive to learning rates, requiring extensive hyperparameter tuning to achieve reliable results. Each hyperparameter combination in the sweep requires significant computational resourccs, approximately 1,736 A6000 hours for the ToUCHDOWN SDR datasct, which translates roughly nine days of computation with eight A6000 GPUs. This cost grows linearly each time we try new hyperparameter combination.
8

---

Published in Transactions on Machine Learning Research (01/2024)
by the maximum value in the map) and thresholding. The threshold is treated as inference hyper- parameter, determined by maximizing the phrase grounding mean-IoU on the validation set.
Task Ace. _ Phrase Grounding M-loU
Correlation
KiloGram
Flickr30k Entitles
Touchdown SDR
0.9
0.9
40
0.5
40
Correlation
M-1ol
0.6
0.6
20
0.3
0.3
20
0.4
0
0
10
60
65
61
Task Anc.
45
30
61
59
0
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Annotated Images (%)
Annotated Images (%)
Annotated Images (%)
Figure 3: Fine-tuning VILT-ALIGNER with varying amounts of dptaset-speciffe phrase grounding annotations. In the figure, the z-axis indicates the proportion of phrases annotated with bounding boxes, while the y-axes represent the metrics for phrase-grounding, lask-grounding correlation, and the task performanccs.
7 Results and Analysis
We study four model variants for each of the datasets, varying the use phrase grounding pretraining and dataset-specific phrase grounding annotations during fine-tuning, as detailed in Section 5.2 We also conduct probing experiments and study the impact of varying the amount of phrase grounding annotation, both using ViLT-ALIGNER, Tabll 2. Table 3 and Table 4 summariiz the results from KILOGRAM, Flickr30k Entities, and TOUCHDOWN SDR. Figure 3 visualizes learning curves with different amounts of phrase grounding annotation. We first summarize overall trends, and then discuss cach benchmark in detail soparately.
Overall, our findings confirm that contemporary vision and language models can perform strongly on tasks, despite weak phrase grounding ability and low gask-grounding correlations. This is especially true when mod- cls are initialized from phrase grounding pro-training. Fine-tuning these models tasks consistently yields strong performance, indicating the utilization of knowledge in grounding phrases to visual regions. However, these models still face challenges with phrase grounding and exhibit low task-grounding correlations. Our probing results further confirm our findings, showing only slightly better phrase grounding performance and lask-groudding correlations, but still lagging significantly behind ViLT-ALIGNER when jointly fine-tnnd with phrase grounding annotations. This suggests that model activations can only ground phrases a limited extent.
Jointly fine-tuning on dataset-specific phrase grounding annotations help with both phrase grounding performance and correlation to task performance. This is maybe not surprising, but it illustrates that although we would expect these capabilities to arisc from task training, this does not happen to degree we would desire, even though models have the ability to do so. Varying the amount of phrase grounding data shows different trends across the datasets (Figure 3). Flickr30k Entities needs the least amount of phrase grounding annotation reach high correlation between task performance and phrase grounding. The other two datasets show that generally more phrase grounding data. This is potentially due to how Flickr30k Entities is related to the pre-training data compared to TOUCHDOWN SDR and, especially, KILOGRAM.
9

---

Published in Transactions on Machine Learning Research (01/2024)
KiloGram
Grounding Annotations
Task(1)
Grounding(1)
Model
Corr(t)
Pretrain
Finetune
Accuracy
Rai
R810
M-JoU
Development Results
Random
10.00
CLIP (zcro-shot) Ji al.
2022
X
15.0
MDETR (zero-shot)
X
0.23
0.74
1,20
1.95
ViLT-AuGGN (zero-shot)
X
0.09
1.99
Humans Ji et al.
2022
x
X
63.0
X
X
9.56
0.00
0.00
0.00
2.12
0.29
MDETR
9.56
0.19
0.58
0.81
1.20
0.37
46.50
6.26
14.87
11.09
0.59
58.91
45.02
66.81
41.35
0.75
20.93
0.00
1.60
23.77
16.49
16.55
0.84
VILT-AUIGEB
x
28.96
0.00
1.62
0.00
/
62.60
59.07
49.31
0.87
VILT-ALIGNER Probing
x
X
20.93
0.00
1.58
0.48
X
28.96
0.16
4.13
0.48
Test Results
X
X
9.69
0.00
0.00
0.00
2.12
0.28
MDETR
9.69
0.14
0.52
0.69
1.11
0.35
6.54
14.88
0.58
45.99
69.10
41.93
0.75
x
19.97
0.00
1.59
0.01
x
V
23.10
16.81
16.90
0.84
ViLT-ALIGNER
x
30.33
0.00
1.60
0.01
V
64.43
60.74
50.81
0.87
Table 2: KILOGRAM fine-tuning summary. Each row provides the model name, model variants (based on phrase grounding pretraining and dataset-specific annotations during fine-tuning). and the task, phrase grounding, and task-groumding correlation metrics. (f) reports using the original floating point precision from al. 2022).
This illustrates that in most cases getting the expected phrase-level reasoning is not just a matter of adding few learning cues, but requires more significant tuning.
visualizes wwo examples of output on TOUCHDOWN SDR with VILT-ALIGNER initialized with phrase grounding pre-training and fine-tuned with dataset-specific phrase grounding annotations. In the top example, ViLT-ALIGNER accurately predicts the target location and effectively maps phrases to image regions. The bottom example shows a failure. ViLT-ALIGNER fails to predict the target location correctly. Upon closer inspection, it appears that VIL.-ALLIGNER predicts similar regions for Touchdown and the circle emblem, which are consistent with the semantics of the text description. However, the model seems to mix up the circle emblem with the circular wheel of the gold-colored old car. Appendix E.2 provides qualitative examples for KILOGRAM and Flickr30k Entities.
Figure 4
Kilogram Table summarizes the KILOGRAM the-tuning results, showing the task relatively challeng- ing. We provide development results from Ji et al. (2022) to contextualize this, including for pre-trained CLIP (Radford et al., 2021), which has an accuracy of 15%, only slightly better than the 10% of random guessing. Overall, observe consistent patterns between the development and the test sets. Fine-tuned MDETR without phrase grounding pre-training initialization performs similarly to random guessing. Accu- racy climbs to 46.50% when initialized with phrase grounding pre-training, despite limited explicit grounding ability (11.09 mean-loU). The poor ability to explicitly ground phrases is not solely altributed lo catastrophic forgetting during fine-tuning tuning; MDETR, even when initialized with phrase grounding pre-training but before fino-tuning (i.c., zcro-shot), also exhibits poor performance grounding phrases (1.95 mcan-IoU). However, phrase grounding pre-training beneficial for task performance. Probing shows similar results, with very low phrase grounding performance.
10

---

Published in Transactions on Machine Learning Research (01/2024)
Flickr30k Entities
Grounding Annotations
Task(1)
Grounding(1)
Model
Corr(1)
Pretrain
Finetune
Accuracy
R&1
R&5
R@10
M-JoU
Test Results
Random
16.66
CLIP (zero-shot)
X
X
61.32
MDETR (zcro-shot)
X
20.64
44.13
53.37
19.16
ViLT- ALIG-AL (zcro-shat)
X
11.95
15.06
Humans
X
91.00
X
18.47
0.00
0.00
0.00
5.01
0.45
MDETR
21.07
0.97
2.60
3.18
3.64
0.49
61.38
10.06
18.02
22 29
14.48
0.39
62.00
47.77
70.84
76.23
42.97
0.67
x
33.61
0.00
4.68
0.04
VILT-ALIGNER
58.54
30.33
0.81
63.37
1.70
65.85
41.91
98
0.85
VILT-AUGNER Probing
33.61
0.83
8.45
0.54
63.37
9.85
20.32
0.65
Table 3: Flickr30k Entities fine-tuning summary. Each row provides the model name, model variants (based on phrase grounding pretraining and dataset-specific annotations during fine-tuning), and the end-task, phrase grounding, and task-grounding correlation metrics. We use the development set for selecting the checkpoint and inference included in Appendix F.T. (f) denotes the author evaluation on randomly sampled 100 reference games.
in Flickr30k Entities; the results for the development set are
Fine-tuning with dataset-specific phrase grounding annotations substantially enhances both phrase ground- ing ability (11.09+441.3) mean-IoU) and task-grounding correlations (0.59-x0.75 correlation coefficients), while also resulting in a significant increase in task accuracy 46.50-+5..91% We observe similar trends for VILT-ALIGNER. Our experiments varying the amount of phrase grounding annotations (Figure 3 show that only limited annotation is needed for high correlation with task performance; at 5% of the data, we get almost the same correlation as with all the data annotated. However, the impact on task and phrase grounding performance is more monotonous; adding more phrase grounding annotations keeps improving both until we exhaust our data.
Flickr30k Entities [Tbll 3 summarizes the Flickr30k Entities fine-tuning results. We report the results from the test set, as we conduct checkpoint and inference hyperparameter selection on the development set As with KILOGRAM, MDETR and ViLT-ALIGNER significantly improve task accuracy through phrase grounding pre-training. However, both struggle to exhibit explicit. phrase grounding ability and strong task- grounding correlations without dataset-specific phrase grounding annotations. Probing shows similar results. While we observe small, but non-trivial phrase grounding probing performance with phrase grounding pre- training, it leaves much be desired terms of performance and correlation.
Fine-tuning phrase grounding annotations helps significantly. Phrase grounding ability improves mean-loU in MDETR and 4.70-+40.98 mean-loU in ViLT-ALIGNER, while task-grounding correlation improves 0.39-+0.67 in MDETR and 0.02-+0.85 in ViLT-ALIGNER. When compared to the KILOGRAM results, the improvements in task accuracy from fine-tuning on datasct-specific phrase ground- ing annotations are modest when initialized from phrase grounding pre-training: MDETR KILOGRAM and ViLT-ALIGNER in seeing most of the impact of phrase annotations at 5% of the data, although this applies not only task correlations, but also to phrase grounding performance, in contrast to KILOGRAM.
(63.37-655.85% Flickr30k Entities learning curves (Figure 3) are similar
Touchdown SDR Table 4 summarizes the TOUCHDOWN SDR fine-tuning results, showing similar trends
to Flickr30k Entities and similar patterns between the development and test sets. This includes experiments
Appendix F. provides development results for reference.
11

---

Published in Transactions on Machine Learning Research (01/2024)
Touchdown SDR
Grounding Ann.
Task
Grounding()
Corr(t)
Pretrain
Finetune
Acc.(1)
Con.(1)
Dist.
RØ1
R&10
M-IoU
Development Results
LINGUNET Cheu al 2010
21.81
7.73
7291
MDETR (zero-shot)
7.70
13.89
17.09
11.17
VELT--LLCNER (zero-shot)
8.85
12:27
X
2.27
0.07
437.66
0.07
0.07
0.07
5.20
0.16
MDETR
11.01
1.35
305.66
1.57
2.98
3.64
5.47
0.34
x
53.56
30.31
166.37
0.28
0.41
0.56
6.09
0.23
54.43
31.97
170.81
28.72
40.72
45.44
31.74
0.45
51.72
29.09
195.65
5.06
7.00
0.04
ViLT-ALIGNER
<
54.48
30.96
186,93
25.65
29.94
0.47
59.19
36.14
174.87
5.78
13.21
0.43
61.45
37.94
162.80
39.38
39.45
0.50
ViLT-ALIGNER Prohing
4.46
7.02
9.96
17.74
Test Results
LINGUNET Chen al
2010b
26.11
8.80
7081
X
x
2.29
0.00
435.37
0.09
0.09
0.09
5.15
0.18
MDETR
10.47
1.44
311.02
1.56
2.89
3.75
5.51
0.36
x
56.70 76
35.42
153.69
0.36
0.59
0.71
6.12
0.19
58.06
36.27
150.17
29.12
41.84
46.65
32.23
0.42
X
x
51.41
28.95
201.49
5.27
7.11
0.08
VILT-ALIGNER
56.29
33.78
173.23
26.46
30.76
0.46
61.89
39.40
137.23
6.45
13.85
0.43
62.99
40.68
152.07
39.63
40.06
0.49
Table 4: TOUCHDOWN SDR fine-tuning summary. For each row, we provide information about the model name, model variants (in terms of phrase grounding pretraining and dataset-specific phrase grounding an- notations during fine-tuning), and the end-task/ /phrase grounding/ htapgg//tts-ggronniiii correlation metrics. Acc., Con., and Dist. correspond to task accuracy, consistency, and pixel distance error (Section 3). The symbol (f) reports scores using the original floating point precision used in Chen et al. (2019b).
with MDETR and VILT-ALIGNER, well probing study. It is noteworthy that the overall performance report on TOUCHDOWN SDR is more than doubles the previously reported accuracies on this task, but there is still considerable room for future improvement.
Finc-tuning with dataset-spocific annotations improves phrase grounding dramatically for both MDETR (6.09-+31.74) and VILT-ALIGNER (13.21-39..) mean-IoU) and lask-grounding correlations (0.23-+0. +0.45 and 0.43-+0.50 correlation coefficients), as well as the task accuracy (53.56-+44.439 and 59.19-61.45% accu- racy). Exceptionally, ViLT-ALIGNES demonstrates a significant task-grounding correlation of 0.43, even in the absence of dataset-specific grounding annotations. This suggests that the model may implicitly perform phrase grounding reasoning, despite not explicitly showing it. Varying the amount of phrase grounding annotations FFguur 3) reveals trends that are bit different than with KILOGRAM and Flickr30k Entities. Phrase grounding performance improves very fast, similar lo what we see with Flickr30k Entities. But corre- lations to task performance are slower to rise, and improve steadily with more data. Compared to Flickr30k Entitics, which shows similar task performance trends, this is potentially bocause of the significantly more complex visual input in TOUCHDOWN.
8 Conclusion
We study the ability of vision and language models to acquire and demonstrate phrase grounding reasoning, when this is the main task they are trained for. We introduce three benchmarks to study this question, and propose to study the relation between task performance and phrase grounding via correlation. Our experiments reveal complex landscape, but with clear repeating trends. Models generally show poor
12

---

Published in Transactions on Machine Learning Research (01/2024)
The establishment your right with the old stone archway has large wooden doors Touchdown is in the middle of the 2 doors) where the handles are.
Notice the gold colored car on the left side of the street Touchdown in the middle of the circle emblem/ on the very back of the car between the taillights.
Figure 4: End-task success and failure illustration of a system that achieves strong phrase grounding perfor- mance and high lask-groundigg correlation in Touchdown SDR. We illustrate the outputs of ViLT-ALIGNER (overlaid in green); this model is initialized with phrase grounding pre-training and fine-tuned with dataset- specific phrase grounding annotations. The illustration is over examples not seen during training.
correlation between task performance and phrase grounding ability. Our probing studics show this is not only an issue with explicit demonstration of phrase grounding, but also with implicit information in the model activations. This issue can be alleviated to a great extent with a brute-force approach that includes explicitly training on phrase grounding annotations. In most cases, significant improvements, especially with respect to correlation task performance, can be achieved with relatively limited phrase grounding annotation. However, providing more such data generally helps.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Our work opens up multiple avenues for future work, especially with the apparent move of the community to pre-trained multi-modal large language models (TL.Ms). As such models become more publicly available we consider applying our methods and resources to study them an important direction for future work. Our experimental results also demonstrate the need for an approach that addresses the issues we observe without requiring explicit phrase grounding annotations. While we focus on within-distrriuttion analysis, an important direction for future work is to study generalization on out-of-distributiion data. The resources we create can aid this, by providing training data for models to be used with other datasets. Finally, an important direction to extend our study is to tasks that require sequential decision making, such as vision- and-language navigation (Ku et al., 2020). We hope our work serves as valuable resource and starting point for researchers aiming to understand and address the opaque reasoning processes of vision and language models.
References
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1955-1960, 2016.
Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split. of the visual question answering (vqa) vl. 0 dataset. arxie preprint arXiv: Nbt:1744.0880 08243, 2017.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.44196 2022.
Morris Alper and Hadar Averbuch- Elor. Kiki or bouba? sound symbolism in vision-and-languen language models. arXiv preprinl ar Xic:23300 16781, 2023.
Morris Alper, Michael Fiman, and Hadar Averbuch-Elor. Is bert blind? exploring the effect of vision-and- language pretraining on visual language understanding. In Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition, pp. 6778-6788, 2023.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang, Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the Conference on Compuler Vision and Pallern Recognition, pp. 6077 6086, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton van den Hengel. fision-and-language navigation: Interpreting disuallyggrounde navi- gation instructions ions in real environments. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018b.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the International Conference on Computer Vision, pp. 1425-2433 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonalan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt, Openflamingo: An open-source framework for training large autoregressive vasion-language models. arXiv preprint Xrxiv:2308. 01390, 2023.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlatton coefficient. In Noise Reduction in Speech Processing, pp. 1-4. Springer, 2009.
Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi Following high-level navigation instructions on a simulated quadcopter with imitation learning. In Proceedings of the Robotics: Science and Systems Conference, 2018
in recent days, concurrently putting the final touches to this manuscript, two re-productions of Flamingo Alayrac et 2022 OpenFlamingo Awadalla al. (2023) and IDEFICS Laurençon al. (2023), have just become publicly available from.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the Euroopann Conference Computer Vision, pp. 213 229, 2020.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the Conference on Computer Vision and Pultern Recognition, 2019a.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoar Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 12538-12547, 2019b.
Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In Proceediiggs of the International Conference on Computer Vision, pp. 824-832 2017.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In Proceedings of the European Conference on Computer Vision, 2020.
Volkan Cirik, Mouis-Philippo Moreney, and Taylor Berg- Kink-Kiikaatrick. Visual referring expression recognition:
What do systems actually learn? In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 781-787, June 2018.
Volkan Cirik, Taylor Berg-Kirkpatrien and I-P Morency. Refer360: A referring expression recognition dataset in 360 images. In Proceedings of the Annual Meeting of the Association for Compulational Lin- guistics, 2020.
Herbert H Clark and Deanna Wilkes-Gibbs Referring as a collaborative process. Cognition, 22(1):1-39, 1986.
Yuqing Cui, Apoorv Khandelwal, Yoav Artzi, Noah Snavely, and Hadar Averbuch-Flor. Who's waldo? linking people across text and images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1374-1384, 2021.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017.
Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. Visual grounding via accu- mulated attention. In Proceedings of the Conference on Compuler Vision and Pallera Recognition, pp. 7746-775 7755, 2018
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query gener- ation for referring segmentation. In Proceedings of the International Conference on Computer Vision, pp. 11921-16330, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, 2020.
Zi-Yi Dou, Aishwarya. Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to- tonfine vision-language pre-training with fusion in the backbone. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
Constantin Eichenberg, Sidncy Black, Samuel Weinbach, Lotitia Parcalabescu, and Anette Frank. Magma- multimodal augmentation of generative models through adapter- based finetuning. arXio preprint arXiv:212.052233 2021.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Martin Engilberge, Louis Chevallier, Patrick Pérez, and Matthieu Cord. Finding beans in burgers: Deep semantic-visual embedding with localization. In Proceedings of the Conference on Compuder Vision and Pattern Recognition, pp. 3984 3993, 2018.
Mark Everingham, SM Eslami, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-11)) 1009888-366661 2015.
Jean E Fox Tree. Listening in on monologues and dialogues. Discourse Processes, 27(1):35 53, 1999.
Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. Vqs: Linking segmentations questions and answers for supervised attention in vqa and question-focused semantic segmentation. Tn Proceedings of the International Conference Computer Vision, pp. 1811 1820, 2017.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 6325 6334, 2017a.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa malter: Elevating the role of image understanding in visual question answering In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 6904 6913, 2017b.
S Das Gupta, Point biserial correlation coefficient and its generalization. Psychometriku, 25(4):393 408, 1960.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Contrastive learning for weakly supervised phrase grounding. In Proceedings of the European Conference on Computer Vision, pp. 752 768, 2020.
Robert D. Hawkins, Michael C. Frank, and Noah D. Goodman. Characterizing the dynamics of learning in repeated reference games. Cognilive Science, 14(6)=e12845 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the Conference on Computer Vision and Pallern Recognition, pp. 770-778 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the Interna- tional Conference on Computer Vision, pp. 2961-2969, 2017.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. 2017.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Proceedings of the European Conference on Computer Vision, pp. 108-124, 2016.
Drew A. Hudson and Christopher D. Manning. Gqa: new dataset for real-world visual reasoning and com- positional question answering. Proceedings of the Conference on Computer Vision and Pattern Recognition, 2019.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene le, and Jason Baldridge. Stay on the path: Instruction fidclity in vision-and-language navigation. In Proceedings of the Annual Meeting of the Association for Compulational Linguislics, pp. 1862 1872, July 2019.
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D. Hawkins, and Yoav Artzi Abstract visual reasoning with tangram shapes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Proccedings of the Association for Computational Linguistics, July 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi-modal understanding. In Proceedings of the International Con- ference on Computer Vision, pp. 1780 1790, 2021.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Aishwarya Kamath, Sara Price, Jonas Pfeiffer, Yann LeCun, and Nicolas Carion. Tried: Testing robust image understanding through contextual phrase detection. 2023. URL https://ashkamatti github. io/
TRICD/assets/paper.PP
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural sccnes. In Proceedings of the Conference on Empérical Methods in Natural Language Processing), pp. 787-798, 2014.
Jin-Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. volume 31, 2018
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution region supervision. In Proceedings of the International Conference on Machine Learning, pp. 5583-5588630333 55944 PMLR, 2021.
Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. Multimodal neural language models. In Proceedings of the International Conference on Machine Learning, pp. 595 603. PMLR, 2014.
Noriyuki Kojima, Hadar Averbuch-Elor Alexander M Rush, and Yoav Artzi. What is learned in visually grounded neural syntax acquisition. In Proceedings of the Association for Computational Linguistics, pp. 2635, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123:32-73. 10322-3-2221 2016.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilin- gual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the Confer- ence on Empirical Methods in Natural Language Processing), pp. 4392-4412, 2020.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanprect Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. arXie preprint arXiv:2306. 1atie2306........ 2023.
Thao Minh Le, Vuong Le, Sunil Gupta, Svetha Venkatesh, and Truyen Tran. Guiding visual question
answering with attention priors. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 4381-4390, 2023.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arxio:190s 03557, 2019.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceed- ings of the Conference Computer Vision and Pottern Recognition, pp. (0965-1099)) 2022.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proceedings of the European Conference on Computer Vision, pp. 121 137, 2020.
Zizhang Li, Mengmeng Wang, Jianbiao Mei, and Yong Liu. Mail: A unified maak--iaag-llagguaa trimodal network for referring image segmentation. arXio preprint arXim:211 11.22111.100 10.10777 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755, 2014a.
Tsung- Yi Lin, Michael Maire, Serge J. Belongic, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the Furopean Conference on Computer Vision, 2014b.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the International Conference on Computer Vision, pp. 2980 2900, 2017.
Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the International Conference Computer Vision, pp. 4673-4682, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019b.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xix preprint 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic representation ions for vision-and-languuaee lasks. arXiv preprinl arXiv: 190000008. 2019.
Jiasen Lu, Vedanuj Goswarni, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-l: Multi-task vision and language representation learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 10437-10446, 2020.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the Conference on Computer Vision and Pultern Recognition, pp. 11 20, 2016a.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the Conference on Computer Vision and Pultern Recognition, pp. 11-20, 2016b.
Harsh Mehla, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. Retouchdown: Adding touch- down to streetlearn as a shareable resource for language grounding tasks in street view. arXic preprint (:Xi::2001.03671 2020.
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1004 1015, 2017.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. In Proceedings of the Advances in neural information processing systems, volume 24, 2011.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the International Conference on Computer Vision, pp. 2641 2641-2649, 2015.
Jordi Pont- Tust, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Proceedings of the European Conference on Computer Vision, pp. 647-664. Springer, 2020.
Tingting Qiao, Jianfeng Dong, and Duanqing Xu. Exploring human-like attention supervision in visual question answering. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp- 8748 8763, 2021.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2591-2600, 2019.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning. In Proceedings of the Associalion for Computational Linguistics, pp. 2556 2565, 2018.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. Visually grounded neural syntax acquisition. In Proceedings of the Annual Meeling of the Association for Compulational Lánguisties, pp. 1842-1861 2019.
Alane Suhr, Mike Lewis, James Ych, and Yoav Artzi A corpus of natural language for visual reasoning. In Proceedings of the Annual Meeling of the Association for Compulational Linguislics, pp.
2017.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoar Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Compulational Linguistics, pp. 6418 6428, 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint Xr::::198. 07490, 2019.
Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. Structured matching for phrase localization. In Proceedings of the European Conference on Computer Vision, pp. 696 711, 2016.
Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language-based image segmentation in the wild. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 10213-10222, 2020.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pp. 2048-2057. PMLR, 2015.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In Proceedings of the European Conference on Computer Vision, pp. 521-539, 2022.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denota- tions. Transactions of the Association of Computational Linguistics, 2:67-78, 2014.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the Conference on Computer Vision and Pattern Recogrition, pp. 1307-1315, 2018
Haotian Zhang, Pengchuan Zhang, Xiaowci Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
Yundong Zhang, Juan Carlos Niebles, and Alvaro Soto. Interpretable visual question answering by visual grounding from attention supervision mining. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 249-357, 2019.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and vqa. In Proceddings of the AAAI Conference on Artificial Intelligence, pp. 13041-13044 2020a.
Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. -477, 2020b.
19

---

Published in Transactions on Machine Learning Research (01/2024)
HO Doun SuTERR
On the left will be a short staircase with black door at the top with red awning, stop in front of this. Touchdown will be on the brass door knob on this black door.
There will be two police parked your right separated by white The touchdown the center of the rear wheel of the in front of the tailor.
A green dumpster in front of parking lot. Touchdown is sitting the slanted top dumpster panel, dead center, between the two handles.
The house is yellowish green house with brown shutters. It has red steps leading up to the door. Touchdown is sitting in the first step right in the middle leaning on the center gate post.
Up ahead is brown building your right with few bicycle parked pole. Don't go pass the bicycle. The touchdown sitting inside the crate attached to the bike.
Figure 5: Annotated examples in TOUCHDOWN SDR.
20

---

Published in Transactions on Machine Learning Research (01/2024)
109
van
single
planter
whi
t
e
scaffolding
phone
pole
spaeer
tip
Number of Phrases
104
103
ye
wind
bicycle
garage
pipe
Low
ow
brown
orange
cone
O
second
lettering
plcture
street
b
bike
Laddort
unbrella
Side
air
mooden
grey
hand
bus
10a
shop
post t
brich
texr
arronnd
reon
opening
bright
10a
letter
two
barrel
point
bottom
diamond
stayprane
road
base
wall
ses
fence
big
gr
een
synbol
arrow
sidewall
s
2
8
16
32
purple
parkes
Number of Boxes per Phrase
stand
doubl
barrier
buildin
last
inter serrssece sec ti gg
awning
first
hydrant
traffica
circle
mai
lbox
(a) Number of annotated boxes phrase
Bong
short
light
median
bag
door
Paran
box
banner
beskeet
pink
stripe
par
bin
Hine
bus
paper truck gold
dumps
arch
e
store
colored
word
Center
cheir
tree
Height
rontt
lar
tall
sat
entrance
ge
trast
g
gato
middle
end.
silver
pot
stor
stop
sath
lasellard
Flowor
lamp
cap
square
small
logo
oillar
black front
cornergray
metal
plant
stubr
face
gar
flag
bage
intorgele
f
e
glass
open
Hertped
concrete
dark graffiti
Width
doorway cons
truction
condit
loner
trashcan
number
(b) Spatial distribution of annolated boxes
(c) Word cloud
Figure 6: Analysis of TOUCHDOWN SDR phrase grounding annotations.
A Details of Touchdown Phrase Grounding Annotation
In this section, we provide statistics about the phrase grounding annotations in the TOUCHDOWN SDR dataset, and delve into the specifies of the annotation procedures.
A.1 Dataset Statistics
The phrase grounding annotations we provide for TOUCHDOWN SDR consist of phrass-conndiionned bounding boxes for 25,391 image-text pairs obtained from the SDR part of the TOUCHDOWN dataset Chen et al. (2019b). We extract phrases from the text descriptions by using the spaCy2 noun chunker D Honnibal & Montani (2017). ). After annotation, we obtain 145,839 grounded phrases, each annotated with at least one bounding box. On average, cach description contains 5.74 grounded phrases (maximum: 20, minimum: 0), with a mean length of 2.43 (maximum: 13, minimum: 1) words per phrase. The vocabulary size of all phrases 2,665 word types I0 Fach grounded phrase is annotated with an average of 1.15 (maximum: 24, minimum: 1) bounding boxes, with an average height, width, and enclosed area of 185.77 pixels, 344.85 pixels, and 92,269.56 squared pixcls. These annotations made on images with the dimensions 800 x 3712 pixels. We illustrate the distribution of annotated bounding boxes per phrase and the spatial distribution of annotated boxes in Figure 6a and Figure 6b respectively.
Figure 5 illustrates annotated examples. Annotated phrases are not limited to visual entities (e.g., a short staircase in the first row) but also correspond to spatial regions in the image (e.g., front in the third row) and pronouns referring visual entities (c.g., it in the fourth row). The annotated bounding boxes for the bruss door knob (first row) and the rear wheel (second row) illustrate that reasoning in TOUCHDOWN often
We remove the keywords bear and touchdown from the datasct they not visible in the images.
10We the Spacy tokenizer to obtain the counts of words and word types: http::://aacc..o///ooooo wttpeeeaaeeeeeeeeeeeeeeeeeeeeeee caapeeenncrreeee______ _er_____e_em_______e__ee_____e__
21

---

Published in Transactions on Machine Learning Research (01/2024)
Instructions
Shortouts
Draw bounding around Nighighted description.
Labels
Text description: On your right you should blue v Touchdown" the middle of the wheel
Search tabels
(1) your right
blue
CUUUUR
(4) Touchdown
IT
the middle
(6) the rear wheel
IHE
Submit
O
Figure 7: A screenshot of TOUCHDOWN SDR phrase grounding annotation interface.
involves resolving phrases to relatively small image areas. In addition, we provide a word cloud visualization of the words annotated phrases in Figure 6c
A.2 Annotation Procedures
We collect bounding box annotations via Amazon Mechanical Turk. Figure illustrates our annotation UT. Given an image, complete text description, and a set of phrases, workers are asked draw bounding boxes all the image regions referred to by cach phrase draw no bounding boxes if the words cannot be grounded. Our annotation UI has zoom feature to allow users to annotate small image regions precisely. During annotation, we use image size of 1600 x 3712 pixels, but we preprocess the images by cropping the top (i.e., empty sky) and bottom (i.e., plain road) regions to construct the final dataset with an image size 800 x 3712 pixels, following the approach bottom regions of the image rarely contains the useful visual region to cues the location of Touchdown, while they consumes extra disk space to store images and consumes significantly more GPU memory during model training.
Chen et al. (2019a). This is due to the top and the
We qualify 185 workers through qualification task. Our qualification task requires watching two short instructional videos and correctly annotating two examples. We split workers into master, mediocre, and bad workers by sampling and manually checking workers' annotations for the first few thousand annotation tasks. We identify 60 master workers and ask them to annotate the rest of the examples and re-annotate the work of bad workers. We base payment of $2 and $0.1 for the qualification and main annotation tasks. For the main annotation task, we issue bomus payment of $0.04 per phrase during the worker selection and $0.05 per phrase for the work done by the master workers. The total annotation cost is $16,645.
22

---

Published in Transactions on Machine Learning Research (01/2024)
Aligner
Aligner
Softmax
s
Aligner
Bilnear
Deconv
Aligner
2x
ReLU
Deconv
VILT
The building Touchdown fire escape The building the right that is color with arched black door and white trimmod windows. The black fire escupe hanging rung is in line with the top of the door. Touchdown is sitting the bottom rung of that fire escape.
way that the bottom
Figure 8: ViLT-ALIGNER model architecture illustration.
B Details of ViLT-Aligner Model Architecture
We extend ViLT Kim et al. 2021 a statc-of-the-art vision and language transformer, to handle phrase grounding. Our approach casts phrase grounding as a language-ooditionned image segmentation, and we integrate a lightweight neural network module Aligner into ViLT. The complete model is shown in Figure 8) The Aligner modeule, depicted on the left side of Figure g takes a query vector q IR and vector representations (i,....,ik) of K image patches as inputs. These patches represent full image of size H x W, with each patch corresponding to P P pixels. The output of Aligner segmentation probability map S E [0, 1| with the pixel values normalized globally and high probability assigned to image regions referred to by the query.
ALIGNER first reshapes the image patch representations (i,...., ix) into feature map O, E RH/Pxw/PXX,, The query vector q then tiled spatially and concatenated with Op to obtain a combined feature map O omb E (gH/PxW/PX(D,tD)) In the feature map O cmb: the query information is replicated and concatenated along the channel dimension. The combined feature map O cmb is then upsampled and its feature dimension is gradually reduced to through series of deconvolution, ReLU activations, and bilinear interpolation. Softmax operation is performed to the final feature map to compute the segmentation map S. For phrase grounding during inference, every pixel value in the segmentation probability map S is divided by the map's maximum value, followed by thresholding to get the final segmentation mask. The optimal threshold is chosen by maximizing the mcan-IoU using the validation split of the datascts.
We integrate ALIGNER with VILT (Kim et al. 2021.. The inputs to ViL.T are tokenized text 7 = (x1, ...,, .T) and an image segmented into square patches (i1...., ii), where the image dimensions are H W, each patch has size P x P. The patches embedded using linear projection, and the text using learned token embeddings. Positional embeddings are added to both. The text and patch embeddings are concatenated as single sequence. VILT uses Vision Transformer layers Dosovitskiy et al. (2020) to iteratively compute representations for each token and patch. The final output of ViLT is a sequence of representations for all tokens and patches xx.....,7,,I,...... ir..,,...,i The integration with ALIGNER straightforward. The patch representations (i.,...,ia ix) are passed directly to ALIGNER. To compute the query vector q, we perform mean-pooling of the representations corresponding to the tokens within the query phrase (Zi...., 2r,), giving q= Σx k- X4. ViLT is pre-trained four large-scale image-caption datasets (Ordonez et. al. 2011 Lin et al. 2014a Krishna et al. 2016 Sharma et al. 2018 using a combination of masked language model- ing and image-text matching objectives, yielding high-quality task-agnostic language-vision representations. However, Aligner must be learned from scratch for phrase grounding.
23

---

Ei=I,...M KL,(S,, Published in Transactions on Machine Learning Research (01/2024)
C Details of ViLT-Aligner Pre-training
we conduct phrase grounding pretraining on ViLT-ALIGNER utilizing the large- scale phrase grounding annotations gathered by Kamath et al. (202I). This dataset includes an extensive collection of phrase grounding annotations from Visual Genome (Krishna et al. 2016), Flickr330k (Young let al. 2014), MSCOCO (Lin et al., 2014b), Flickr30k Entities (Plummer al. 2015), and GQA (Hudson & Manning 2019) train balanced set, encompassing total of 1.3 million image-text pairs. The instructions for downloading and preprocessing the dataset are available at https:///gtthie httptttub...m/asaaaaaaaaaaaaaaaa wttp//////aaaaaaaaaaaaeeeeeeeeee
As outlined in Section 5.1
Each example in the pre-training dataset is represented by a tuple (1, (pI, PM), (S...... SM)), where I is an image, H is a text, (pi..... PPs) are annotated phrases, and (Si..... SM) are corresponding gold segmentation maps. Here, each phrase p is a sub-sequence of tokens from J, and each gold segmentation map S has the same dimensions as J. For each phrase p, VIL.T-LIGNER predicts the probabilistic segmentation map S, conditioned on both the image and the text σ. Annotations in the pre-training dataset are provided bounding boxes, from which generate gold segmentation maps S by assigning a value of one to the enclosed region and zero otherwise. We then normalize the pixel values in the map such that they sum up to 1.0. To pretrain the model, we use the KL-divergence loss between the predicted and gold segmentation maps, given by:
L Lpretrain
M
We find a KL-divergence loss to be preferred to binary cross-entropy loss (He et al. 2017), as it avoids severe class imbalance issues arising from the gold segmentation ion regions of objects with different. sizes
Prior to phrase grounding pre-training, we initialize ViLT with the image-caption pre-trained weight, which is provided by (Kim et al., 2021) and can be found at https://github. atttp:://gtttub..cc//daael111111 http:////dddl//////////eeaaaaaa
fowmload/200s/vitt 2_0k_00k_aaaaa mlm itm. ckpt. This image-caption pre-training conducted four large-
scale image-caption datasets (Ordonez et al.) 2011 Lin et al. 2014a Krishna et al., 2016 Sharma et al.,
2018) using a combination of masked language modeling and image-lext matching objectives. The weights of ALIGNER is initialized randomly, and phrase grounding pre-training is performed end-to-end. We use an AdamW optimizer with a base learning rate of 1c-4 and weight decay of lc-2 during phrase grounding pre-training, similar to the fine-tuning setup described in [Section 6 The learning rale is warmed up for 1% of the total training steps and decayed linearly to zero for the remainder. An exponential moving average (EMA) with a decay rate of 0.9998 also utilized. Image augmentation, consisting of random resizing and cropping input images and gold segmentation maps, is performed using the approach detailed in Kamath et al. (202)) We pre-train models for 20 epochs with batch size of 16, using 8 NVIDIA A6000 GPUs for one week.
Details of Fine-tuning
We provide the hyperparameter details for fine-tuning VILT-ALIGNER and MDETR in Table 5 and Table respectively. In Table and [Table 6 the a hyperparameters denote the coefficient for the weighted sum that balances the different losses during fine-tuning. When fine-tuned with phrase grounding annotations, ViLT-ALIGNER computes the final fine-tuning loss by taking the weighted sum of the task loss and the phrase grounding loss, as follows:
Läne-tune = Otask * Irask + O'grounding Ligraunding
For MDETR, phrase grounding involves three distinct losses: bounding box detection losses (L1 and GloU), soft-token prediction loss, and contrastive aligument loss. Therefore, the final fine-tuning loss is calculated as follows:
L-fin-tunc = -ftask Trtak + (OL1 * I4. + aGIoU LGGot:) + Ossoft Traott + Ocontrastive Ieconrrsttvv-
Note that MDETR uses three different learning rates for the text-encoder, the vision backbone and the rest of the model (i.e., the transformer encoder-decoder and task-specific modules) as summarized in Table 6
11 Manually tuning a focal loss Lin al. 2017) can also be an alternative approach mitigate class imbalance problems.
24

---

Published in Transactions on Machine Learning Research (01/2024)
Kilogram
Flickr30k Entities
Touchdown SDR
Training epochs
20
20
100
Batch size
16
16
16
Optimizer
AdamW
AdamW
AdamW
Learning rate
le-4
le-4
le-4
Warmup steps
1%
1%
1%
Weight decay
0.01
0.01
0.01
Drask
0.5
0.5
0.5
Agrounding
0.5
0.5
0.5
Table 5: Fine-tuning hyperparameters for ViLT-ALIGNER.
Kilogram
Flickr30k Entities
Touchdown SDR
Training epochs
20
20
100
Batch size
16
16
16
Optimizer
AdamW
AdamW
AdamW
Learning (text encoder)
2.1e-5
2. 2.15
7e-5
Learning rate (vision backbone)
4.2e-6
4.2e-6
1.4e-5
Learning rate (rest)
4.2c-6
4.2c-6
1.4c-4
Warmup steps
1%
1%
1%
Weight decay
0.01
0.01
0.01
Otask
9
9
01,
5
OGGoU
2
2
2
Qgoft
1
I
1
Ocontrustive
1
Table 6: Fine-tuning hyperparameters for MDETR.
Flickr30k Entities
Grounding Annotations
Task(t)
Grounding())
Model
Corr(t)
Pretrain
Finctune
Accuracy
R@01
Re55
R@10
M-loU
Development Results
CLIP (zero-shot)
59.37
MDETR (zero-shot)
19.95
42.41
51.26
18.73
(zero-shot)
11.76
14.81
18.03
0.00
0.00
0.00
4.94
0.43
MDETR
20.86
1.10
2.47
2.94
3.51
0.51
60.79
10.44
18.15
22.22
14.53
0.39
61.09
45.74
68.43
74.22
41.28
0.67
X
35.06
0.00
4.55
0.00
VILT-ALIGNER
/
56.90
26.43
29.10
0.80
61.83
0.00
4.66
0.01
65.04
43.30
39.70
0.84
Table 7: The summary of our fine-tuning study on the development set of Flickr30k Entities.
E Additional Results and Analysis
E.1
Flickr30k Entities Development Results
While both the KILOGRAM and TOUCHDOWN SDR datasets have a dedicated validation set (Chen et al. 2019a; Ji et al. 2022) for selecting checkpoints and inference hyperparameters, the Flickr30k Entities datasct does not. Therefore, we select checkpoints and inference hyperparameters using the development and evaluate the same data in Table 7. Comparing to the fine-tuning results evaluated the test set in Table 3
25

---

Published in Transactions on Machine Learning Research (01/2024)
Prediction
End-task
body
legs
head
neck
Prediction
End-task
a rear body half
a rear
body half
front
head
This looks like a wriggling worm with rear body half, a rear, a front body half, and head.
This looks like llama with body, legs, head, neck.
Figure 9: Task success (left) and failure (right) illustration of system that achieves good phrase grounding performance and high task-grounding correlation in Kilogram reference games. We illustrate the outputs of MDETR, initialized with phrase grounding pre-training and fine-tuned with datasct-specific phrase ground- ing annotations. Model predictions are indicated by blue circles or bounding boxes, while ground-truth annotations are denoted by red circles or bounding boxes. Only the top-1 prediction for phrase grounding bounding box is shown in the illustration, while ground-truth phrase grounding annotations may contain multiple bounding baxes. These outputs are based examples that were not seen during the training process.
we do not observe a significant improvement by over-fitting checkpoints and inference hyperparameters the development set, shown in Table 7
E.2 Qualitative Examples
In this section, we extend our qualitative illustration from Figure to demonstrate how the system performs in KILOGRAM and Flickr30k Entitics reference games, as shown in Figure and Figure 10, respectively. To account for model diversity, we illustrate MDETR for KILOGRAM and VILT-ALIGNER for Flickr30k Entities. Both models are initialized with phrase grounding pre-training and fine-tuned with dataset-specific
26

---

Published in Transactions on Machine Learning Research (01/2024)
End-Lask Prediction
A group of people
End-Lask
Several bike riders
American
bridge
bost
Onlookers
Prediction
paraphernalla
Several bike riders in parade,, wearing American parapherualin with
group people bridge beside
onlookers neatby.
Figure 10: Task success (left) and failure (right) illustration of system that achieves good phrase grounding performance and high task-grounding correlation in Flickr30k Entitics reference games. We illustrate the outputs of ViLT-Alinger, initialized with phrase grounding pre-training and fine-tuned with dataset-specifie phrase grounding annotations. Model predictions are indicated by blue circles or green segmentation masks, while ground-truth annotations are denoted by red circles or bounding boxes. These outputs are based on examples that were not seen during the training process.
grounding annotations. In Figure 9 we present MDETR's output Kilogram. In the left examples, MDETR accurately predicts the correct image in the reference game. Furthermore, the top-1 bounding box predictions of MDETR's phrase grounding effectively map phrases from the sentences to relevant image regions, such a body, legs, head, and a neck. However, in the right example, MDETR fails to predict the correct image in the reference game. MDETR struggles to map each phrase to a relevant region, evidenced by the lack of consensus over multiple images in the top-1 bounding box predictions for each phrase. In Figure 10, we present ViLT-ALIGNER'S 's output Flickr30k Entities. the left examples, ViLT-ALIGNER accurately predicts the correct image in the reference game. Additionally, ViL.TALLIGER'SS phrase grounding predictions point to relevant image regions for phrases such a group of people, bridge, boot, and boat likely plays decisive role in making the correct prediction. However, in the right example, ViLT-ALIGNER fails to predict the correct image in the reference game. ViLT-ALIGNER confuses American paraphernalia with the color pattern of the biker's uniform in the sccond bottom image, resulting in an incorrect prediction.
27


Reviews:
[{"review_id": "Wsg6gaCbxO", "reviewer": "Reviewer_EnRd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper examines the relationship between task performance and phrase grounding through correlation and introduces three corresponding benchmarks. Furthermore, it proposes a brute-force training approach for ground phrase annotations to address the inconsistency between the ability to ground phrases and perform tasks.\n\n2. Datasets are expanded through the annotation of correspondences between phrases in the input language and regions in the image context.\n\nWeaknesses:\n1. The authors claim that \"poor correspondence between task success and phrase grounding would indicate the model does not acquire the reasoning the benchmark aims to study, even if task performance itself is relatively high.\" I have some concerns about this statement. If the model can achieve good performance, it suggests that the model can understand the semantic meaning of both textual and visual information and establish connections between them. If, as the authors suggest, this is not the case, then what is the reason for this good performance? It might be beneficial to conduct a more in-depth investigation into this matter.\n\n2. Basically, the performance improvement resulting from the proposed brute-force training on ground phrase annotations is primarily attributed to the detailed datasets, which offer a fine-grained correlation between textual and visual information. It's important to note that the dataset will only be made publicly available upon publication.\n\n3. From Tables 2, 3, and 4, it is evident that the performance of MDETR is notably influenced by Pretraining, as opposed to Finetuning, where Pretraining can significantly enhance its performance. However, in the case of the ViLT-ALIGNER method, both Pretraining and Finetuning appear to have a similar impact on its performance. What could be the reason behind this difference? It might be beneficial to include a discussion regarding this matter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines the relationship between task performance and phrase grounding through correlation. The experiments demonstrate a weak correlation between task performance and phrase grounding ability, attributable to both the explicit demonstration of phrase grounding and the implicit information in the model activations. To address this issue, the authors propose a brute-force approach, which involves explicit training on phrase ground annotation. If the dataset is made publicly available, it has the potential to benefit various communities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xeV84sSxuA", "reviewer": "Reviewer_ZZTb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper studies an interesting question of whether the models are performing well at the task for the right reasons. I agree with the authors that ideally, models should show high correlation between downstream task performance, and the underlying ability required to do well on the task.\n- The paper presents a diverse set of experiments to measure this correlation. Touchdown requires grounding phrases to regions in a panoramic image, kilogram requires abstract visual reasoning, while Flickr30k entities is used to study visual grounding in the context of reference games (choosing an image from a pool of distractor images based on the description). I also appreciate the authors trying two kinds of architecture (ViLT aligner which predicts segmentation masks, and MDETR which produces bounding boxes for phrase grounding).\n- I appreciate the authors explaining the annotation procedure for collecting phrase grounding annotations in great detail.\n\n### Weaknesses\n\n- While I agree generally that ideally, models should demonstrate correct reasoning abilities, I am not fully convinced that phrase grounding should emerge for the tasks evaluated in the paper. For instance, in the touchdown SDR task, the model might directly learn to find the the last object described in the description instead of following every detail of the description.  E.g., a task description like “A light-colored building with the several red doors and arched doorways. There is an American flag by the last door. Touchdown is sitting on the stars of the flag” doesn’t involve reasoning about red doors and arched doorways, as long as the method can identify the red American flag.\n- Secondly, the inner understanding of the model might not be fully reflected in the specific evaluation  / metric used to measure their understanding. Approaches like GradCam have shown in the past that the models do exhibit some useful interpretable behaviours (focusing on the cat when classifying an image as a cat, and focusing on dog, when classifying the image as a dog). Models that focus on interpreting this latent understanding might demonstrate a higher correlation than explicitly checking for the underlying reasoning abilities using a hand-defined metric.\n- I wonder if a task like RxR is better suited to measure phrase grounding and downstream task performance because it requires explicit \"interaction\" / intermediate actions to successfully navigate to the target object, which might not be possible without visual grounding abilities? \n- The paper is missing discussing important works like Localized Narratives (ECCV 20), RxR (EMLNP 2020) etc that collect a large scale dataset of dense visual grounding. These works not only form important pre-training data for the methods, but also form good evaluation benchmarks. Other papers like (Taking a HINT, Selvaraju et al, ICCV 2019) which looked at the correlation between task performance and visual grounding are also missing from the discussion.\n\nLow-level comments / clarifications\n\n- During phrase-grounding evaluation, what is considered “recall”? Is there a threshold on mean-IoU above which the predicted region is considered a match with the GT region?\n- Section 5.2 mentions that the gold segmentation maps are generated by assigning a value of one to the enclosed region by a bounding box and zero otherwise. It’d probably be better to use an approach like Segment Anything (SAM) to generate a segment mask that is more accurate and can reason about object boundaries.\n- It’s unclear to me why the performance on the task is so low when performing task-specific fine-tuning, but without pre-training. MDETR on TouchDown is 11.01% compared to 54.43% when pre-training and fine-tuning. Is that because the model is over-fitting (and not generalising to the test set?)\n- Similarly, pre-training on the phrase grounding task should show decent performance on the grounding task but in Table 4, MDETR shows 0.28 R@1 which seems ver low given that the pre-training was on the phrase grounding task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the connections between the performance of vision-language models on their ability to ground phrases in images,  and performance on downstream tasks involving images and text. Some tasks involve pointing to regions in the images, while others involve picking an image from a set of images given an input text description.  The papers show that models like MDETR and ViLT show  poor correlation between their ability to ground phrases and solve tasks. The authors also show that while explicit phrase grounding is poor despite strong task performance, even implicit phrase grounding (as evaluated by training a linear probe over model’s feature maps) is poor. The authors find that with explicitly training on phrase grounding can help alleviate the poor correlation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improved related works discussing other methods that have studied visual grounding in relation to task performance. Also have a discussion about datasets that have dense annotations for visual grounding like Localized Narratives and RxR. \n- Improve certain explanations in the paper such as evaluation metrics, and the difference in task performance without and without pre-training / fine-tuning. \n- Discussion around why the model is expected to be good at phrase grounding as measured by explicit metrics (mean-IOU / Recall @k). And why we can say with a very high degree of confidence that low performance on these metrics certainly mean that models don't display virtual grounding.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FYQB7Un5QD", "reviewer": "Reviewer_Mbrg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n-\tVisual phrase grounding is an important task. It is well-motivated to use visual grounding as a common ground to enhance vision and language reasoning tasks.\n\n-\tThe task description is clear, and the paper is generally easy to follow.\n\n-\tNew datasets are released that can be beneficial for future research in this area.\n\nCons:\n\n-\tWhile I appreciate new datasets introduced, fine-grained annotations at the phrase level are too expensive. This results in relatively small and not scalable datasets. This limitation may hinder their use for large-scale models and in practical settings. From Fig. 3, it seems that task performance (not saturated yet) improves linearly with the increased amount of training grounding annotations when the designated task and visual grounding are highly correlated. I am wondering what is considered a sufficient amount of grounding annotations for improved task performance?  \n\n-\tSince visual regions’ sizes vary across phrases, I do not think taking average mean-IoU to quantify overall phrase grounding may not be a suitable choice. I suggest to think about a weighted counterpart, taking into account the relative importance of cross-modality pairs of a text phrase and its corresponding visual region.\n\n-\tThere are multiple typos and incomplete sentences in the paper (i.e., Sec 5.2, Sec 7). The writing needs some improvement. \n\n-\tThe related literature section should discuss previous works that have explored the benefits of phrase-level visual grounding for vision-language tasks, including references [1, 2].\n\nReferences:\n\n[1] Le, T. M., Le, V., Gupta, S., Venkatesh, S., & Tran, T. (2023). Guiding Visual Question Answering with Attention Priors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4381-4390).\n\n[2] Zhou, Y., Wang, M., Liu, D., Hu, Z., & Zhang, H. (2020). More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4777-4786).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the correlation between vision-language task performance and phrase grounding. The authors repurpose three existing datasets with additional phrase grounding annotations to investigate the relation between the original task and the grounding task. Experiments showed inconsistency in current approaches regarding their ability to ground phrases to corresponding visual regions, despite their strong performance in their designated tasks. The paper also demonstrates the benefits of leveraging grounding annotations to improve task performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The minor revision should include the links and information provided to access the dataset, code, and models.

One reviewer recommends including a discussion on how correct reasoning steps could also be evaluated for out-of-distribution data. I think this dataset could serve as OOD data when trained on a different dataset. While experiments are likely out of scope for a minor revision, I think the authors could add a short discussion on this aspect, e.g. including if the training on one dataset e.g. Flickr 30k and evaluating on Touchdown SDR, could be used for such a setting as part of future work.
Decision: Accept (poster)