Introduction
Language understanding is a task proving difficult to automatize, because, among other factors, much of the information that is needed for the correct interpretation of an utterance is not explicit in text BIBREF0. This contrasts with how natural is language understanding for humans, who can cope easily with information absent in text, using common sense and background knowledge like, for instance, typical spatial relations between objects. From another perspective, it is well-known that the visual modality provides complementary information to that in the text. In fact, recent advances in deep learning research have led the field of computer vision and natural language processing to significant progress in tasks that involve visual and textual understanding. Tasks that include visual and textual content include Image Captioning BIBREF1, Visual Question Answering BIBREF2, and Visual Machine Translation BIBREF3, among others.
On the other hand, progress in language understanding has been driven by datasets which measure the quality of sentence representations, specially those where inference tasks are performed on top of sentence representations, including textual entailment BIBREF4, BIBREF5 and semantic textual similarity (STS). In STS BIBREF6, for instance, pairs of sentences have been annotated with similarity scores, with top scores for semantically equivalent sentences and bottom scores for completely unrelated sentences. STS provides a unified framework for extrinsic evaluation of multiple semantic aspects such as compositionality and phrase similarity. Contrary to related tasks, such as textual entailment and paraphrase detection, STS incorporates the notion of graded semantic similarity between the pair of textual sentences and is symmetric.
In this paper we extend STS to the visual modality, and present Visual Semantic Textual Similarity (vSTS), a task and dataset which allows to study whether better sentence representations can be built when having access to the corresponding images, in contrast with having access to the text alone. Similar to STS, annotators were asked to score the similarity between two items, but in this case each item comprises an image and a textual caption. Systems need to predict the human score. Figure FIGREF1 shows an instance in the dataset, with similarity scores in the captions. The example illustrates the need to re-score the similarity values, as the text-only similarity is not applicable to the multimodal version of the dataset: the annotators return a low similarity when using only text, while, when having access to the corresponding image, they return a high similarity. Although a dataset for multimodal inference exists (visual textual entailment BIBREF7) that dataset reused the text-only inference labels.
The vSTS dataset aims to become a standard benchmark to test the contribution of visual information when evaluating the similarity of sentences and the quality of multimodal representations, allowing to test the complementarity of visual and textual information for improved language understanding. Although multimodal tasks such as image captioning, visual question answering and visual machine translation already show that the combination of both modalities can be effectively used, those tasks do not separately benchmark the inference capabilities of multimodal visual and textual representations.
We evaluate a variety of well-known textual, visual and multimodal representations in supervised and unsupervised scenarios, and systematically explore if visual content is useful for sentence similarity. For text, we studied pre-trained word embeddings such as GloVe BIBREF8, pre-trained language models like GPT-2 and BERT BIBREF9, BIBREF10, sentence representations fine-tuned on an entailment task like USE BIBREF11, and textual representations pre-trained on a multimodal caption retrieval task like VSE++ BIBREF12. For image representation we use a model pre-trained on Imagenet (ResNet BIBREF13). In order to combine visual and textual representations we used concatenation and learn simple projections. Our experiments show that the text-only models are outperformed by their multimodal counterparts when adding visual representations, with up to 24% error reduction.
Our contributions are the following: (1) We present a dataset which allows to evaluate visual/textual representations on an inference task. The dataset is publicly available under a free license. (2) Our results show, for the first time, that the addition of image representations allows better inference. (3) The best text-only representation is the one fine-tuned on a multimodal task, VSE++, which is noteworthy, as it is better than a textual representation fine-tuned in a text-only inference task like USE. (4) The improvement when using image representations is observed both when computing the similarity directly from multimodal representations, and also when training siamese networks. At the same time the improvement holds for all textual representations, even those fine-tuned on a similarity task.
Related Work
The task of Visual Semantic Textual Similarity stems from previous work on textual inference tasks. In textual entailment, given a textual premise and a textual hypothesis, systems need to decide whether the first entails the second, they are in contradiction, or none of the previous BIBREF4. Popular datasets include the Stanford Natural Language Inference dataset BIBREF5. As an alternative to entailment, STS datasets comprise pairs of sentences which have been annotated with similarity scores. STS systems are usually evaluated on the STS benchmark dataset BIBREF6. In this paper we present an extension of STS, so we present the task in more detail in the next section.
Textual entailment has been recently extended with visual information. A dataset for visual textual entailment was presented in BIBREF7. Even if the task is different from the text-only counterpart, they reused the text-only inference ground-truth labels without re-annotating them. In fact, they annotate a small sample to show that the labels change. In addition, their dataset tested pairs of text snippets referring to a single image, and it was only useful for testing grounding techniques, but not to measure the complementarity of visual and textual representations. The reported results did not show that grounding improves results, while our study shows that the inference capabilities of multimodal visual and textual representations improve over text-only representations. In related work, BIBREF14 propose visual entailment, where the premise is an image and the hypothesis is textual. The chosen setting does not allow to test the contribution of multimodal representationn with respect to unimodal ones.
The complementarity of visual and text representations for improved language understanding was first proven on word representations, where word embeddings were combined with visual or perceptual input to produce multimodal representations BIBREF15. The task of Visual Semantic Textual Similarity is also related to other multimodal tasks such as Image Captioning BIBREF16, BIBREF17, Text-Image Retrieval BIBREF18, BIBREF19 and Visual Question Answering BIBREF2.
Image Captioning is a task that aims to generate a description of a given image. The task is related to ours in that it is required an understanding of the scene depicted in the image, so the system can generate an accurate description of it. Unlike vSTS, image captioning is a generation task in which evaluation is challenging and unclear, as the defined automatic metrics are somewhat problematic BIBREF20. On the other hand, Text-Image Retrieval task requires to find similarities and differences of the items in two modalities, so we can distinguish relevant and irrelevant texts and images regarding the query. Apart from not checking inference explicitly, the other main difference with regards to vSTS is that, in retrieval, items are ranked from most to least similar, whereas the vSTS task consists on scoring an accurate real valued similarity. A comprehensive overview is out of the scope, and thus we focus on the most related vision and language tasks. We refer the reader to BIBREF21 for a survey on vision and language research.
Many of these tasks can be considered as extensions of previously existing NLP taks. For instance, Image Captioning can be seen as an extension of conditional language modeling BIBREF22 or natural language generation BIBREF23, whereas Visual Question Answering is a natural counterpart of the traditional Question Answering in NLP.
Regarding multimodal and unimodal representation learning, convolutional neural networks (CNN) have become the standard architecture for generating representations for images BIBREF24. Most of these models learn transferable general image features in tasks such as image classification, and detection, semantic segmentation, and action recognition. Most used transferable global image representations are learned with deep CNN architectures such as AlexNet BIBREF25, VGG BIBREF26, Inception-v3 BIBREF27, and ResNet BIBREF13 using large datasets such as ImageNet BIBREF1, MSCOCO BIBREF28 and Visual Genome BIBREF29. Recently, Graph Convolution Networks (GCN) showed to be promising way to distill multiple input types multimodal representations BIBREF30.
Language representation is mostly done with pretrained word embeddings like Glove BIBREF8 and sequence learning techniques such as Recurrent Neural Networks (RNN) BIBREF31. Recently, self-attention approaches like Transformers BIBREF32 provided transferable models (BERT, GPT-2, among others BIBREF9, BIBREF10) that significantly improve many state-of-the-art tasks in NLP. Alternatively, sentence representations have been fine-tuned on an entailment task BIBREF11. We will present those used in our work in more detail below.
The Visual STS Dataset
STS assesses the degree to which two sentences are semantically equivalent to each other. The annotators measure the similarity among sentences, with higher scores for more similar sentences. The annotations of similarity were guided by the scale in Table TABREF4, ranging from 0 for no meaning overlap to 5 for meaning equivalence. Intermediate values reflect interpretable levels of partial overlap in meaning.
In this work, we extend the STS task with images, providing visual information that models use, and assess how much visual content can contribute in a language understanding task. The input of the task now consists of two items, each comprising an image and its corresponding caption. In the same way as in STS, systems need to score the similarity of the sentences with the help of the images. Figure FIGREF1 shows an example of an instance in the dataset.
In previous work reported in a non-archival workshop paper BIBREF33, we presented a preliminary dataset which used the text-only ground-truth similarity scores. The 819 pairs were extracted from a subset of the STS benchmark, more specifically, the so called STS-images subset, which contains pairs of captions with access to images from PASCAL VOC-2008 BIBREF34 and Flickr-8K BIBREF35. Our manual analysis, including examples like Figure FIGREF1, showed that in many cases the text-only ground truth was not valid, so we decided to re-annotated the dataset but showing the images in addition to the captions (the methodology is identical to the AMT annotation method mentioned below). The correlation of the new annotations with regard to the old ones was high (0.9$\rho $) showing that the change in scores was not drastic, but that annotations did differ. The annotators tended to return higher similarity scores, as the mean similarity score across the dataset increased from 1.7 to 2.1. The inter-tagger correlation was comparable to the text-only task, showing that the new annotation task was well-defined.
From another perspective, the fact that we could only extract 819 pairs from existing STS datasets showed the need to sample new pairs from other image-caption datasets. In order to be effective in measuring the quality of multimodal representations, we defined the following desiderata for the new dataset: (1) Following STS datasets, the similarity values need to be balanced, showing a uniform distribution; (2) Paired images have to be different to avoid making the task trivial, as hand analysis of image-caption datasets showed that two captions of the same image tended to be paraphrases of each other; (3) The images should not be present in more than one instance, to avoid biases in the visual side; (4) It has to contain a wide variety of images so we can draw stronger conclusions. The preliminary dataset fulfilled 2 and 3, but the dataset was skewed towards low similarity values and the variety was limited.
The Visual STS Dataset ::: Data Collection
The data collection of sentence-image pairs comprised several steps, including the selection of pairs to be annotated, the annotation methodology, and a final filtering stage.
The Visual STS Dataset ::: Data Collection ::: 1. Sampling data for manual annotation.
We make use of two well-known image-caption datasets. On one hand, Flickr30K dataset BIBREF36 that has about 30K images with 5 manually generated captions per image. On the other hand, we use the Microsoft COCO dataset BIBREF28, which contains more than 120K images and 5 captions per image. Using both sources we hope to cover a wide variety of images.
In order to select pairs of instances, we did two sampling rounds. The goal of the first run is to gather a large number of varied image pairs with their captions which contain interesting pairs. We started by sampling images. We then combined two ways of sampling pairs of images. In the first, we generated pairs by sampling the images randomly. This way, we ensure higher variety of paired scenes, but presumably two captions paired at random will tend to have very low similarity. In the second, we paired images taking into account their visual similarity, ensuring the selection of related scenes with a higher similarity rate. We used the cosine distance of the top-layer of a pretrained ResNet-50 BIBREF13 to compute the similarity of images. We collected an equal number of pairs for the random and visual similarity strategy, gathering, in total, $155,068$ pairs. As each image has 5 captions, we had to select one caption for each image, and we decided to select the two captions with highest word overlap. This way, we get more balanced samples in terms of caption similarity.
The initial sampling created thousands of pairs that were skewed towards very low similarity values. Given that manual annotation is a costly process, and with the goal of having a balanced dataset, we used an automatic similarity system to score all the pairs. This text-only similarity system is an ensemble of feature-based machine learning systems that uses a large variety of distance and machine-translation based features. The model was evaluated on a subset of STS benchmark dataset BIBREF6 and compared favorably to other baseline models. As this model is very different from current deep learning techniques, it should not bias the dataset sampling in a way which influences current similarity systems.
The automatic scores were used to sample the final set of pairs as follows. We defined five similarity ranges ($(0, 1], \ldots ,(4, 5]$) and randomly selected the same amount of pairs from the initial paired sample. We set a sampling of maximum 3000 instances (i.e 600 instances per range). Given the fact that the high similarity range had less than 600 instances, we collected a total of 2639 potential text-image candidate pairs for manual annotation. Figure FIGREF8 shows the proposed methodology can sample approximately a uniform distribution with the exception of the higher similarity values (left and middle plots). In addition, we show that the lower predicted similarities are mainly coming from random sampling, whereas, as expected, the higher ones come from similar images.
The Visual STS Dataset ::: Data Collection ::: 2. Manual annotations.
In order to annotate the sample of 2639 pairs, we used Amazon Mechanical Turk (AMT). Crowdworkers followed the same instructions of previous STS annotation campaigns BIBREF6, very similar to those in Table TABREF4. Annotators needed to focus on textual similarity with the aid of aligned images. We got up to 5 scores per item, and we discarded annotators that showed low correlation with the rest of the annotators ($\rho < 0.75$). In total 56 annotators took part. On average each crowdworker annotated 220 pairs, where the amounts ranged from 19 to 940 annotations. Regardless the annotation amounts, most of the annotators showed high correlations with the rest of the participants. We computed the annotation correlation by aggregating the individual Pearson correlation with averaged similarity of the other annotators. The annotation shows high correlation among the crowdworkers ($\rho = 0.89$ $\pm 0.01$) comparable to that of text-only STS datasets.
Table TABREF10 shows the average item similarity and item disagreement in the annotation. We defined item disagreement as the standard deviation of the annotated similarity value. The low average similarity can be explained by the high number of zero-similarity pairs. Item disagreement is moderately low (about 0.6 points out of 5) which is in accordance with the high correlation between the annotators.
The Visual STS Dataset ::: Data Collection ::: 3. Selection of difficult examples.
In preliminary experiments, the evaluation of two baseline models, word overlap and the ensemble system mentioned before, showed that the sampling strategy introduced a large number of trivial examples. For example, the word overlap system attained $0.83$ $\rho $. This high correlation could be the result of using word-overlap in the first sampling round. In order to create a more challenging dataset where to measure the effectiveness of multimodal representations, we defined the easiness metric to filter out some of the easy examples from the annotated dataset.
We defined easiness as an amount of discrepancy provided by an example regarding the whole dataset. Taking the inner product of the Pearson correlation formula as basis, we measure the easiness of an annotated example $i$ as follows:
where $o_{i}$ is the word-overlap similarity of the $i$-th pair, $\overline{o}$ is the mean overlap similarity in the dataset, and $s_{o}$ is the standard deviation. Similarly, variable $gs_{i}$ is the gold-standard value of the $i$-th pair, and $\overline{gs}$ and $s_{gs}$ are the mean and standard deviation of gold values in the dataset, respectively. We removed 30% of the easiest examples and create a more challenging dataset of 1858 pairs, reducing $\rho $ to $0.57$ for the word-overlap model, and to $0.66$ $\rho $ (from $0.85$) for the ML based approach.
The Visual STS Dataset ::: Dataset Description
The full dataset comprises both the sample mentioned above and the 819 pairs from our preliminary work, totalling 2677 pairs. Figure FIGREF14 shows the final item similarity distribution. Although the distribution is skewed towards lower similarity values, we consider that all the similarity ranges are sufficiently well covered.
Average similarity of the dataset is $1.9$ with a standard deviation of $1.36$ points. The dataset contains 335 zero-valued pairs out of the 2677 instances, which somehow explains the lower average similarity.
Evaluation of Representation Models
The goal of the evaluation is to explore whether representation models can have access to images, instead of text alone, have better inference abilities. We consider the following models.
ResNet BIBREF13 is a deep network of 152 layers in which the residual representation functions are learned instead of learning the signal representation directly. The model is trained over 1.2 million images of ImageNet, the ILSRVC subset of 1000 image categories. We use the top layer of a pretrained ResNet-152 model to represent the images associated to text. Each image is represented with a vector of 2048 dimensions.
GloVe. The Global Vector model BIBREF8 is a log-linear model trained to encode semantic relationships between words as vector offsets in the learned vector space, combining global matrix factorization and local context window methods. Since GloVe is a word-level vector model, we build sentence representations with the mean of the vectors of the words composing the sentence. The pre-trained model from GloVe considered in this paper is the 6B-300d, with a vocabulary of 400k words, 300 dimension vectors and trained on a dataset of 6 billion tokens.
BERT. The Bidirectional Encoder Representations from Transformer BIBREF9 implements a novel methodology based on the so-called masked language model, which randomly masks some of the tokens from the input, and predicts the original vocabulary id of the masked word based only on its context. The BERT model used in our experiments is the BERT-Large Uncased (24-layer, 1024-hidden, 16-heads, 340M parameters). In order to obtain the sentence-level representation we extract the token embeddings of the last layer and compute the mean vector, yielding a vector of 1024 dimensions.
GPT-2. The Generative Pre-Training-2 modelBIBREF10 is a language model based on the transformer architecture, which is trained on the task of predicting the next word, given all the previous words occurring in some text. In the same manner to BERT and GloVe, we extract the token embeddings of the last layer and compute the mean vector to obtain the sentence-level representation of 768 dimensions. The GPT-2 model used in our experiments was trained on a very large corpus of about 40 GB of text data with 1.5 billion parameters.
USE. The Universal Sentence Encoder BIBREF11 is a model for encoding sentences into embedding vectors, specifically designed for transfer learning in NLP. Based on a deep averaging network encoder, the model is trained for varying text lengths, such as sentences, phrases or short textbfs, and in a variety of semantic tasks including STS. The encoder returns the vector of the sentence with 512 dimensions.
VSE++. The Visual-Semantic Embedding BIBREF12 is a model trained for image-caption retrieval. The model learns a joint space of aligned images and captions. The model is an improvement of the original introduced by BIBREF37, and combines a ResNet-152 over images with a bidirectional Recurrent Neural Network (GRU) over the sentences. Texts and images are projected onto the joint space, obtaining representations of 1024 dimension both for images and texts. We used projections of images and texts in our experiments. The VSE++ model used in our experiments was pre-trained on the Microsoft COCO dataset BIBREF28 and the Flickr30K dataset BIBREF36. Table TABREF15 summarizes the sentence and image representations used in the evaluation.
Evaluation of Representation Models ::: Experiments ::: Experimental Setting.
We split the vSTS dataset into training, validation and test partitions sampling at random and preserving the overall score distributions. In total, we use 1338 pairs for training, 669 for validation, and the rest of the 670 pairs were used for the final testing. Similar to the STS task, we use the Pearson correlation coefficient ($\rho $) as the evaluation metric of the task.
Evaluation of Representation Models ::: Experiments ::: STS models.
Our goal is to keep similarity models as simple as possible in order to directly evaluate textual and visual representations and avoid as much as possible the influence of the parameters that intertwine when learning a particular task. We defined two scenarios: the supervised and the unsupervised scenarios.
In the supervised scenario we train a Siamese Regression model in a similar way presented in BIBREF38. Given a sentence/image pair, we wish to predict a real-valued similarity in some range $[1,K]$, being $K=5$ in our experiments. We first produce sentence/image representations $h_{L}$ and $h_{R}$ for each sentence in the pair using any of the unimodal models described above, or using a multimodal representations as explained below. Given these representations, we predict the similarity score $o$ using a regression model that takes both the distance and angle between the pair ($h_{L}$, $h_{R}$):
Note that the distance and angle concatenation ($[h_{x}, h_{+}]$) yields a $2 * d$-dimensional vector. The resulting vector is used as input for the non-linear hidden layer ($h_{s}$) of the model. Contrary to BIBREF38, we empirically found that the estimation of a continuous value worked better than learning a softmax distribution over $[1,K]$ integer values. The loss function of our model is the Mean Square Error (MSE), which is the most commonly used regression loss function.
In the unsupervised scenario similarity is computed as the cosine of the produced $h_{L}$ and $h_{R}$ sentence/image representations.
Evaluation of Representation Models ::: Experiments ::: Multimodal representation.
We combined textual and image representations in two simple ways. The first method is concatenation of the text and image representation (concat). Before concatenation we applied the L2 normalization to each of the modalities. The second method it to learn a common space for the two modalities before concatenation (project).
The projection of each modality learns a space of $d$-dimensions, so that $h_{1}, h_{2} \in \mathbb {R}^{d}$. Once the multimodal representation is produced ($h_{m}$) for the left and right pairs, vectors are directly plugged into the regression layers. Projections are learned end-to-end with the regression layers and the MSE as loss function.
Evaluation of Representation Models ::: Experiments ::: Hyperparameters and training details.
We use the validation set to learn parameters of the supervised models, and to carry an exploration of the hyperparameters. We train each model a maximum of 300 epochs and apply early-stopping strategy with a patience of 25 epochs. For early stopping we monitor MSE loss value on validation. For the rest, we run a grid search for selecting the rest of the hyperparameter values. We explore learning rate values (0.0001, 0.001, 0.01, 0.05), L2 regularization weights (0.0, 0.0001, 0.001, 0.01), and different hidden layer ($h_{s}$) dimensions (50,100, 200, 300). In addition, we activate and deactivate batch normalization in each layer for each of the hyperparameter selection.
Evaluation of Representation Models ::: Results ::: The unsupervised scenario.
Table TABREF26 reports the results using the item representations directly. We report results over train and dev partitions for completeness, but note that none of them was used to tune the models. As it can be seen, multimodal representations consistently outperform their text-only counterparts. This confirms that, overall, visual information is helpful in the semantic textual similarity task and that image and sentence representation are complementary. For example, the bert model improves more than 13 points when visual information provided by the resnet is concatenated. glove shows a similar or even larger improvement, with similar trends for use and vse++(text).
Although vse++(img) shows better performance than resnet when applying them alone, further experimentation showed lower complementarity when combining with textual representation (e.g. $0.807\rho $ in test combining textual and visual modalities of vse++). This is something expected as vse++(img) is pre-trained along with the textual part of the vse++ model on the same task. We do not show the combinations with vse++(img) due to the lack of space.
Interestingly, results show that images alone are valid to predict caption similarity ($0.627 \rho $ in test). Actually, in this experimental setting resnet is on par with bert, which is the best purely unsupervised text-only model. Surprisingly, gpt-2 representations are not useful for text similarity tasks. This might be because language models tend to forget past context as they focus on predicting the next token BIBREF39. Due to the low results of gpt-2 we decided not to combine it with resnet.
Evaluation of Representation Models ::: Results ::: The supervised scenario.
Table TABREF29 show a similar pattern to that in the the unsupervised setting. Overall, models that use a conjunction of multimodal features significantly outperform unimodal models, and this confirms, in a more competitive scenario, that adding visual information helps learning easier the STS task. The gain of multimodal models is considerable compared to the text-only models. The most significant gain is obtained when glove features are combined with resnet. The model improves more than $15.0$ points. In this case, the improvement over bert is lower, but still considerable with more than $4.0$ points.
In the same vein as in the unsupervised scenario, features obtained with a resnet can be as competitive as some text based models (e.g. BERT). gpt-2, as in the unsupervised scenario, does not produce useful representations for semantic similarity tasks. Surprisingly, the regression model with gpt-2 features is not able to learn anything in the training set. As we did in the previous scenario, we do not keep combining gpt-2 with visual features.
Multimodal version of vse++ and use are the best model among the supervised approaches. Textual version of use and vse++ alone obtain very competitive results and outperforms some of the multimodal models (the concatenate version of glove and bert with resnet). Results might indicate that text-only with sufficient training data can be on par with multimodal models, but, still, when there is data scarcity, multimodal models can perform better as they have more information over the same data point.
Comparison between projected and concatenated models show that projected models attain slightly better results in two cases, but the best overall results are obtained when concatenating vse++(text) with resnet. Although concatenation proofs to be a hard baseline, we expect that more sophisticated combination methods like grounding BIBREF40 will obtain larger gains in the future.
Discussion ::: Contribution of the Visual Content
Table TABREF31 summarizes the contribution of the images on text representations in test partition. The contribution is consistent through all text-based representations. We measure the absolute difference (Diff) and the error reduction (E.R) of each textual representation with the multimodal counterpart. For the comparison we chose the best text model for each representation. As expected we obtain the largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations. Note that unsupervised models are not learning anything about the specific task, so the more information in the representation, the better. In the case of use and vse++ the improvement is significant but not as large as the purely unsupervised models. The best text-only representation is the one fine-tuned on a multimodal task, VSE++, which is noteworthy, as it is better than a textual representation fine-tuned in a text-only inference task like USE.
Improvement is consistent for the supervised models. Contrary to the unsupervised setting, these models are designed to learn about the task, so there is usually less room for the improvement. Still, glove+resnet shows an error reduction of $12.9$ in the test set. Finally, use and vse++ show smaller improvements when we add visual information into the model.
Figure FIGREF32 displays some examples where visual information positively contributes predicting accurately similarity values. Examples show the case where related descriptions are lexicalized in a different way so a text-only model (glove) predicts low similarity between captions (top two examples). Instead, the multimodal representation glove+resnet does have access to the image and can predict more accurately the similarity value of the two captions. The examples in the bottom show the opposite case, where similar set of words are used to describe very different situations. The text based model overestimates the similarity of captions, while the multimodal model corrects the output by looking at the differences of the images.
On the contrary, Figure FIGREF33 shows that images can also be misleading, and that the task is not as trivial as combining global representations of the image. In this case, related but different captions are supported by very similar images, and as a consequence, the multimodal model overestimates their similarity, while the text-only model focuses on the most discriminating piece of information in the text.
Discussion ::: The effect of hyperparameters
Neural models are sensitive to hyperparameters, and we might think that results on the supervised scenario are due to hyperparameter optimization. Figure FIGREF35 displays the variability of $\rho $ in development across all hyperparameters. Due to space constraints we show text-only and multimodal concatenated models. Models are ordered by mean performance. As we can see, combined models show better mean performance, and all models except Glove exhibit tight variability.
Conclusions and Future Work
The long term goal of our research is to devise multimodal representation techniques that improve current inference capabilities. We have presented a novel task, Visual Semantic Textual Similarity (vSTS), where the inference capabilities of visual, textual, and multimodal representations can be tested directly. The dataset has been manually annotated by crowdsourcers with high inter-annotator correlation ($\rho = 0.89$). We tested several well-known textual and visual representations, which we combined using concatenation and projection. Our results show, for the first time, that the addition of image representations allows better inference. The best text-only representation is the one fine-tuned on a multimodal task, VSE++, which is noteworthy, as it is better than a textual representation fine- tuned in a text-only inference task like USE. The improvement when using image representations is observed both when computing the similarity directly from multimodal representations, and also when training siamese networks.
In the future, we would like to ground the text representations to image regions BIBREF40, which could avoid misleading predictions due to the global representation of the image. Finally, we would like to extend the dataset with more examples, as we acknowledge that training set is limited to train larger models.
This research was partially funded by the Basque Government excellence research group (IT1343-19), the NVIDIA GPU grant program, the Spanish MINECO (DeepReading RTI2018-096846-B-C21 (MCIU/AEI/FEDER, UE)) and project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018). Ander enjoys a PhD grant from the Basque Government.

Question: What multimodal representations are used in the experiments?
Answer: The second method it to learn a common space for the two modalities before concatenation (project), The first method is concatenation of the text and image representation (concat)

Question: How much better is inference that has addition of image representation compared to text-only representations? 
Answer:  largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations

Question: How they compute similarity between the representations?
Answer: similarity is computed as the cosine of the produced $h_{L}$ and $h_{R}$ sentence/image representations

Question: How big is vSTS training data?
Answer: 1338 pairs for training