Title: LEARNING GROUNDED SENTENCE REPRESENTATIONS BY JOINTLY USING VIDEO AND TEXT INFORMATION

Abstract: Visual grounding of language is an active research field aiming at enriching textbased representations with visual information. In this paper, we propose a new way to leverage visual knowledge for sentence representations. Our approach transfers the structure of a visual representation space to the textual space by using two complementary sources of information: (1) the cluster information: the implicit knowledge that two sentences associated with the same visual content describe the same underlying reality and (2) the perceptual information contained within the structure of the visual space. We use a joint approach to encourage beneficial interactions during training between textual, perceptual, and cluster information. We demonstrate the quality of the learned representations on semantic relatedness, classification, and cross-modal retrieval tasks.

1. INTRODUCTION:

Building linguistic vectors that represent semantics is a long-standing issue in Artificial Intelligence. Distributional Semantic Models (Mikolov et al., 2013; Peters et al., 2018) are well-known recent efforts in this direction, making use of the distributional hypothesis (Harris, 1954 ) on text corpora to learn word embeddings. At another granularity level, having high-quality general-purpose sentence representations is crucial for all models that encode sentences into semantic vectors, such as the ones used in machine translation (Bahdanau et al., 2014) or question answering (Sagara & Hagiwara, 2014) . Moreover, encoding semantics of sentences is paramount because sentences describe relationships between objects and thus convey complex and high-level knowledge better than individual words, which mostly refer to a single concept (Norman, 1972) .

Relying only on text can lead to biased representations and unrealistic predictions (e.g., text-based models could predict that "the sky is green" (Baroni, 2016) ). Besides, it has been shown that human understanding of language is grounded in physical reality and perceptual experience (Fincher-Kiefer, 2001) . To overcome this limitation, one emerging approach is the visual grounding of language, which consists of leveraging visual information, usually from images, to enhance word representations. Two methods showing substantial improvements have emerged: (1) the sequential technique combines textual and visual representations that were separately learned (Bruni et al., 2014; Silberer & Lapata, 2014) , and (2) the joint method learns a common multimodal representation from multiple sources simultaneously (Lazaridou et al., 2015) . In the case of words, the latter has proven to produce representations that perform better on intrinsic and downstream tasks.

While there exist numerous approaches to learning sentence representations from text corpora only, and to learning multimodal word embeddings, the problem of the visual grounding of sentences is quite new to the research community. To the best of our knowledge, the only work in the field is Kiela et al. (2018) . The authors propose a sequential model: linguistic vectors, learned from a purely textual corpus, are concatenated with grounded vectors, which were independently learned from a captioning dataset. However, the two sources are considered separately, which might prevent beneficial interactions between textual and visual modalities during training.

We propose a joint model to learn multimodal sentence representations, based on the assumption that the meaning of a sentence is simultaneously grounded in its textual and visual contexts. In our case, the textual context of a sentence consists of adjacent sentences in a text corpus. Within a distinct dataset, the visual context is learned from a paired video and its associated captions. Indeed, we propose to use videos instead of images because of their temporal aspect, since sentences often describe actions grounded in time. The key challenge is to capture visual information. Usually, to transfer information from the visual space to the textual one, one space is projected onto the other (Kiela et al., 2018; Lazaridou et al., 2015) . However, as pointed out by Collell & Moens (2018) , projections are not sufficient to transfer neighborhood structure between modalities. In our work, we rather propose to exploit the visual space by preserving the overall structure, i.e. conserving the similarities between related elements across spaces. More precisely, we take visual context into account by distinguishing two types of complementary information sources. First, the cluster information, which consists in the implicit knowledge that sentences associated with the same video refer to the same underlying reality. Second, the perceptual information, which is the high-level information extracted from a video using a pre-trained CNN.

Regarding these considerations, we formulate three Research Questions (RQ):

• RQ1: Is perceptual information useful to improve sentence representations?

• RQ2: Are cluster and perceptual information complementary, and does their combination compete with previous models based on projections between visual and textual spaces?

• RQ3: Is a joint approach better suited than a sequential one regarding the multimodal acquisition of textual and visual knowledge?

Our contribution is threefold: (1) We propose a joint multimodal framework for learning grounded sentence representations; (2) We show that cluster and perceptual information are complementary sources of information; (3) To the best of our knowledge, obtained results achieve state-of-the-art performances on multimodal sentence representations.

2. JOINT MULTIMODAL FRAMEWORK FOR SENTENCE REPRESENTATION:



2.1. MODEL OVERVIEW:

Our framework learns multimodal representations for sentences by jointly leveraging the textual and visual contexts of a sentence. The textual resource is a large text corpus C T of ordered sentences. The visual resource is a distinct video corpus C V , whose videos are associated with one or more descriptive captions.

A sentence S is represented by s = F θ (S) and its corresponding video V S by v s = G θ (V S ), where F (resp. G) is a sentence (resp. video) encoder parameterized by θ (resp. θ ). We propose to use a joint approach where the sentence encoder F θ is learned by jointly optimizing a textual objective L T (θ) on C T and a visual objective L V (θ, θ ) on C V . So far, this method has only been applied to words, with good results (Lazaridou et al., 2015; Zablocki et al., 2018) . Note that C T and C V are not parallel corpora but that θ is shared between both objectives; in other terms, sentence representations are influenced by their distinct textual and visual contexts. Any sentence encoder F θ and textual objective L T can be used such as SkipTought (Kiros et al., 2015) , FastSent (Hill et al., 2016) or QuickThought (Logeswaran & Lee, 2018) . In this paper, we focus on SkipThought, and present evidences that our approach also improves over FastSent (section 4.3). In the following, we introduce hypotheses and their derived objectives to tackle the modeling of L V .

2.2. LEVERAGING THE VISUAL CONTEXT:

Most visual grounding works use projections between the textual space and the visual space (Kiela et al., 2018; Lazaridou et al., 2015) to integrate visual information. However, when a cross-modal mapping is learned, the projection of the source modality does not resemble the target modality, in the sense of neighborhood topology (Collell & Moens, 2018) . This suggests that projections between spaces is not an appropriate approach to incorporate visual semantics. Instead, we propose a new way to structure the textual space with the help of the visual modality.

Without even considering the content of videos, the fact that sentences describe or not a same underlying reality is an implicit source of information that we name the cluster information. For convenience, two sentences are said to be visually equivalent (resp. visually different) if they are associated with the same video (resp. different videos), i.e. if they describe the same (resp. different) underlying reality. We call cluster a set of visually equivalent sentences. Leveraging the cluster information may be useful to improve the structure of the textual space: intuitively, representations of visually equivalent sentences should be close, and representations of visually different sentences should be separated. We thus formulate the following hypothesis (see red elements in Figure 1 ): Red arrows represent the gradient of the loss derived from the cluster hypothesis (C), which gathers visually equivalent sentences. For clarity's sake, the term in equation 1 that separates negative pairs is not represented.

V S G θ v s L S L C

The green arrow and angles illustrate the loss derived from the perceptual hypothesis (S), which requires cosine similarities to correlate across the two spaces. The point at the center of each space is the origin.

Cluster Hypothesis (C): A sentence should be closer to a visually equivalent sentence than to a visually different sentence.

We translate this hypothesis into the constraint cos(s, s + ) ≤ cos(s, s -), where s + (resp. s -) is a visually equivalent (resp. different) sentence to s. Following Karpathy & Li (2015) ; Carvalho et al. (2018) , we use a max-margin ranking loss to ensure the gap between both terms is higher than a fixed margin m:

EQUATION

where (s, s + ) cover visually equivalent pairs; visually different sentences s -are randomly sampled.

The cluster hypothesis ignores the structure of the visual space and only uses the visual modality as a proxy to assess if two sentences are visually equivalent or different. Moreover, a ranking loss simply drives visually different sentences apart in the representation space, even if their corresponding videos are closely related.

To cope with this limitation, we suggest to take into account the structure of the visual space and use the content of videos, and then propose a novel approach which does not require cross-modal projections. The intuition is that the structure of the textual space should be modeled on the structure of the visual one to extract visual semantics. We choose to preserve similarities between related elements across spaces. We thus formulate the following hypothesis, illustrated with green elements in Figure 1 :

Perceptual hypothesis (P): The similarity between two sentences in the textual space should be correlated with the similarity between their corresponding videos in the visual space.

We translate this hypothesis into the loss L P = -ρ vis , where ρ vis = ρ(cos(s, s ), cos(v s , v s )) and ρ is the Pearson correlation.

The final multimodal loss is a linear combination of the aforementioned objectives, weighted by hyperparameters α T , α P and α C :

L(θ, θ ) = α T .L T (θ) textual context + α P .L P (θ, θ ) + α C .L C (θ) visual context L V (2)

2.3. VIDEO MODELING:

To evaluate the impact of visual semantics on sentence grounding, we examine several types of visual context. As done in Yao et al. (2016); Guo et al. (2016) , visual features are extracted using the penultimate layer of a pretrained CNN. A video is represented as a set of n images

(I k ) k∈[1,n] . Let (i k ) k∈[1,n]

be the representations of these images obtained with the pre-trained CNN. We present below three simple ways to represent a video V . Note that our model can be generalized to more complex video representations (Ji et al., 2010; Simonyan & Zisserman, 2014b) .

One Frame (F ): this simple setting amounts at keeping the first frame and ignoring the rest of the sequence (any other frame might be used). The visual context vector is v = i 1 .

Average (A): the temporal aspect is ignored, and the scene is represented by the average of the individual frame features: v = 1 n n k=1 i k (Zha et al., 2015) . Temporal Grounding (T ): the intuition is that, in a video, not all frames are relevant to sentence understanding. An attention mechanism allows us to focus on important frames. We set: v = n k=1 β k i k , where β k = softmax(< w u w , N.i k >). The sum ranges over the words w of the sentence s, u w is the fixed pretrained word embedding of w, and N is a learned projection.

3. EVALUATION PROTOCOL:

3.1 DATASETS Textual dataset. Following Kiros et al. (2015) ; Hill et al. (2016) , we use the Toronto BookCorpus dataset as the textual corpus C T . This corpus consists of 11K books: this makes a total of 74M ordered sentences, with an average of 13 words per sentence. Visual dataset. We use the MSVD dataset (Chen & Dolan, 2011) as the visual corpus C V . This video captioning dataset consists of 1970 videos and 80K English descriptions. On average, a video lasts 10 seconds and has about 41 associated sentences.

3.2. BASELINES AND SCENARIOS:

Model Scenarios. We test different variants of our multimodal model presented in section 2. We note these variants M I V (α T , α P , α C ), which depend on: • the initialization I ∈ {p, ∅}: the sentence encoder F θ is either pretrained using the textual objective L T (I = p), or initialized randomly (I = ∅).

• the visual representation V ∈ {F, A, T, R}: where F , A or T are the video modelings described in Section 2.3. We introduce a baseline R, where visual vectors are randomly sampled from a normal distribution to measure the information brought by the video content. Baselines. We propose two extensions of multimodal word embedding models to sentences: • Projection (P): Inspired by Lazaridou et al. (2015) , this baseline is projecting videos in the textual space, while our model keeps both spaces separated. The visual loss is a ranking objective:

L V = s v- max(0, m -cos(s, W.v -) + cos(s, W.v s )) (3)

where W is a trainable projection matrix and m a fixed margin. We note P I V (α T ) the variants of this baseline using the global loss L = α T .L T + L V .

• Sequential (SEQ): Inspired by Collell Talleda et al. ( 2017), we learn a linear regression model (W, b) to predict the visual representation from the SkipThought representations. The multimodal sentence embedding is the concatenation of the original SkipThought vector and its predicted representation: ST ⊕ W ST + b, projected into a lower-dimensional space using PCA. This baseline can also be seen as a simpler variant of the model in Kiela et al. (2018) .

3.3. TASKS AND MEASURES:

In line with previous works on sentence embeddings (Kiros et al., 2015; Hill et al., 2016) , we consider several benchmarks to evaluate the quality of our learned multimodal representations:

Semantic relatedness: We use two well-known semantic similarity benchmarks: STS (Cer et al., 2017) and SICK (Marelli et al., 2014) , which consist of pairs of sentences that are associated with human-labeled similarity scores. STS is subdivided in three textual sources: Captions contains sentences with a strong visual content, describing everyday-life actions, whereas the others contain more abstract sentences: news headlines in News and posts from users forum in Forum. Correlations (Spearman/Pearson) are measured between the cosine similarity of our learned sentence embeddings and human-labeled scores. Hyperparameters are tuned on SICK/trial (results on SICK/train+test are reported in tables).

Classification benchmarks: We use six sentence classification benchmarks: paraphrase identification (MSRP) (Dolan et al., 2004) , opinion polarity (MPQA) (Wiebe & Cardie, 2005) , movie review sentiment (MR) (Pang & Lee, 2005) , subjectivity/objectivity classification (SUBJ) (Scott et al., 2004) , question-type classification (TREC) (Voorhees, 2001) and customer product reviews (CR) (Hu & Liu, 2004) . For each dataset, a logistic regression classifier is learned from the extracted sentence embeddings; we report the classification accuracy.

Cross-modal retrieval on COCO: We consider the image search/annotation tasks on the MS COCO dataset (Lin et al., 2014) . A pairwise triplet-loss is optimized in order to bring corresponding sentences and images closer in a multimodal latent space. Evaluation is performed using Recall@K.

Structural measures::

To analyze the quality of the textual space, we report some measures (computed in %) defined on the MSVD test set:

• ρ vis measures if the similarities between sentences correlate with the similarities between videos.

• E intra = E vs=v s [cos(s, s )] measures the homogeneity of each cluster, by measuring the average similarity of sentences within a cluster.

• E inter = E vs =v s [cos(s, s )] measures how well clusters are separated from each other (i.e. average similarity between sentences of two different clusters).

3.4. IMPLEMENTATION DETAILS:

Videos are sampled at a 3 frames per second rate; afterwards, frames are processed using a pretrained VGG network (Simonyan & Zisserman, 2014a) . The multimodal loss L is optimized with Adam optimizer (Kingma & Ba, 2014) and a learning rate λ = 8.10 -4 . Hyperparameters are tuned using the Pearson correlation measure on SICK trial: m = m = 0.5, µ = 2.5.10 -4 , and mini-batch size of 32 for L V . We perform extensive experiments with L T based on the SkipThought model, using an embedding size of 2400 and the same network hyperparameters as in Kiros et al. (2015) .

4. EXPERIMENTS AND RESULTS:



4.1. VALIDATION OF THE PERCEPTUAL HYPOTHESIS (RQ1):

The perceptual hypothesis holds that the information within videos is useful to ground sentence representations. In our model, this hypothesis translates into the perceptual loss L = L P (i.e. model M p . (0, 1, 0)). Since the perceptual loss is the only component exploiting video content, we compare, in Table 1 , the different video encoders on intrinsic evaluation benchmarks, namely semantic relatedness. The first observation is that our model M outperforms the purely textual baseline ST for all video encoders, which shows that perceptual information from videos is useful to improve representations. We also observe that using random visual anchors (R) improves over ST. This validates our cluster hypothesis, since grouping visually equivalent sentences improves representation -even when anchors bear no perceptual semantics. We further observe that F, A, T > R, which shows that the perceptual information from videos brings a more semantically meaningful structure to the representation space. Finally, regarding the different ways to encode a video, we observe that leveraging more than one frame can be slightly beneficial to learn grounded sentence representations, e.g. A obtains +3.3% average relative improvement over F on ρ SICK P earson . Selecting relevant frames (T ) in the video rather than considering all frames with equal importance (A) improves the quality of the embeddings.

It is worth noting that discrepancies between the modeling choices F , A, T are relatively low. This could be explained by the fact that videos from the MSVD dataset are short (10 seconds on average) and contain very few shot transitions. Thus, nearly all frames can provide a relevant visual context for associated sentences. We believe that higher differences would be exhibited for a dataset containing longer videos. In the remaining experiments, we therefore select A as the video model, since it offers a good balance between effectiveness (T ) and efficiency (F ). We study here the influence of perceptual and cluster information on the embedding space structure.

To do so, we report, in Table 2 , the structural measures on three versions of our model -M c (cluster information), M p (perceptual information) and M b (combination of both), as well as on baselines ST and P. For M and P, we discard the textual loss to isolate the effect of the different hypotheses.

As expected, solely using cluster information leads to the highest E intra and lowest E inter , which suggests that M c is the most efficient model at separating visually different sentences. Using only perceptual information in M p logically leads to highly correlated textual and visual spaces (highest ρ A vis ), but the local neighborhood structure is not well preserved (lowest E intra and highest E inter ). M b and P are optimized for both forming well-separated clusters and capturing the perceptual information within the representation space. This translates into a high E intra and low E inter . However, the main difference lies in the fact that M b is better at preserving the geometry of the visual space (higher ρ A vis ). This difference results in better performances for our model M b in terms of semantic relatedness compared to P. It reinforces our claim that both visual and perceptual information complement each other for sentence representation. Therefore, in the remaining experiments, we use the combined model M • A (α T , .1, 1), that we note M • A (α T ) for clarity's sake.

4.3. PERFORMANCE OF SCENARIOS AND BASELINES (RQ2 AND RQ3):

Table 4 .3 reports the effectiveness of the sentence embeddings obtained from our scenarios and baselines on semantic relatedness and classification tasks. We first observe that multimodal models generally outperform the text-only baseline ST on both semantic relatedness and classification benchmarks. Interestingly, we notice that the STS/Captions benchmark gives the highest discrepancies compared to the text-only baseline, probably because these sentences have a highly visual content. Second, we notice that a high α T leads to high classification scores, whereas a low α T leads to high semantic relatedness scores. There is a trade-off between semantic relatedness and classification scores, that we can set properly by tuning α T . Indeed, properly weighting the textual contribution in the global loss L is task-dependent, for every grounding model. This echoes the problem reported in Faruqui et al. (2016) in the context of word embeddings: there is no strong correlation between the semantic relatedness scores and extrinsic evaluation (e.g. classification) scores.

As a qualitative analysis, we illustrate in Table 3 that, due to our multimodal model, concrete knowledge acquired via visual grounding can be transferred to abstract sentences. To do so, we manually build abstract sentence queries using words with low concreteness (between 2.5 and 3.5) from the USF dataset (Nelson et al., 2004) . Then, nearest neighbors are retrieved from all sentences of the MS COCO training set. We see that our multimodal model is more accurate than the purely textual model to capture visual meaning, even for sentences that are not inherently visual. For example, on the first line of Table 3 , ST's sentence contradicts the query by depicting the man as "smiling", whereas M's sentence gives a concrete vision of horror: "grabs his head while screaming". The observation that perceptual information propagates from concrete sentences to abstract ones is analogous to findings made in previous research on word embeddings (Hill & Korhonen, 2014) . To further answer RQ2, we compare our model M with the projection baseline P. Our model obtains higher results than P on semantic relatedness tasks and comparable ones on classification tasks. For example, M p has 5%/3% average relative improvement over P on semantic relatedness tasks. This suggests that preserving the structure of the visual space is more effective than learning cross-modal projections, as outlined in section 4.2. Indeed, this statement is strengthened by the fact that our model also improves over a sequential state-of-the-art model (Kiela et al., 2018) . Since their textual baseline is weaker than ours (due to differences in the encoder and the dimensionality), we do not report their results in Table 4 .3. However, we compare, between both approaches, the discrepancy ∆ between the best multimodal model and the respective text-only baseline, while keeping dimensionality constant. On the benchmarks MPQA, MR, SUBJ and MSRP, our ∆ is higher than theirs. For example, ∆ Kiela et al.

MSRP:

= 0.7 and ∆ M p A (500) MSRP = 74.4 -71.6 = 2.8.

To answer RQ3, we compare joint and sequential approaches. We notice that joint models M and P globally perform better than the sequential baseline SEQ on classification and semantic relatedness tasks. For instance, M ∅ A (500) has 5%/9% average relative improvement (resp. 1%) over SEQ on semantic relatedness (resp. classification benchmarks). Therefore, the joint approach shows superior performances to the sequential one, confirming results reported for grounded word embeddings (Zablocki et al., 2018) . Finally, our models trained from scratch perform slightly better than pretrained ones. This might be due to the fact that visual and textual information are integrated in a joint manner from the beginning of training, which leads to better interactions between visual and textual modalities.

To further evaluate the quality of the embeddings, we perform cross-modal retrieval experiments on the COCO dataset (Lin et al., 2014) . In Table 5 , we report the results of our best performing models, which corroborates our previous statements on semantic relatedness and classification.

Finally, we probe that our model is independent from the choice of the textual encoder and objective L T , we use the FastSent model (Hill et al., 2016) instead of the SkipThought model. We observe similar improvements in performances (e.g. ∆ STS = 4/4 and ∆ SICK = 7/7 for the best performing model M p A (0)), confirming that our visual grounding strategy applies to any textual model.

Table 5 : Model performances on COCO cross-modal retrieval task. R@k is the recall at k metric. R@1 R@5 R@10 R@1 R@5 R@10 Sentence representations: Several approaches have been proposed over the last years to build semantic representations for sentences. On the one hand, supervised techniques produce task-specific sentence embeddings. For example, in a classification context, they are built using recurrent networks with LSTM (Hochreiter & Schmidhuber, 1997) , recursive networks (Socher et al., 2013) , convolutional networks (Kalchbrenner et al., 2014) , or self-attentive networks (Lin et al., 2017) .

On the other hand, unsupervised methods aim at producing more general and task-independent sentence representations. Closer to our contribution, SkipThought (Kiros et al., 2015) and FastSent (Hill et al., 2016) are based on the distributional hypothesis (Harris, 1954) applied to sentences, i.e. sentences that appear in similar contexts should have similar meanings. In the SkipThought model, a sentence is encoded with a GRU network, and two GRU decoders are trained to reconstruct the adjacent sentences. In FastSent, the embedding of a sentence is the sum of its word embeddings; the learning objective is to predict all words in the adjacent sentences using a negative sampling loss.

The present paper extends these works by integrating visual information.

Language grounding: To understand the way language conveys meaning, the traditional approach consists of considering language as a purely symbolic system based on words and syntactic rules (Chomsky, 1980; Burgess & Lund, 1997) . However, Fincher-Kiefer (2001); W. Barsalou (1999) insist on the intuition that language has to be grounded in the real world and perceptual experience. The importance of real-world grounding is stressed in Gordon & Van Durme (2013) , where an important bias is reported: the frequency at which objects, relations, or events occur in natural language are significantly different from their real-world frequency. Thus, leveraging visual resources, in addition to textual resources, is a promising way to acquire common-sense knowledge (Lin & Parikh, 2015; Yatskar et al., 2016) and cope with the bias between text and reality.

Following this intuition, Multimodal Distributional Semantic Models have been developed to cope with the lack of perceptual grounding in Distributional Semantic Models (Mikolov et al., 2013; Pennington et al., 2014) . Two lines of work can be distinguished. First, the sequential approach separately builds textual and visual representations and combines them, via concatenation (Kiela & Bottou, 2014; Collell Talleda et al., 2017) , linear weighted combination (Bruni et al., 2011) , and Canonical Correlation Analysis (Loeub & Reichart, 2016) . Second, the joint approach is intuitively closer to the way humans learn language semantics by hearing words and sentences in perceptual contexts. The advantage is that the visual information of concrete words is transferred to more abstract words that do not necessarily have associated visual data (Hill et al., 2014) . Closer to our contribution, Lazaridou et al. (2015) presents the Multimodal Skip-Gram model, where the Word2vec objective (Mikolov et al., 2013) is optimized jointly with a max-margin ranking objective aiming at bringing concrete word vectors closer to their corresponding visual features. Similarly, Zablocki et al. (2018) show that not only the visual appearance of objects is important to word understanding, but also their context in the image, i.e. surroundings and neighboring objects. However, these models learn word representations while our model is intended to learn sentence representations.

Very recently, Kiela et al. (2018) have set ground for multimodal sentence representations. The authors propose a sequential method: language-only representations obtained from a text corpus (Toronto BookCorpus) are concatenated to grounded sentence vectors obtained from a caption dataset (MS COCO). A LSTM sentence encoder is trained to predict the representation of the corresponding image using a ranking loss and/or to predict other captions depicting the same image. Our work is different in several ways from theirs: we use a joint approach instead of a sequential one, and we distinguish and exploit cluster and perceptual information; moreover, we use videos instead of sentences and our framework is applicable to any textual sentence representation model.

6. CONCLUSION:

In this paper, we proposed a joint multimodal model to learn sentence representations and our learned grounded sentence embeddings show state-of-the-art performances. Besides, our main findings are the following: (1) Both perceptual and cluster information are useful to learn sentence representations, in a complementary way. (2) Preserving the structure of the visual space, by modeling textual similarities on visual ones, outperforms a strategy based on projecting one space into the other. (3) A joint approach is more appropriate than a sequential method to learn multimodal representation for sentences. As future work, we would investigate the contribution of the temporal knowledge contained in videos for sentence grounding.

:



Figures and Tables:

Figure 1: Illustration of the cluster and perceptual hypotheses. Red circles indicate visual clusters.

RQ1: Validation of the semantic hypothesis by comparing video modelings on semantic relatedness. We use model M p• (0, 1, 0). Correlation results are given in the form ρ Spearman /ρ Pearson .

Table content: <table><tr><td/><td>Model</td><td colspan="4">STS/All STS/Cap. STS/News STS/For.</td><td>SICK</td></tr><tr><td>Text-only</td><td>ST</td><td>40/41</td><td>44/42</td><td>38/42</td><td>21/22</td><td>52/55</td></tr><tr><td/><td>R</td><td>51/53</td><td>62/62</td><td>39/43</td><td>23/24</td><td>56/57</td></tr><tr><td>Visual info.</td><td>F A</td><td>57/59 58/60</td><td>75/75 75/75</td><td>41/46 41/45</td><td>24/26 23/26</td><td>60/61 59/63</td></tr><tr><td/><td>T</td><td>57/60</td><td>76/76</td><td>41/46</td><td>25/27</td><td>60/63</td></tr></table>

RQ2: Influence of visual hypotheses on the structure of the representation space.

Table content: <table><tr><td/><td>ρ A v i s</td><td>E i n t r a</td><td>E i n t e r</td><td>S T S / A l l</td><td>S T S / C a p .</td><td>S T S / N e w s</td><td>S T S / F o r .</td><td>S I C K</td></tr><tr><td colspan="4">Model Structural measures</td><td/><td colspan="3">Semantic relatedness</td></tr><tr><td>Text-only Projection P Cluster Mc Perceptual Mp Both combined M b M p M p A (0, 0, 1) ST P p A (0) M p A (0, 1, 0) A (0, 0.1, 1)</td><td>16 37 38 53 46</td><td>43 63 66 44 63</td><td>25 06 01 18 02</td><td colspan="5">40/41 44/42 38/42 21/22 52/55 62/67 82/84 43/48 29/31 61/75 62/66 83/84 41/46 22/24 62/76 58/60 75/75 41/45 23/26 59/63 64/68 84/85 44/49 27/29 62/76</td></tr><tr><td colspan="8">4.2 COMPLEMENTARITY OF CLUSTER AND PERCEPTUAL INFORMATION (RQ2)</td></tr></table>

Qualitative analysis: finding the nearest neighbor of a given query in the textual space.

Table content: <table><tr><td>Query</td><td>ST</td><td>M p A (0, 0.1, 1)</td></tr></table>

RQ2,3: Semantic relatedness and classification performances. M(α T ) stands for M(α T , 0.1, 1). Note that, in all models, sentence vectors have the same dimension (2400).

Table content: <table><tr><td/><td/><td/><td>S T S / A l l</td><td>S T S / C a p . S T S / N e w s S T S / F o r .</td><td>S I C K</td><td>M S R P</td><td>M P Q A</td><td>M R S U B J</td><td>T R E C</td><td>C R</td></tr><tr><td/><td/><td>Model</td><td colspan="2">Semantic relatedness</td><td/><td/><td colspan="2">Classification</td></tr><tr><td/><td>Text-only</td><td colspan="7">ST 40/41 44/42 38/42 21/22 52/55 71.6 86.2 75.9 92.1 89.4 82.5</td></tr><tr><td>Baselines</td><td>Sequential Projection</td><td colspan="7">SEQ 47/44 70/59 37/44 29/24 58/69 70.0 86.1 75.8 92.2 89.2 81.9 A (500) 41/39 57/47 38/43 19/19 54/59 73.1 86.2 76.7 92.5 88.8 81.3 P p P p A (100) 46/46 64/60 38/43 18/18 54/61 72.9 86.4 77.0 92.6 89.8 81.0 P p A ( 10 ) 54/57 76/76 38/44 20/20 59/69 72.2 86.3 77.0 92.9 88.8 81.4 P p A ( 1 ) 58/62 81/82 40/46 22/22 62/74 71.3 86.3 76.0 92.2 89.6 81.2 P p A ( 0 ) 62/67 82/84 43/48 29/31 61/75 70.8 86.0 71.6 88.2 87.4 77.8</td></tr><tr><td/><td/><td colspan="7">M ∅ A (500) 50/51 71/69 37/43 20/19 60/70 72.2 86.3 77.2 93.0 90.4 81.0</td></tr><tr><td>Our models</td><td>From scratch Pretrained</td><td colspan="7">M ∅ A (100) 54/56 78/76 38/44 23/22 64/75 72.9 86.6 77.3 92.8 89.0 81.7 M ∅ A ( 10 ) 54/56 77/76 39/44 24/23 64/73 71.2 86.4 76.0 92.6 87.2 81.3 M ∅ A ( 1 ) 55/59 77/79 40/46 26/25 64/71 71.1 86.3 76.0 92.2 89.6 81.1 M ∅ A ( 0 ) 58/60 83/83 45/50 37/34 60/72 65.3 76.2 60.4 71.2 69.0 65.9 M p A (500) 44/43 61/54 39/44 19/19 54/60 74.4 86.1 76.3 92.6 88.8 81.3 M p A (100) 49/50 70/66 38/43 17/17 56/65 73.2 86.2 76.8 92.5 89.2 81.5 M p A ( 10 ) 56/59 79/78 38/44 19/19 61/73 72.6 86.7 76.2 92.4 88.6 81.5 M p A ( 1 ) 60/64 82/83 40/46 22/23 63/76 71.6 86.2 76.0 92.0 88.8 81.1 M p A ( 0 ) 64/68 84/85 44/49 27/29 62/76 70.1 85.9 72.7 89.6 86.8 78.0</td></tr></table>

Peer review (iclr2019):
 This paper presents a method for improving text-based sentence embeddings through a joint multimodal framework. Different from typical multimodal work, the proposed model DOES NOT project text and vision derived representations into a single space. Instead the proposed model uses a joint loss function that allows text-derived representations and the vision-derived representations to influence each other, while remaining in their separate representation spaces. In particular the loss functions ensure the following: Sentences that describe the same image should have similar text-derived representations. The similarity of text-derived representations of a pair of sentences should correlate with similarity of the representations of their corresponding images. The paper then describes multiple experiments that test whether the vision-based loss functions lead to improvements over text-only representations and if this method of incorporation vision information is better than projection based methods for multimodal representations. Strengths: 1. The problem is well motivated and the approach is interesting in that it tries to avoid a difficult problem of projecting text and visual features into a single space. These modalities don’t often have 1-1 correspondence in the kinds of information they contain. 2. The main idea can be implemented in a simple fashion using loss functions, without requiring any changes to the original models that compute the representations. Issues: The main issue I had is with respect to the empirical evaluations. In general I found the presentation hard to follow and the results don’t look convincing for an end application. Here are some follow up questions: 1. The experimental results in Table 4 seem to suggest that there is no single setting works consistently better than the projection method across all tasks. 2. Skipthought’s spearman rho numbers in [Kiros et al., 2015] is much higher (~77 whereas whats reported in Table 1 here is only 52). Is this due to simple testing of the cosine distance as opposed to training? 3. Temporal grounding discussion was unclear for me. There appears to be an attention model that somehow picks out relevant frames using the word embeddings and the frame representation. There seem to be two corresponding vectors that are used but how is this turned into a softmax weight? Is there a dot product that is missing somewhere? 4. Are the sentences associated with each frame treated separately (or) all sentences for a video simply treated as multilabels? 5. Can you argue whether the gains arise due to vision related information or because this setting simply introduces some additional training data . It would be cleaner to control for this if possible (ensuring that the text only and grounded models have same amounts of training somehow). Maybe this is a non issue because the size of the text available from the vision side (i.e. the # of sentences associated with the videos) is orders of magnitude smaller to what you get for text alone. I suspect this is the case but it would be helpful to point this out or control for it. 6. I did not understand the explanation for random frame choice . What do you mean by “even when anchors bear no perceptual semantics”? To summarize, I liked the main idea behind the paper. It is simple and avoids a tricky issue of grounding different modalities in one place. The empirical evaluation is a bit hard to follow because of missing details and the results with respect to end tasks does not look convincing.