Venue: EMNLP 2017

# Deriving continous grounded meaning representations from

referentially structured multimodal contexts

Sina Zarriess &David Schlangen

Dialogue Systems Group // CITEC // Faculty of Linguistics and Literary Studies

Bielefeld University, Germany

{sina.zarriess,david.schlangen}@uni-bielefeld.de

###### Abstract

Corpora of referring expressions paired with their visual referents are a good source for learning word meanings directly grounded in visual representations. Here, we explore additional ways of extracting from them word representations linked to multi-modal context: through expressions that refer to the _same_ object, and through expressions that refer to _different_ objects in the _same scene_. We show that continuous meaning representations derived from these contexts capture complementary aspects of similarity, even if not outperforming textual embeddings trained on very large amounts of raw text when tested on standard similarity benchmarks. We propose a new task for evaluating grounded meaning representations--detection of potentially co-referential phrases--and show that it requires precise denotational representations of attribute meanings, which our method provides.

## 1 Introduction

Various routes for linking language to extralinguistic context have been explored in recent years. A lot of research has looked at integrating visual representations, either directly (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013; Yu et al., 2016; Schlangen et al., 2016) or through mapping into a multi-modal distributional space (Feng and Lapata, 2010; Bruni et al., 2012; Kiela and Bottou, 2014; Lazaridou et al., 2015). Young et al. (2014) have explored a less direct link, by representing the extension of phrasal expressions as sets of images, and deriving from this a precise notion of denotational similarity. In very recent work, Cocos and Callison-Burch (2017) use spatial context from geo-located tweets to induce word embeddings that capture situational similarity between lexical items.

In this paper, we explore an approach that combines aspects of several of these paths. Starting point is the observation that corpora of exophoric referring expressions provide richly structured contexts that go beyond just linking individual expressions with their denotations. As an example consider the scene in Figure 1 depicting several referents and corresponding referring expressions produced by different speakers. This scene provides a learner not only with an example of a referent for the word _lady_, it also provides the information that _lady_ can co-refer with _girl_, and that its denotations can spatially / situationally co-occur with those of _table_ and _cake_. From these types of information we infer word embeddings, following the method from Levy and Goldberg (2014) for training embeddings on arbi

Figure 1: Dimensions of context in referential, visually grounded language, and similarity relations that can be derived from it, image from MsCoco (Lin et al., 2014)

trary non-linear context, and we show that these capture complementary aspects of word similarity that purely textual induction methods conflate. We also show that these representations handle a more directly referential similarity task better.

## 2 Word Embeddings from Multi-Modal Referential Contexts

We base our study on the ReferIt and RefCoo corpus Kazemzadeh et al. (2014); Yu et al. (2016) building upon image collections by Grubinger et al. (2006) and Lin et al. (2014); for the latter, we also use referring expressions collected by Mao et al. (2015). This corpus gives us visual scenes containing sets of objects, \(s=o_{1},\dots,o_{n}\). Each object is associated with a set of referring expressions \(r_{1},\dots,r_{m}\); and we use a standard method (a ConvNet) for providing a visual representation \(vis_{i}\) for it. Each referring expression, in turn, is defined as a linear sequence of words \(r_{i}=w_{1}\ldots w_{k}\). In the following, we structure this context into four dimensions--visual, textual, situational and denotational--which we use to derive different word embeddings.

### Textual Context (txt)

We learn standard distributional word embeddings from our corpus, ignoring extra-linguistic context. We train a skip-gram model Mikolov et al. (2013) with negative sampling with window width 5, 300 dimensions. For comparison, we also use the textual word embeddings provided by Baroni et al. (2014), trained on a much larger web corpus (5-word context window, 10 negative samples, 400 dimensions). We distinguish the two textual embeddings using the subscripts txt\({}_{ref}\), xt\({}_{web}\).

### Visual Grounding (vis)

Given a set of referring expressions containing the word \(w\) and their corresponding referent \((o_{j},r_{j}),w\in r_{j}\), we can derive a visual context for the word \(w\) by averaging over the visual representations of its referents \( vis_{j}\), as proposed for instance by Kiela and Bottou (2014). The visual context of a word can be seen as a 'visual prototype'. We derive representations of our visual inputs with a convolutional neural network, "GoogLeNet" Szegedy et al. (2015), that was trained on data from the ImageNet corpus Deng et al. (2009), and extract the final fully-connected layer before the classification layer, to give us a 1024 dimensional representation of the region. Following Schlangen et al. (2016), we also add 7 features that encode information about the region relative to the image, the full representation hence is a vector of 1031 features. Each word is then represented as the average over its visual vectors.

### Situational Grounding (sit)

We also train word embeddings (dim. 300) that predict words paired with their situational context, following the method by Levy and Goldberg (2014). This captures similarities between words occurring for different objects in the same scene, e.g. _cake_ in the context of _table_ in Figure 1. Given a pair of referring expressions \((r_{i},o_{i}),(r_{j},o_{j}),o_{i}\neq o_{j}\), \(r_{i}\) and \(r_{j}\) are co-situational expressions. Thus, for a word \(w_{i}\in r_{i}\), we consider all words \(w_{j}\in r_{j}\) as its situational context. In practice, we compute situational contexts only for the head nouns of each referring expression, as we expect situational similarities to be useful for capturing similarities between nouns.

### Denotational Grounding (den)

As our data typically records multiple co-referential expressions for an object (3 expressions on average in the refoco data), we define the denotational context based on sets of expressions referring to the same object \((r_{1},o_{i})\dots(r_{n},o_{i})\). For a word \(w_{i}\in r_{i}\), we consider all words \(w_{j_{l}}\) (with \(w_{j_{l}}\in r_{j}\)) as denotational context, where \(r_{j}\) and \(r_{i}\) refer to the same object. When two words occur in a denotational context, we have strong evidence that they are semantically compatible, i.e. can refer to the same objects as _girl_ and _lady_ in Figure 1 do. Similar to our training procedure for situational embeddings, we now learn 300-dimensional word embeddings that predict occurrences of a word based on co-referential contexts, pairing each word with all words from referring expressions describing the same object.

## 3 Word Similarity and Relatedness

We now have four different continuous representations for words; in the following, we evaluate them for how well they predict semantic relations.

SimilarityWe evaluate on some similarity data sets, reporting Spearman \(\rho\) correlations between human ratings and cosine similarities for word vectors. We use the MEN Bruni et al. (2012) and 

Silberer and Lapata (2014)'s data with semantic (SemSim) and visual similarity (VisSim) ratings.

CompatibilityAs generic semantic similarity judgements are known to be "fuzzy" Faruqui et al. (2016), we also evaluate on Kruszewski and Baroni (2015)'s benchmark on semantic compatibility. They define two words as being _semantically compatible_ "if they can potentially refer to the same thing". We expect our denotational and visual embeddings to be highly useful for this task. We report unsupervised results obtained from cosine similarities between word embeddings.

Hypernym DirectionalityWe adopt an evaluation procedure by Kiela et al. (2015) on hypernym pairs in the BLESS data set Baroni and Lenci (2011). Given a general (e.g. 'animal') and a concrete noun (e.g. 'dog') that stand in the hypernym relation, the task is to identify the noun that is more general. Lazaridou et al. (2015) found that the generality or concreteness of a noun's meaning is reflected in the entropy of its embedding, and we adopt that measure for our purposes. Thus, we compute entropies of our word embeddings and report accuracies corresponding to the proportion of noun pairs where the entropy of the more general noun is higher than the more concrete noun.

VocabularyWe intersect the vocabularies covered by the different embeddings, which amounts to 1960 words in total. We restrict evaluation to the corresponding word pairs in the above data sets, coverage is reported in Table 1.

ResultsAs shown in Table 1, the performance of embeddings learned on referring expression corpora are generally below state-of-the-art distributional vectors trained on large web corpora. However, some interesting tendencies can be observed by comparing embeddings learned from different context dimensions. Denotational embeddings in isolation provide a precise representation of meaning that outperforms the other types of embeddings on semantic similarity judgements in MEN and SemSim, and detects hypernym directionality most accurately. An interesting exception is the compatibility data set where visual embeddings clearly outperform textual and denotational embeddings. Situational embeddings perform less well than textual and denotational embeddings but, interestingly, are similar in performance to visual embeddings on semantic similarity, suggesting that visual and situational similarity seem to be equally important aspects of general semantic similarity. Concatenation of denotational and textual embeddings yields the best results for correlations with human similarity judgements. This is expected as denotational similarity is probably too restricted for generic semantic similarity. We experimented with further embedding combinations, but only the fusion of the textual and denotational dimension outperformed the embeddings obtained from a particular grounding dimension.

Table 2 shows correlations on cosine similarities on all word pairs from MEN, SemSim, VisSim and Compatibility between our word embeddings. This further corroborates the finding that different dimensions of grounding lead to complementary notions of similarity. In particular, correlation between visual and situational embeddings is relatively low, as compared to more fuzzy textual embeddings which correlate well with denotational embeddings. For a qualitative analysis, more examples are shown in Appendix A.

Qualitative DiscussionTable 3 illustrates similarities learned from different grounding dimensions by means of some qualitative examples. Whereas denotational and visual embeddings rank semantically compatible words on top (e.g. _grassgrassy_), situational embeddings clearly focus more on topical similarity (_grass-clouds_). Given these examples, the finding that visual embeddings outperform denotational embeddings on the semantic compatibility task (see Table 1) seems rather contradictory. A preliminary error analysis suggests that the compatibility ratings that humans provide 'out of context' in a rating task differ

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Model & MNen & SemSim & VisSim & Compat. & Hyp.Dir. \\ \hline _# pairs_ & _989_ & _2041_ & _2041_ & _4843_ & _334_ \\ \hline vis & 0.404 & 0.469 & 0.427 & **0.241** & 78.14 \\ Txt\(ref\) & 0.550 & 0.584 & 0.484 & 0.230 & 55.69 \\ den & 0.646 & 0.583 & 0.491 & 0.163 & **81.14** \\ sit & 0.470 & 0.468 & 0.371 & 0.134 & 59.58 \\ den\(|\mathrm{txt}_{ref}\) & **0.654** & **0.632** & **0.531** & 0.207 & 79.94 \\ \hline txt\(web\) & 0.799 & 0.708 & 0.578 & 0.262 & 90.42 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Word similarity and relatedness evaluation

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Model & txt & men & sit & vis \\ \hlinetxt & 1 & 0.60 & 0.45 & 0.30 \\ den & 0.60 & 1 & 0.45 & 0.35 \\ sit & 0.45 & 0.35 & 1 & 0.26 \\ vis & 0.30 & 0.35 & 0.26 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Model correlations 

to some extent from referential choices in our corpus. As an example, in the compatibility data set, the words _pigeon_ and _mother_ are rated as being equally similar to _animal_. However, in our corpus of referring expressions, _mother_ is never used to refer to animal entities and our denotational embeddings predict them to be highly dissimilar, whereas visual embeddings are slightly more robust in this case.

More generally, textual embeddings learned from referring expressions captures a much more fuzzy and generic notion of similarity than denotational, visual or situational embeddings, e.g. _grass_ is similar to _shrubs_ and to _sand_ in the textual space. This fuzziness has been found for word embeddings trained on large amounts of raw text as well Faruqui et al. (2016).

## 4 Approximate Co-Reference Detection

Another important testbed for models of lexical meaning is their ability to capture semantic inference, with textual entailment as a well-known paradigm: here the task is to predict whether a textual hypothesis \(h\) can be inferred from a given premise \(p\)Dagan et al. (2006). Young et al. (2014) have proposed a less strict variant of this called "approximate textual entailment". The main idea is that premise and hypothesis candidates can be automatically extracted from a corpus of captioned images. Given a set of captions known to describe the same image and an hypothesis, the task is to determine whether the hypothesis can describe the same image as the premise.

Inspired by this approach, we use the multi-modal corpus of referring expressions to set up a new task for evaluating word embeddings, which consists of capturing approximate inferential relations between referring expressions. Thus, in our case, the hypothesis and the premise are expressions referring to objects, and the task is to determine whether they could (potentially) refer to the same object. Note that this is also similar to the notion of semantic compatibility proposed by Kruszewski and Baroni (2015), but extended to phrasal expressions. We can automatically extract positive and negative pairs from the data (see Section 2) by looking at pairs of expressions referring to objects in the same image and distinguishing **coreferential expressions** referring to the same entity (e.g. _grandma - old lady_), and **non-coreferential expressions** referring to different entities, e.g. _old lady - young lady_. In contrast to the majority of existing similarity and relatedness benchmarks which are centered around nouns, this task requires precise meaning representations for attribute-like words (e.g. _left-right_, _old-young_) which occur frequently in our data and which are frequently used to distinguish between objects occurring in the same situation. In particular, as the scenes in our data sets contain many objects of the same category (e.g. in the RefCoo data), the distinction can often not be made by looking at the noun only, e.g. for classifying '_old_ lady' - '_young_ lady_' as non-coreferential.

We call this task _approximate_ coreference detection as the premise and hypothesis might describe complementary aspects of the same object such that the distinction cannot be made perfectly without the original perceptual context. For instance, in some cases, _lady in blue_ and _young lady_ might denote the same referent, in others not (see Figure 1). Thus, we note that the upper bound for automatic (or human) performance in this task is clearly not 100%. In future work, we plan to combine this with a reference resolution system that grounds the expressions in a given image.

Data and Set-upGiven an image with several objects and a set of expressions referring to these, we compute the set of expression pairs \(P\) for that image. This set now divides into positive instances, i.e. expressions that both refer to the same object in the image, and negative instances, i.e. expressions that describe distinct entities in the scene. As this gives us a lot of data, we adopt a supervised learning approach for modeling the task of approximate co-reference detection. Thus, we use our embeddings to extract a range of similarity

\begin{table}
\begin{tabular}{c|c|c} woman & txt\({}_{ref}\) & lady, girl, man, chick \\  & den & lady, girl, women, blues \\  & sit & gird, guy, man, lady \\  & vis & lady, girl, women, chicken \\ \hline sidewalk & kt\({}_{ref}\) & pavement, ground, walkway, steps \\  & den & street, sidejukak, walkway, pavement \\  & sit & buhlid, bldg, lamppost, street \\  & vis & pavement, street, walkway, concrete \\ \hline grass & kt\({}_{ref}\) & shrubs, dirt, bushes, sand \\  & den & grassy, patch, bounded, plains \\  & sit & clouds, church, trees, building \\  & vis & grassy, path, shrubs, bushes \\ \hline couch & kt\({}_{ref}\) & sofa, chair, bench, bed \\  & den & sofa, pillows, cushions, loveseat \\  & sit & sofa, leather, armchair, seater \\  & vis & sofa, pillow, pillows, love \\ \end{tabular}
\end{table}
Table 3: Top nearest neighbours for some example noun embeddings 

measures between the expression pairs and feed these metrics as features into a classifier, trained to predict whether two phrases co-refer. This set-up is largely similar to Young et al. (2014)'s evaluation setting for approximate textual entailment.

Similarity MeasuresGiven a pair \(P\) of expressions \(r_{i}=w_{i_{1}}\dots w_{i_{n}},r_{j}=w_{j_{1}}\dots w_{j_{m}}\), we extract pairwise cosine similarities between the embeddings \(cos(w_{i_{x}},w_{j_{y}})\), using average (\(\sum_{(w_{i},w_{j})\in P}cos(w_{i},w_{j})\times\frac{1}{|P|}\)), maximum (\(\max_{(w_{i},w_{j})\in P}cos(w_{i},w_{j})\)) and minimum distance (\(\min_{(w_{i},w_{j})\in P}cos(w_{i},w_{j})\)) as features for classification. Furthermore, we restrict the words in each expression such that they are disjunct sets excluding words that occur in both expressions, \(w_{i}\neq w_{j},\forall(w_{i},w_{j})\in P\). We extract the same average, maximum and minimum distance measures on these lexically disjunct expressions. Finally, we compose word embeddings for each expressions via addition (\(r_{i}=w_{i_{1}}+\ldots+w_{i_{n}}\)) and add the cosine between the composed embeddings (\(cos(r_{i},r_{j})\)) to our list of features. Here, we compare textual, visual and denotational embeddings, as our situational embeddings only cover nouns.

TrainingFrom ReferIt, we extract 161K training and 18K test pairs, dividing into 66% non-coreferential and 34% coreferential expressions. We re-train our embeddings on the training portions of this data. We only consider non-coreferential expressions that refer to objects of the same type, according to their label annotated in the data set. From RefCoco, we extract 300k pairs from the training set and 95k pairs from the test set, dividing into roughly 70% non-coreferential and 30% coreferential expressions. We randomly sample these pairs, the overall number of possible pairs in refcoco exceeds 2 million. We train a binary logistic regression classifier on each corpus, given the similarity measures extracted for each word embedding.

ResultsWe report accuracies on co-referential expression detection in Table 4, on ReferIt and RefCoco. Similarities derived from denotational embeddings clearly outperform the other classifiers on both data sets, including state-of-the-art textual embeddings learned on a much larger web corpus. On RefCoco, only denotational embeddings lead to a clear improvement over the majority baseline. While the low performance of standard distributional embeddings is rather expected on this task (see previous findings on e.g. predicting antonyms Nguyen et al., 2016)), the clear advantage of denotational over visual embeddings is noteworthy. Whereas visual grounding is relatively effective for modeling compatibility between nouns (see Table 1), it does not seem to capture attribute meaning accurately as illustrated in Table 5. Here, the average of all visual objects referred to as e.g. _small_ seems to be rather noisy and lead to high similarity with rather random words (_directly_) whereas denotational embeddings model accurate compatibility relations between e.g. _small-smaller_.

## 5 Conclusion

Whereas it is notoriously difficult to tailor or specialise distributional meaning representations inferred from text to particular aspects of semantic relatedness Kiela et al. (2015a); Nguyen et al. (2016); Rimell et al. (2017), this work has shown that a multi-modal corpus of referring expressions can be used to derive a range of continuous meaning representations grounded in different aspects of context, capturing different notions of similarity. As compared to visual embeddings used in previous works, we found that denotational embeddings are particularly useful for detecting semantic relations. Other, recently proposed tasks related to modeling word association Vulic et al. (2017), commonsense knowledge Vedantam et al. (2015) or child-directed input Lazaridou et al. (2016) provide interesting testbeds for future work.

\begin{table}
\begin{tabular}{l c l} \hline \hline  & ReferIt & RefCoco \\ \hline Majority & 66.05 & 71.64 \\ \hline vis & 70.14 & 71.63 \\ TX\({}_{ref}\) & 68.49 & 71.57 \\ DEN & **73.67** & **74.32** \\ \hline TX\({}_{web}\) & 69.16 & 71.89 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracies for co-referential expression detection

\begin{table}
\begin{tabular}{c|c|p{142.3pt}} top & tUt\({}_{ref}\) & upper, bottom, bottom, bottem \\ den & upper, topmost, tippy, above \\ vis & upper, above, of, corner \\ \hline red & text & yellow, purple, maroon, blue \\  & den & maroon, redman, reddish, allmiddle \\  & vis & and, purple, yellow, pink \\ \hline small & tUt\({}_{ref}\) & large, smaller, big, tiny \\  & den & smaller, smallest, little, littiest \\  & vis & directly, of, between, slightly \\ \hline \hline \end{tabular}
\end{table}
Table 5: Top nearest neighbours for some example adjectives embeddings 



## Acknowledgments

We acknowledge support by the Cluster of Excellence "Cognitive Interaction Technology" (CITEC; EXC 277) at Bielefeld University, which is funded by the German Research Foundation (DFG).

## References

* Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In _ACL (1)_, pages 238-247.
* Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In _Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics_, pages 1-10. Association for Computational Linguistics.
* Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. 2012. Distributional semantics in technicolor. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 136-145. Association for Computational Linguistics.
* Cocos and Callison-Burch (2017) Anne Cocos and Chris Callison-Burch. 2017. The language of place: Semantic value from geospatial context. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 99-104. Association for Computational Linguistics.
* Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognizing tectual entailment_, pages 177-190. Springer.
* Deng et al. (2009) Jia Deng, W. Dong, Richard Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In _CVPR09_.
* Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In _Proc. of the 1st Workshop on Evaluating Vector Space Representations for NLP_.
* Feng and Lapata (2010) Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 91-99. Association for Computational Linguistics.
* Grubinger et al. (2006) Michael Grubinger, Paul Clough, Henning Muller, and Thomas Deselaers. 2006. The IAPR TC-12 benchmark: a new evaluation resource for visual information systems. In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006)_, pages 13-23, Genoa, Italy.
* Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. 2014. ReferItGame: Referring to Objects in Photographs of Natural Scenes. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2014)_, pages 787-798, Doha, Qatar.
* Kiela and Bottou (2014) Douwe Kiela and Leon Bottou. 2014. Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP-14)_.
* Kiela et al. (2015a) Douwe Kiela, Felix Hill, and Stephen Clark. 2015a. Specializing word embeddings for similarity or relatedness. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2044-2048, Lisbon, Portugal. Association for Computational Linguistics.
* Kiela et al. (2015b) Douwe Kiela, Laura Rimell, Ivan Vulic, and Stephen Clark. 2015b. Exploiting image generality for lexical entailment detection. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 119-124, Beijing, China. Association for Computational Linguistics.
* Krishnamurthy and Kollar (2013) Jayant Krishnamurthy and Thomas Kollar. 2013. Jointly learning to parse and perceive: Connecting natural language to the physical world. _Transactions of the Association for Computational Linguistics_, 1:193-206.
* Kreuszewski and Baroni (2015) German Kruszewski and Marco Baroni. 2015. So similar and yet incompatible: Toward the automated identification of semantically compatible words. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 964-969, Denver, Colorado. Association for Computational Linguistics.
* Lazaridou et al. (2016) Angeliki Lazaridou, Grzegorz Chrupala, Raquel Fernandez, and Marco Baroni. 2016. Multimodal semantic learning from child-directed input. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 387-392, San Diego, California. Association for Computational Linguistics.
* Lazaridou et al. (2015) Angeliki Lazaridou, Nghihe The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 153-163, Denver, Colorado. Association for Computational Linguistics.



Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 302-308, Baltimore, Maryland. Association for Computational Linguistics.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollf, and C.Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _Computer Vision ECCV 2014_, volume 8693, pages 740-755. Springer International Publishing.
* Mao et al. (2015) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2015. Generation and comprehension of unambiguous object descriptions. _ArXiv/ CoRR_, abs/1511.02283.
* Matuszek et al. (2012) Cynthia Matuszek, Nicholas Fitzgerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012. A Joint Model of Language and Perception for Grounded Attribute Learning. In _Proceedings of the International Conference on Machine Learning (ICML 2012)_.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proceedings of NIPS 2013_, pages 3111-3119, Lake Tahoe, Nevada, USA.
* Nguyen et al. (2016) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for synonym distinction. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 454-459, Berlin, Germany. Association for Computational Linguistics.
* Rimell et al. (2017) Laura Rimell, Amandla Tabona, Luana Bulat, and Douwe Kiela. 2017. Learning to negate adjectives with bilinear models. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 71-78. Association for Computational Linguistics.
* Schlangen et al. (2016) David Schlangen, Sina Zarriess, and Casey Kennington. 2016. Resolving references to objects in photographs using the words-as-classifiers model. In _Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (ACL 2016)_, page To appear.
* Silberer and Lapata (2014) Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 721-732, Baltimore, Maryland. Association for Computational Linguistics.
* Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In _CVPR 2015_, Boston, MA, USA.
* Vedantam et al. (2015) Ramakrishna Vedantam, Xiao Lin, Tammay Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Learning common sense through visual abstraction. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 2542-2550.
* Volic et al. (2017) Ivan Volic, Douwe Kiela, and Anna Korhonen. 2017. Evaluation by association: A systematic study of quantitative word association evaluation. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 163-175. Association for Computational Linguistics.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockennaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2:67-78.
* Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. _Modeling Context in Referring Expressions_. Springer International Publishing.