Venue: NAACL 2016

# Vision and Feature Norms: Improving automatic feature norm learning through cross-modal maps

 Luana Bulat, Douwe Kiela and Stephen Clark

Computer Laboratory

University of Cambridge

ltf24,douwe.kiela,stephen.clark@cl.cam.ac.uk

###### Abstract

Property norms have the potential to aid a wide range of semantic tasks, provided that they can be obtained for large numbers of concepts. Recent work has focused on text as the main source of information for automatic property extraction. In this paper we examine property norm prediction from visual, rather than textual, data, using cross-modal maps learnt between property norm and visual spaces. We also investigate the importance of having a complete feature norm dataset, for both training and testing. Finally, we evaluate how these datasets and cross-modal maps can be used in an image retrieval task.

## 1 Introduction

Many cognitive theories of conceptual organisation assume that concepts are distributed representations over semantic primitives, often referred to as features or properties1(Tyler et al., 2000; Randall et al., 2004). That is, we can understand the meaning of a concept through its properties. For example, understanding the meaning of banana is closely related to understanding that it has properties such as _is a fruit, is yellow, is long, is sweet_, and knowing how these properties overlap with or differ from the properties of other concepts.

Footnote 1: Throughout the paper we will be using the terms _properties_, _features norms_ and _attributes_ interchangeably.

A number of property norm datasets, where humans were asked to list attributes of given concepts, have been collected to test this hypothesis (McRae et al., 2005; Vinson and Vigliocco, 2008; Devereux et al., 2013). After having been used to test models of conceptual representation in cognitive science for decades (Randall et al., 2004; Cree et al., 2006), these datasets have proved to be useful in a wide range of semantic NLP tasks as well, including text simplification for limited vocabulary groups. More recently, property norms have been used as a proxy for perceptual information in a number of studies on multi-modal semantics (Andrews et al., 2009; Riordan and Jones, 2011; Silberer and Lapata, 2012; Roller and Im Walde, 2013; Hill and Korhonen, 2014). Such models aim to address the grounding problem (Harnad, 1990) that distributional semantic models of language (Turney and Pantel, 2010; Clark, 2015) suffer from.

Property norms are a valuable source of semantic information, and can potentially be applied to a variety of NLP tasks, but are expensive to obtain because they involve intensive human annotation. The largest property norm dataset to date consists of just 638 concepts (Devereux et al., 2013), and the most widely cited one presents properties for only 541 concepts (McRae et al., 2005). If we are to use these datasets in large-scale semantic tasks, we would need to extend the currently available property norms by obtaining annotations for more than just a few hundred words.

The alternative to collecting more data through human annotation is to increase the coverage of property norms datasets by automatically inferring properties of concepts from easily accessible resources, such as textual data. Considering the fact that concepts, as well as their properties, are in linguistic form, the task then becomes a bootstrapping

one where we take advantage of the abundance of freely available textual corpora.

There are two strands of research that attempt to automatically obtain property norm data for new concepts. One approach is to automatically generate feature norms from text corpora by mining text data for a set of generalised property patterns (Kelly et al., 2014; Baroni et al., 2010; Barbu, 2008). Another avenue of research is inspired by Lazaridou et al. (2014) and Mikolov et al. (2013b) and tries to increase the coverage of feature norms through cross-modal mapping from linguistic information (Fagarasan et al., 2015).

Here, we follow recent trends in multi-modal semantics and explore automatic property norm extraction from visual, rather than textual, data. Obtaining property norms from visual information makes intuitive sense: information contained in the property norm datasets can often be attributed to extra-linguistic modalities--a large proportion of relevant properties are visual, auditory or tactile, rather than linguistic (e.g. _is_round, makes_noise, is_yellow_).

We show that such conceptual properties can be more accurately predicted through cross-modal mappings from raw perceptual information (i.e. image data) or multi-modal models (i.e. text and image data combined) rather than from purely textual information (Section 3). Furthermore, we analyse the quality of human collected property norm datasets and conclude that these are sparse and incomplete, meaning that there will be a lot of property annotations missing for a given concept (e.g. _has_legs_ is not listed as a property of tortoise). We show that having a complete dataset can drastically increase the performance of automatic feature prediction, resulting in a truer evaluation (Section 3.5). Lastly, we demonstrate how property norm datasets could be used in an image retrieval task (Section 4), which opens up intriguing possibilities for retrieving concepts based on their visual properties.

## 2 Property norms

Property norming studies are set up in the following way: participants are asked to freely write down a list of properties for a given concept, whilst being encouraged to consider different kinds of properties (how the concept feels, smells, what it is used for _etc_).

Besides collecting lists of properties for the concepts of interest, a number of useful property statistics are also collected during these studies. For example, the number of participants that have produced the same property for a given concept (also called _production frequency_) and the number of concepts for which a particular property is listed in the dataset (number of _concepts per feature_) have been proposed as fundamental organising principles of cognitive models (Cree et al., 2006).

One of the most widely used property norm datasets is the one collected by McRae et al. (2005), henceforth mcrae. It contains feature norms for a set of 541 concrete nouns. Each concept was seen by 30 participants and only features that were listed by at least 5 participants were recorded. The published dataset contains a total of 2526 features, with a mean of 13.7 features per concept. The numbers of features registered for a given concept range between 6 (for concepts like colander or harmonica) and 26 (for fawn). Table 1 lists some examples of properties that have been produced for banana and cello, taken from the mcrae dataset.

The largest feature norm dataset published to date was developed by the Cambridge Centre for Speech, Language and Brain (Devereux et al., 2013). It contains semantic properties for 638 concrete concepts, with 415 of these also appearing in mcrae. The data collection experiment was done similarly to McRae et al. (2005), using a production frequency cutoff of 5. The final dataset lists a total of 4359 features for the 638 concepts, with an average of 2.15 features per concept more than mcrae. Although most property norm datasets have only collected property norms for nouns, Vinson and Vigliocco

\begin{table}
\begin{tabular}{l|l} \hline \hline banana & cello \\ \hline is\_yellow, 29 & a\_musical\_instrument, 26 \\ a\_fruit, 25 & has\_strings, 16 \\ is\_edible, 13 & made\_of\_wood, 16 \\ is\_soft, 12 & found\_in\_orchestrars, 13 \\ grows\_on\_trees, 11 & is\_large, 13 \\ eaten\_by\_peeling, 10 & requires\_a\_bow, 9 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of features together with their production frequencies from mcrae 

(2008) also include verbs in their study.

All the experiments presented in this paper were conducted on mcrae. Our choice is motivated by the fact that this dataset has also been used in previous work on automated property norm prediction [14, 3], besides being one of the largest publicly-available property norm datasets.

One aspect of feature norms that previous work [15, 6, 7, 9] fails to capture is their multi-modal nature. Even though the attributes are elicited in a linguistic form, and some properties (e.g. what things look like) are easier to verbalise than others (e.g. what things smell like), these datasets contain a variety of property types, ranging from visual and auditory to encyclopaedic and behavioural. Table 2 shows some examples for each of the 10 property types as defined and annotated in mcrae. More than 25% of all features are visual (e.g. _is_yellow, is_round, made_of_metal_); hence a natural question that follows is whether images can be used in the property norm prediction task and how their performance compares to that of predicting properties from text.

## 3 Predicting feature norms from images through cross-modal mapping

Cross-modal maps represent a formalisation of the reference problem. For example, by inducing cross-modal maps between visual vectors and linguistic ones we can learn which images (represented as visual vectors) refer to which concepts (represented as text-based distributional vectors) [11]. This represents an extension of the object recognition problem, since we want to associate images with semantic representations of their depicted objects, rather than just with their label [16, 12].

The benefit of this approach lies in its generalisation power: once a function between the two semantic spaces is learnt, it can be used to see how an unseen concept relates to other concepts, just by looking at an image of that concept. This is referred to as the zero-shot learning task [13, 14]. Our task is to increase the coverage of the property norm datasets, meaning that we want to predict properties for new (unseen) concepts. For example, the concept wolf is not included in mcrae, but it would be desirable to know which of the properties in the dataset apply to it (e.g. _sanimal_, _has_4_legs_) and which don't (e.g. _a_bird, made_of_metal_).

### Building modality-specific representations

We obtain distributed representations of concepts in the property-norm semantic space (henceforth propnorm) by simply treating mcrae as a bag of 2526 properties, with the production frequencies representing the "co-occurrence counts" (Table 3).

Our visual space (henceforth visual) consists of visual representations for all the 541 concepts in mcrae, built as follows. First, we retrieve 10 images per concept from Google Images,2 following previous work [13, 14]. The image representations are then obtained by extracting the pre-softmax layer

\begin{table}
\begin{tabular}{l|c|l} \hline \hline Property type & Count & Examples \\ \hline encylopaedic & 739 & associated\_with\_vampires \\ function & 794 & used\_for\_cutting \\ smell & 7 & is\_musty, smells\_bad \\ sound & 55 & barks, produces\_music \\ tactile & 39 & is\_scaly, is\_hot, is\_soft \\ taste & 12 & is\_delicious, tastes\_sour \\ taxonomic & 207 & an\_inset, a\_vegetable \\ usual(colour) & 34 & is\_black, is\_white \\ visual(form) & 544 & has\_a\_motor, made\_of\_face \\ visual(motion) & 95 & flies, jumps, runs\_fast \\ \hline total & 2526 & - \\ \hline \hline \end{tabular}
\end{table}
Table 2: Property types and associated examples from mcrae

\begin{table}
\begin{tabular}{p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}} \hline \hline  & is\_yellow & a\_fruit & is\_edible & is\_soft \\ \hline banana & 29 & 25 & 13 & 12 \\ apple & 7 & 24 & 0 & 0 \\ bed & 0 & 0 & 0 & 13 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Subspace of propnorm. Important to note that mcrae is not complete, meaning that even though some properties are true of a given concept, they have not been produced by the human participants (e.g. the _is_edible_ property for apple holds the value 0).

 

from a forward pass in a convolutional neural network that has been trained on the ImageNet classification task using Caffe (Jia et al., 2014). We aggregate images associated with a concept into an overall visually grounded representation by taking the mean of the individual image representations. The dimensionality of the visual vectors is 4096.

We also build three linguistic spaces (distrib, svd and embed), along the lines of Fagarasan et al. (2015). distrib is a 10K-dimensional "vanilla" distributional semantic space, where the contexts are the top 10K most frequent lemmatised words (excluding stopwords) from the October 2013 Wikipedia dump. We use raw frequency counts with context windows being defined as sentence boundaries. svd is a 300-dimensional SVD-reduced version of distrib where PPMI has been applied to the raw counts. embed stands for the continuous vector representations from the log-linear skipgram model of Mikolov et al. (2013a). We used the publicly-available3 representations that were trained on part of the Google News dataset (about 100 billion words).

Footnote 3: [https://code.google.com/p/word2vec/](https://code.google.com/p/word2vec/)

We will also employ three multi-modal semantic spaces (visual+distrib, visual+svd, visual+embed), in which the visual (visual) and respective linguistic representations (distrib, svd, embed) are combined into a multi-modal representation by concatenating their respective L2-normalized representations.

### Method and evaluation

Following previous work (Fagarasan et al., 2015; Kiela et al., 2015) we use partial least squares regression (PLSR)4 to learn cross-modal maps to the property-norm space (propnorm) from the visual (visual), linguistic (distrib, svd, embed) and multi-modal semantic spaces (visual+distrib, visual+svd, visual+embed). At training time, we take advantage of the fact that we possess both visual/linguistic/multi-modal and property norm information for the concepts in mcrae. Let's consider the visual\(\rightarrow\)propnorm setting as an example. We use this cross-modal vocabulary to learn a mapping function between visual and propnorm: this function will learn to map visual dimensions to property dimensions. During testing, we use the learnt function to map the visual information of a previously unseen concept (e.g. cat) to the property norm space and obtain a _predicted property vector_ for that concept. Ideally, we want this predicted property vector to be closer to the gold-standard property vector for cat than to any other property vector (i.e. the label of its nearest neighbour in propnorm to be cat).

Footnote 4: The number of components in the linear regression was set to 100 for all experiments.

We use the standard evaluation metric for this task: average percentage correct at N (P@N) (Fagarasan et al., 2015; Lazaridou et al., 2014; Kiela et al., 2015). This measures how many of the test instances were ranked within the top \(N\) highest ranked nearest neighbors (using the cosine measure). All the results reported in Table 4 use the zero-shot learning procedure--for each of the 541 concepts in mcrae, we train a mapping on the remaining 540 concepts and record whether the correct label is retrieved among the top \(N\) neighbours--and are averaged over the entire dataset. We also compare to a random baseline, for which a concept's nearest neighbours list is obtained by randomly ranking the list of target words.

Since the cross-modal map allows us to obtain property vectors for any concept, we were also able to evaluate these semantic representations on a standard NLP task, such as the well known conceptual similarity and relatedness task. The MEN test collection (Bruni et al., 2014) contains human similarity ratings for 3000 concept pairs. Performance on this dataset is usually measured by computing the Spearman \(\rho_{s}\) correlation between the ranking produced by the similarity scores of the learnt property vectors and that produced by the human-annotated concept similarity scores. Similarity between concept pairs is calculated using cosine similarity.

For each of the semantic spaces presented in Table 5 we learn a cross-modal map to propnorm using all the concepts in mcrae at training time. During testing, we predict property vectors for all concepts in men-nouns, a subset of the MEN dataset consisting of 1285 noun pairs that don't occur in mrcae. Table 5 reports the Spearman \(\rho_{s}\) correlation of the predicted property vectors and the gold-standard relatedness scores on men-nouns (column \(\rightarrow\)propnorm), as well as the correlation of the 

original semantic spaces (e.g. distrib or svd) and the gold standard scores (column ss).

### Quantitative results

The results presented in Table 4 show that visual information is a overall better predictor of a concept's properties than linguistic information. The cross-modal maps from the visual space visual outperform all those from linguistic spaces distrib, svd, embed, and the addition of linguistic information to the visual one (maps from visual+distrib, visual+svd, visual+embed) seem to only slightly improve the performance.

It is also important to point out that, even though the P@1 numbers may appear small, similar results have been reported for other zero-shot cross-modal maps Lazaridou et al. (2014); Kiela et al. (2015). Overall results are good for higher values of N and the qualitative results (Table 6) demonstrate how well the mapping is performing.

A model will achieve a perfect score on this task if it is able to predict, for a given concept, exactly those features (and associated production frequencies) listed in mcrae. However, close-to-perfect performance in this task is impossible, since almost 30% of the features only occur with one concept, and hence can't be reconstructed for that particular concept. Consider the case of the _a_baby_deer_ property: this only occurs in the mcrae dataset as an attribute of fawn. When predicting properties of fawn as part of the zero-shot learning procedure, the _a_baby_deer_ property can't be learned, since it doesn't occur with any other concept.

Columns ss and \(\rightarrow\)propnorm in Table 5 report correlations with the men-nouns ratings. The predicted property vectors obtain a high correlation with the MEN scores, showing that the property vectors do capture lexical similarity well, although not as well as the linguistic vectors, which was expected Bruni et al. (2012). An useful finding is that in some cases, the predicted property vectors obtain a better correlation with the MEN scores than their predictors (i.e. the visual and multi-modal vectors). This shows a potential strength of the attribute-centric semantic representations: their capability to perform better on some lexical similarity/relatedness tasks than representations that contain raw perceptual information.

### Qualitative results

In order to gain more insight into the differences between the _from vision_ and _from language_ mappings, we performed two types of qualitative analysis: we looked at the differences in nearest neighbours of the predicted property-norm representations (Table 6), as well as the top predicted properties of a concept (Table 6). In the _from language_ setting we learned the mapping using the embed space, as it was the best performing linguistic space at P@1 and P@5 as shown in Table 4. We obtained the list of nearest neighbours as follows: at test time, we use the learnt cross-modal map to project the visual or linguistic representation of the unseen concept onto a property-norm representation. Using cosine similarity, we then obtain a ranked list of neighbours from all the 541 gold-standard property vectors. By in

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline From & P@1 & P@5 & P@10 & P@20 \\ \hline distrib & 1.30 & 6.88 & 16.54 & 26.58 \\ syd & 2.79 & 22.12 & 38.10 & 57.99 \\ embed & 3.90 & 23.42 & 36.80 & 55.02 \\ \hline visual & 3.35 & 28.44 & 47.96 & 64.50 \\ \hline visual+distrib & 2.60 & 23.23 & 39.41 & 56.13 \\ visual+svd & 2.97 & 28.44 & 50.74 & 65.43 \\ visual+embed & 3.16 & 28.44 & 51.12 & 65.06 \\ \hline random & 0.0 & 0.74 & 2.42 & 3.90 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Zero-shot learning performance when mapping to the property-norm space (propnorm)

\begin{table}
\begin{tabular}{c c||c|r} \hline \hline  & Semantic space (ss) & ss & \(\rightarrow\)propnorm \\ \hline  & distrib & **0.68** & 0.42 \\ Linguistic & svd & **0.68** & 0.58 \\  & embed & **0.75** & 0.69 \\ \hline Visual & visual & 0.56 & **0.60** \\ \hline Multi-modal & distrib+visual & **0.56** & 0.45 \\  & svd+visal & 0.57 & **0.60** \\  & embed+visual & 0.56 & **0.60** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance (Spearman \(\rho_{s}\) correlation) of various uni-modal and multi-modal semantic spaces (column ss), together with that of the property vectors they predict (column \(\rightarrow\)propnorm) on a semantic relatedness task (men-nouns) 

specting the nearest neighbour predictions, we can check where the unseen concept is mapped to (e.g. banana is mapped close to yellow fruits). In order to retrieve the top predicted properties of a concept, we rank the dimensions of propnorm according to the weights in the predicted property vector (e.g. the predicted property vector for banana has high weights for a_fruit and is_green when mapped from language).

By looking at the nearest neighbour predictions, we observe that, when mapping from visual input, the predicted vector will be mapped into a subspace containing visually-similar things. When mapping from linguistic input, the neighbours tend to be con

\begin{table}
\begin{tabular}{c c} \hline \hline
**Concept** & **Nearest neighbours (from vision)** \\ \hline banana & **banana**, lemon, corn, pear, graprefuit, pineapple \\ cabbage & lettuce, asparagus, spinach, celery, broccoli, cucumber \\ crocodile & alligator, **crocodile**, frog, turtle, iguana, taad \\ cello & violin, guitar, banjo, harp, harpischord, **cello**, flute \\ drum & pot, pan, coin, skillet, bucket, peg, cap\textunderscore(bottle) \\ fox & **forx**, cougar, cyoote, deer, mink, elk, chipunk \\ harpoon & sword, machete, **harpoon**, dagger, rifle, knife, gun \\ muzzle & donkey, horse, ox, dog, cat, goat, cow \\ pants & jeans, trousers, **pants**, shirt, blouse, jacket, coat \\ prune & plum, blueberry, nectarine, peach, tangerine, raisin \\ rice & cauliflower, turnip, pie, **rice**, cabbage, biscuit, plate \\ stool & **stool**.(**furtheriture**), table, peg, chair, gate, desk, door \\ swan & pelican, goose, dove, seagull, partridge, raven, falcon \\ tortoise & turtle, **tortotise**, crocodile, alligator, otter, frog, walrus \\ worm & eel, rattlesnake, **worm**, shrimp, bat\textunderscore(baseball), python \\ \hline \hline
**Concept** & **Top predicted features (from language)** \\ \hline banana & is\_yellow, is\_black*, is\_round, is\_long, a\_fruit \\ cabbage & is\_green, a\_vegetable, is\_edible, eaten\_in\_salads \\ crocodile & is\_green, an\_animal, lives\_in\_water, beh\textunderscore\(b\)\(b 

cepts that are semantically related or denoted by words that occur in the same context as the target concept (e.g. worms are found in plums and nectarines).

A notable result is that when mapping from vision, the top neighbours tend to share the same colour (top neighbours for banana are yellow fruits, for swan are white birds) or shape as the target concept (top neighbours for worm are long things with no legs). One possible clue as to why vision is better at predicting a concept's properties is given by the fact that it obtains better results on concepts such as pants or stool, where the only difference to very similar concepts like trousers or chair are visual (a stool has no backtext as opposed to a chair).

However, there are cases in which the visual attributes of an object are not very useful in predicting its most important features: e.g. drum is mapped into a subspace of round objects (from vision), and not instruments (from language).

Besides the difference in top predicted features, Table 6 also indicates a shortcoming of mcrae, specifically that this is not complete, meaning that not all properties that apply to a given concept were produced by the human annotators. Most of the top predicted attributes that don't occur in the dataset (those marked with * in Table 6) are highly plausible properties for the given concepts: _tastes\(\_\)sweet_ for banana or _has\(\_\)legs_ for tortoise. This also means that the model is being unfairly penalised.

In order to obtain a _complete_ version of mcrae, every possible (concept, _property_) pair would have to be checked for validity and annotated accordingly depending on whether _property_ is a valid attribute of concept.

### Importance of complete data

We were interested in measuring the impact that a _complete_ dataset of features would have on the performance of the cross-modal zero-shot learning task. Silberer et al. (2013) conducted a study using a subset of the concepts and properties in mcrae, whereby every property was annotated if it was a plausible attribute of the concept.

The published dataset (silberer) consists of visual attribute annotations for 512 concepts (that also occur in mcrae ) and 693 visual properties. The annotation was done on a per-concept basis by looking at 10 images retrieved from ImageNet Deng et al. (2009) and selecting all the attributes that were considered to be generally true for the given concept, even if not depicted in the images. For example, _has\(\_\)pit_ is a valid visual attribute for plum, even though not all retrieved images of plans show the pit.

Since not all of the 693 visual properties covered in silberer can be found in mcrae, we will only be concerned with the subset of silberer which contains only those visual properties that also occur in mcrae, henceforth silb-vis. These datasets are _complete_, since they were exhaustively annotated as explained above.

Let us also define m-vis as the subset of mcrae that contains the 512 concepts listed in silberer and the 283 properties that are common to silberer and mcrae, together with their production frequencies as in mcrae. This will act as our _incomplete_ dataset. Table 7 lists all the aforementioned datasets, together with statistics related to their number of concepts, features and concept-feature pairs. It also demonstrates the sparsity of mcrae: it contains fewer (concept, _property_) pairs than silberer, even though it contains 4 times more properties.

All the experiments performed using these

\begin{table}
\begin{tabular}{l|r|r|r} \hline \hline Dataset & \#concs & \#props & \#(conc,prop) pairs \\ \hline silberer & 512 & 693 & 7743 \\ silb-vis & 512 & 283 & 5335 \\ m-vis & 512 & 283 & 2140 \\ \hline mcrae & 541 & 2526 & 7259 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of various datasets, according to the number of concepts and properties covered, as well as the pairs of (concept, _property_) contained

\begin{table}
\begin{tabular}{l|l|r|r|r|r} \hline \hline Train & Test & P@1 & P@5 & P@10 & P@20 \\ \hline M-vis & M-vis & 0.59 & 7.91 & 15.02 & 19.97 \\ M-vis & silb-vis & 7.11 & 27.67 & 43.68 & 56.92 \\ silb-vis & silb-vis & 5.93 & 35.77 & 54.74 & 71.54 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Zeroshot learning performance for the vision to norms cross-modal map on different training and test sets 

datasets are listed in Table 8. These are identical in methodology to the zero-shot cross-modal maps from visual to propnorm; the only difference being the datasets that these are run on.

The row (train:m-vis, test:m-vis) represents the setting where the cross-modal map learning and testing are both done on the incomplete set of data, just like we would do using mcrae. We notice a huge improvement in performance by using the complete data only at test-time (row (train:m-vis, test:silb-vis)). Note that, in this scenario, the learning is carried out in the same way, but the model can't be penalised for ranking plausible features near the top during test time, since we are testing against a complete dataset. This new setting provides a truer evaluation scenario and demonstrates the weakness in using mcrae as a test set.

Performance improves even more if the complete dataset is used at training time as well (the row (train:silb-vis, test:silb-vis)), showing the benefit of also learning the mapping from complete data, as well as evaluating on it.

## 4 Property based query engine

An interesting question follows from the good performance of the cross-modal mapping in Section 3, and that is whether we can reliably predict what concepts look like based on their semantic properties. For example, does something that flies, has wings and a leak look like a bird?

This task could be formalised as a property-based query engine, where we can train the cross-modal mapping to learn which concepts refer to which images. We follow the same experimental setup as detailed in Section 3.2 in order to learn a cross-modal map from propnorm to visual. We also learn cross-modal maps from the linguistic spaces

Figure 1: Nearest neighbours of the predicted visual vectors



(distrib, svd, embed) to visual in order to see whether conceptual properties or linguistic input are better at predicting visual information.

Table 9 shows the results of our quantitative evaluation: the average percentage of correctly retrieved mean visual representations at N.

A qualitative analysis of the propnorm to visual cross-modal map is shown in Figure 1. Because there are no images associated with the predicted mean visual representation, we retrieve and display the top neighbouring images. These images look surprisingly good, considering that the representation for taxi in propnorm is a sparse vector where only the features _is_yellow, requires_drivers, used_for_transportation, a_car, requires_money, found_in_New_York, is_expensive, used_for_passengers, a_cab, is_fast are activated.

## 5 Conclusions

We have studied the automatic prediction of property norms for unseen concepts, through learning the cross-modal mapping from image data. Following previous work, we evaluated on a zero-shot learning task and show that raw visual information (images) is a better predictor for conceptual properties than linguistic input (text). We also presented a short case study demonstrating the importance of having complete annotations in the property norm datasets, for both testing and training. Lastly, we demonstrated a possible use case for property norm datasets in an image retrieval task.

Our contributions are two-fold: first, we show that property norms can be successfully predicted from non-linguistic modalities and secondly, we quantify the need to have _complete_ property norm datasets, where a production frequency of 0 for a (concept, _property_) pair can always be interpreted as "_property_ is not true of concept_".

### Acknowledgments

LB is supported by an EPSRC Doctoral Training Grant. DK is supported by EPSRC grant EP/I037512/1. SC is supported by ERC Starting Grant DisCoTex (306920) and EPSRC grant EP/I037512/1. We thank the anonymous reviewers for their helpful comments. A link to the data used for the experiments in this paper is available at [http://www.cl.cam.ac.uk/~ltf24/](http://www.cl.cam.ac.uk/~ltf24/).

## References

* [Andrews et al.2009] Mark Andrews, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. _Psychological review_, 116(3):463.
* Barbu [2008] Eduard Barbu. 2008. Combining methods to learn feature-norm-like concept descriptions. In _Proceedings of the ESSLLI Workshop on Distributional Lexical Semantics_, pages 9-16.
* Baroni et al. [2010] Marco Baroni, Brian Murphy, Eduard Barbu, and Massimo Poesio. 2010. Strudel: A corpus-based semantic model based on properties and types. _Cognitive Science_, 34(2):222-254.
* Bergsma and Goebel [2011] Shane Bergsma and Randy Goebel. 2011. Using visual information to predict lexical preference. In _RANLP_, pages 399-405.
* Bruni et al. [2012] Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_, pages 136-145. Association for Computational Linguistics.
* Bruni et al. [2014] Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. _J. Artif. Intell. Res.(JAIR)_, 49(1-47).
* Clark [2015] Stephen Clark. 2015. Vector space models of lexical meaning. _Handbook of Contemporary Semantics-second edition. Wiley-Blackwell_.
* Creee et al. [2006] George S Creee, Chris McNorgan, and Ken McRae. 2006. Distinctive features hold a privileged status in the computation of word meaning: Implications for theories of semantic memory. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 32(4):643.
* Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_, pages 248-255. IEEE.
* Devereux et al. [2013] Barry J Devereux, Lorraine K Tyler, Jeroen Geertzen, and Billi Randall. 2013. The centre for speech, language and the brain (cslb) concept property norms. _Behavior research methods_, pages 1-9.
* Fagarasan et al. [2015] Luana Fagarasan, Eva Maria Vecchi, and Stephen Clark. 2015. From distributional semantics to feature norms: grounding semantic models in human perceptual data. In _Proceedings of the 11th International Conference on Computational Semantics_, pages 52-57, London, UK, April. Association for Computational Linguistics 

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. 2013. Devise: A deep visual-semantic embedding model. In _Advances in Neural Information Processing Systems_, pages 2121-2129.
* Harnad (1990) Stevan Harnad. 1990. The symbol grounding problem. _Physica D: Nonlinear Phenomena_, 42(1):335-346.
* Hill and Korhonen (2014) Felix Hill and Anna Korhonen. 2014. Learning abstract concept embeddings from multi-modal data: Since you probably cant see what i mean. _Proceedings of EMNLP. ACL_.
* Jia et al. (2014) Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In _Proceedings of the ACM International Conference on Multimedia_, pages 675-678. ACM.
* Kelly et al. (2014) Colin Kelly, Barry Devereux, and Anna Korhonen. 2014. Automatic extraction of property norm-like data from large text corpora. _Cognitive Science_, 38(4):638-682.
* Kiela and Bottou (2014) Douwe Kiela and Leon Bottou. 2014. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In _Proceedings of EMNLP_, volume 2014.
* Kiela et al. (2015) Douwe Kiela, Luana Bulat, and Stephen Clark. 2015. Grounding semantics in olfactory perception. In _Proceedings of ACL_, volume 2, pages 231-6.
* Lazaridou et al. (2014) Angeliki Lazaridou, Elia Bruni, and Marco Baroni. 2014. Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1403-1414, Baltimore, Maryland, June. Association for Computational Linguistics.
* McRae et al. (2005) Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. _Behavior research methods_, 37(4):547-559.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Mikolov et al. (2013b) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. _arXiv preprint arXiv:1309.4168_.
* Palatucci et al. (2009) Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. 2009. Zero-shot learning with semantic output codes. In _Advances in neural information processing systems_, pages 1410-1418.
* Randall et al. (2004) Billi Randall, Helen E Moss, Jennifer M Rodd, Mike Greer, and Lorraine K Tyler. 2004. Distinctiveness and correlation in conceptual structure: behavioral and computational studies. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 30(2):393.
* Riordan and Jones (2011) Brian Riordan and Michael N Jones. 2011. Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation. _Topics in Cognitive Science_, 3(2):303-345.
* Roller and Im Walde (2013) Stephen Roller and Sabine Schulte Im Walde. 2013. A multimodal lda model integrating textual, cognitive and visual modalities. _Seattle, Washington, USA_, pages 1146-1157.
* Silberer and Lapata (2012) Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 1423-1433. Association for Computational Linguistics.
* Silberer et al. (2013) Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2013. Models of semantic representation with visual attributes. In _ACL (1)_, pages 572-582.
* Socher et al. (2014) Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. _Transactions of the Association for Computational Linguistics_, 2:207-218.
* Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. _Journal of artificial intelligence research_, 37(1):141-188.
* Tyler et al. (2000) Lorraine K Tyler, HE Moss, MR Durrant-Peatfield, and JP Levy. 2000. Conceptual structure and the structure of concepts: A distributed account of category-specific deficits. _Brain and language_, 75(2):195-231.
* Vinson and Vigliocco (2008) David P Vinson and Gabriella Vigliocco. 2008. Semantic feature production norms for a large set of objects and events. _Behavior Research Methods_, 40(1):183-190.