Understanding Mental Representations Of Objects Through Verbs Applied To Them
Authors: Ka Chun Lam
Francisco Pereira
Maryam Vaziri-Pashkam
Kristin Woodard
Emalie McMahon
Venue:  (2021)
Keywords: Affordance
affordance embedding
object representation

Abstract: In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and the extent to which they rely or have an effect on the properties of the object. This knowledge is called the object "affordance". We propose an approach for creating an embedding of objects in an affordance space, in which each dimension corresponds to an aspect of meaning shared by many actions, using text corpora. This embedding makes it possible to predict which verbs will be applicable to a given object, as captured in human judgments of affordance, better than a variety of alternative approaches. Furthermore, we show that the dimensions learned are interpretable, and that they correspond to typical patterns of interaction with objects. Finally, we show that the dimensions can be used to predict a state-of-the-art mental representation of objects, derived purely from human judgements of object similarity.

Under review as a conference paper at ICLR 2021
UNDERSTANDING MENTAL REPRESENTATIONS OF OBJECTS THROUHH VERBS APPLIED To THEM
Anonymous authors Paper under double-blind review
ABSTRACT
In order interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and the extent to which they rely or have an effect on the properties of the object. This knowledge is called the object "affordance". We propose an approach for creating an embedding of objects in an affordance space, in which each dimension corresponds to an aspect of meaning shared by many actions, using text corpora. This embedding makes it possible to predict which verbs will applicable to given object, as captured in human judgments of affordance, better than variety of alternative approaches. Furthermore, we show that the dimensions learned are interpretable, and that they correspond to typical patterns of interaction with objects. Finally, show that the dimensions can be used to predict a state-of-the-arr mental representation of objects, derived purely from human judgements of object similarity.
INTRODUCTION
In order to interact with objects in our environment, we rely on an understanding of the actions that can be performed on them, and their dependence (or effect) on properties of the object. Gibson (1979) coined the term "affordance" to describe what the environment "provides or furnishes the animal" TM Norman (2013) developed the term to focus the properties of objects that determine the action possibilities. The notion of "affordance" emerges from the relationship between the properties of objects and human actions. If we consider "object" as meaning anything concrete that one might interact with in the environment, there will be thousands of possibilities, both animate and inanimate (see WordNet (Miller 1998)) The same is true if we consider "action" as meaning any verb that might be applied to the noun naming an object (see VerbNet (Schuler 2005)). Intuitively, only relatively small fraction of all possible combinations of object and action will be plausible. Of those, many will also be trivial, e.g. "see" or "have" may apply to almost every object. Finally, different actions might reflect a similar mode of interaction, depending on the type of object they are applied to (e.g. "chop" and "slice" are distinct actions, but they are both used in food preparation).
Mental representations of objects encompass many aspects beyond function. Several studies (McRac al. 2005] Devereux et al. 2014 Hovhannisyan et al. (020) have asked human subjects to list binary properties for hundreds of objects, yielding thousands of answers. Properties could be taxonomic (category), functional (purpose), encyclopedic (attributes), or visual-perceptual (appearance), among other groups. While some properties were affordances in themselves (e.g. "edible"), others reflected many affordances once (e.g. "is vegetable" means that it could be planted, cooked, sliced, etc). More recently, Zheng et al. (2019): Hebart et al.] (2020) introduced SPoSE, a model of the mental representations of objects. The model was derived from a dataset of 1.5M Amazon Mechanical Turk (AMT) judgments of object similarity, where subjects were asked which of a random triplet of objects was the odd one out. The model was an embedding for objects where each dimension was constrained to be sparse and positive, and where triplet judgments were predicted as function of the similarity between embedding vectors of the three objects considered. The authors showed that these dimensions were predictable as a combination of elementary properties in the Devereux et al. (2014) norm that often co-occur across many objects. Hebart et al. (2020) further showed that 1) human subjects could coherently label what the dimensions were "about", ranging from categorical (c.g. is animate, food, drink, building) to functional (e.g. container, tool) or structural (e.g. made of metal or wood, has inner structure). Subjects could also predict what dimension values objects would

---

Under review as a conference paper at ICLR 2021
have, based on knowing the dimension value for a few other objects. SPoSE is unusual in its wide coverage 1,854 objects - and in having been validated in independent behavioral data.
Our first goal is to produce analogous affordance embedding space for objects, where each dimension groups together actions corresponding to a particular "mode of interaction" affoodancce mining. Our second goal is to understand the degree which affordance knowledge underlies the mental representation of objects, as instantiated in SPoSE. In this paper, we will introduce and evaluate an approach for achieving both of these goals. Our approach is based on the hypothesis that, if a set of verbs apply to the same objects, they apply for similar reasons. We start by identifying applications of action verbs to nouns naming objects, in large text corpora. We then use the resulting dataset to produce an embedding that represents each object as a vector in a -ww-dmmnniional space, where each dimension groups verbs that tend to be applied to similar objects. We do this for larger lists of objects and action verbs than previous studies (thousands in each case). Combining the weights on each verb assigned by various dimensions yields a ranking over verbs for each concept. We show that this allows us to predict which verbs will be applicable to a given object, as captured in human judgments of affordance. Further, we show that the dimensions learned are interpretable, and they group together verbs that would all typically occur during certain complex interactions with objects. Finally, we show that they can be used to predict most dimensions of the SPoSE representation. in particular those that categorical or functional. This suggests that affordance knowledge underlies much of the mental representation of objects, in particular semantic categorization.
interactionaaffforaance
2 RELATED WORK
The problem of determining, given an action and an object. whether the action can apply to the object was defined as "affordance mining" Chao et al. (2015). The authors proposed complementary methods for solving the affordance mining problem by predicting plausibility score for each combination of object and action. The best method used word co-occurrences in two ways: n- gram counts of verb-noun pairs, similarity between verb and noun vectors in Latent Semantic Analysis (Deerwester et al., 1990) or Word2Vee (Mikolov et al. 2013) word embeddings. For evaluation, they collected AMT judgements of plausibility ("is it possible to verb > object >") for every combination of 91 objects and 957 action verbs. The authors found they could retrieve small number of affordances for each item, but precision dropped quickly with higher recall. Subsequent work (Rubinstein et al. 2015 Lucy & Gauthier] [017]] [tsumi] 2020) predicted properties of objects in the norms above from word embeddings (Mikolov et al. [213]] Pennington et al.) 2014). albeit without a focus on affordances. Forbes et al. (2019) extracted 50 properties (some were affordances) from [Deeereux et al. (0114), for set of 514 objects, to generate positive and negative examples for 25,700 combinations. They used this data to train a small neural network to predict these properties. The input to the network was either the product of the vectors for object and property, using word embeddings (Mikolov et al. 2013 Pennington et al. 2014 Levy & Goldberg 22014) or the representation of a synthesized sentence combining them, if using contextualized embeddings (Peters)
et al. 2018 Devlin et al. 2018 They found that the latter outperformed the former for property prediction, but none allowed reliable affordance prediction. In addition to object/action plausibility prediction, Ji et al. (2020) addressed the problem of determining whether a object jatject//ctioon/bjeee22 (target of the action with objectl) was feasible. They selected a set of 20 actions from Chao et al.) (2015) and combined them with the 70 most frequent objects in ConceptNet (Speer & Havasi (2012) into 1400 object/action pairs, which were then labelled as plausible or not; given rater disagreements, this yielded 330 positive pairs and 1070 negative ones. They then combined the positive pairs with other objects as "tails" (recipients of the action), yielding 3900 triplets. They reached F1 scores of 0.81 and 0.52 on the two problems, respectively. Other papers focus on understanding the relevant visual features in objects that predict affordances, e.g. (Myers et al.) 2015] Sawatzky et al. 2017 Wang & Tarr 2020). The latter collected affordance judgments on AMT ("what can you do with < obbject >" for 500 objects and harmonized them with WordNet synsets for 334 action verbs. For validation of the rankings of verb applicability predicted by our model, we will use the datasets from
Chao et al. (2015) and Wang & Tarr (2020), as they the largest available human rated datasets.
In robotics research, affordance refer to relation between agent, action and the environment, under the constraints of motor and sensing capabilities of the agent Lopes et al. 2007). Affordance modeling for robotics have been studied extensively, see recent surveys (Jamone et al. 2016 Zech et al.
2017 Hassanin et al. 2018). Due to the restriction in action possibilities and the complexity of
2

---

Under review as a conference paper at ICLR 2021
real world scenarios, poor semantic generalization of affordance inference is observed in visual and speech-based robotic systems. Attempts to use semantic role labelling or text corpora, in tandem with
visual features, have been proposed (c.g Persiani & Hellström (2019): Chen et al. (2020). In general,
though, this work focuses on a restricted set of objects and manipulation actions.
In computational linguistics, (Resnik 1993) introduced computational approaches determining selectional preference, the degree to which particular semantic class tends to be used as an argument to given predicate. From this perspective, the method we introduce in this paper provides way of scoring verbs for how well they would apply to objects. All methods proposed to do this take
advantage of some grouping of verbs and objects into classes (e.g. WordNet in Resnik (9966.. or
no-occurrence statistics of words in a corpus (Erk) 2007 Padó et al.) 2007] Séaghdha 2010 Van
De Cruys 2014 Zhang et al. 2020 Erk 2007 used similarities of co-occurrence patterns for
words to compute selectional preferences for semantic roles in FrameNet. (aaó et al. (0077 used
the same similarity function to predict the plausibility of verb/relaaton//agumenn triples. Seaghdha
(2010) learned a custom topic model with separate latent variables (and dictionaries) for verbs and
nouns. Van De Cruys (2014) trained a neural network to predict preference scores for combinations
of verbs or objects, represented via embedding vectors. Zhang et al.. (2020) learns embeddings for individual words together with modifications for when the word is used in a certain relation. It scores combinations of words by similarity of the modified embedding vectors. All of these methods could be used to score verbs by how applicable they are to given noun, the ancillary task we use to make sure our proposed embedding carries the relevant information. That said, they would require substantial implementation effort, even where the code is publicly available. This said, making that prediction is not our main goal. Our proposed embedding space is latent variable model for verb-noun applications. While this is also the case for the dual topic model in (Séaghdha (2000) the internal representation in Van De Cruys] 2014). or the embeddings of nouns/verbs in /Zhang et al.) (2020). they would all require extensive modification to add sparsity assumptions important for interpretability- - and to produce combined verb rankings for embedding vectors.
DATA AND METHODS
3.1 OBJECTS AND ACTIONS CONSIDERED
In this paper, we use the list of 1854 object concepts introduced in (Hebart et al., 2019) and for which SPoSE embeddings are available. This list sampled from concrete, picturable, and nameable nouns in American English, and was further expanded by a crowdsourced Amazon Mechanical Turk study to elicit category membership ratings. The following 27 categories accounted for most of the objects: food, animal, clothing, tool, drink, vehicle, fruit, vegetable, body part, toy, container, bird, furniture, sports equipment, musical instrument, dessert, part of car, weapon, plant, insect, kitchen tool, office supply, clothing accessory, kitchen appliance, home decor, medical equipment, and electronic device. As we are not doing sense disambiguation for each noun that names an object, we will use noun or object interchangeably throughout the paper. We created our own verb list by having three annotators go through all verb categories on VerbNet (Schuler) (005)) and selecting those that included verbs that corresponded an action performed by a human on an object. We kept all the verbs in each selected category, and selected only those categories where all annotators agreed. Those VerbNet categories contained 10 50 verbs sharing thematic roles and selectional preferences (c.g. fill-9.8, amalgamate-22.2. manner-speaking-37. 37.3, build-26.1, 1, remove-100 10.1, cooking-45.33 create-26.4, destroy-44, mix-22.1, vehicle-51.4.11 dress-41.1.1), The resulting list has 2541 verbs.
3.2 EXTRACTION OF VERB APPL.CCTIIONS TO NOUNS FROM TEXT CORPORA
We used the UKWaC and Wackypedia corpora Ferraresi et al.) containing, approximately. 2B and 1B tokens, and 88M and 43M sentences, respectively. The former the result of crawl of British web pages, while the latter is a subset of Wikipedia. Both have been extensively cleaned up and have clearly demarcated sentences, which makes them ideal for dependency parsing. We replaced all common bigrams in Brysbaert et al. (2014) by a single token. We identified all sentences containing both verbs and nouns in our list, and we used Stanza (Qi et al. 2020) to produce dependency parses for them. We extracted all the noun-verb pairs in which the verb was syntactic head of a noun having obj (object) or n.uu i : paass s (passive nominal subject) dependency relations. We compiled raw counts of how often each verb was used on each noun, producing count matrix M. Note that
3

---

Under review as a conference paper at ICLR 2021
this is very different from normal co-occurtence counts those would register count whenever verb and noun were both present within a short window (e.g. up to words away from each other), regardless of whether the verb applied to the noun, they were simply in the same sentence. Out of the 1854 nouns considered, there were 1755 with at least one associated action, and this is the list we will use in the remainder of this paper. Note also that the counts pertain to every possible meaning of the noun, given that no word sense disambiguation was performed.
Finally, we converted the matrix M into Positive Pointwise Mutual Information (PPMI (Trnny & Pantel 2010)) matrix P where, for each object i and verb k:
P(Mx)) k(i,k) :- max (1og P(M(,)) P(M.))) 9).
(1)
where P(M,.) and P(M.x) are respectively the marginal probability of and k. can be viewed as pair-pattern matrix (Lin & Pantel 2001), where the PPMI helps in separating frequency from informativess of the co-occurrence of nouns and verbs (Turney & Pantel 2010; Turney & Littman 2003). However, PPMI is also biased and may be large for rare co-occurrecess, e.g. for (o,, Ub)) that co-occur only once in M. This is addressed in the process described in the next section.
3.3 OBJECT EMBEDDING IN VERB USAGE SPACE
Object embedding via matrix factorization Our embedding is based on factorization of the PPMI matrix P (m objects by n verbs) into the product of low-rank matrices O (m objects by d dimensions) and V (n verbs by d dimensions), yielding P == OVT z P. O is the object embedding in d-dimensional space, and V is the weighting placed on each verb by each dimension.
Intuitively, if two verbs occur often with the same objects, they will both have high loadings on one of the d-dimensions; conversely, the objects they occur with will share high loadings on that dimension. The idea of factoring a count matrix (or a transformation of it) dates back to Latent Semantic Analysis (Landauer & Dumais 1997). and was investigated by many others (Tumney & Pantel 2010) is good review). Given that PPMI is biased towards rare pairs of noun/verb, the matrix P is not necessarily very sparse. If factorized into a product of two low-rank matrices, however, the structure of the matrix can be approximated while excluding noise or rare events (Bullinaria & Levy 2012).
Optimization problem Given that PPMI is positive, the matrices O and V are as well. This allows us obtain them through a non-negative matrix factorization (NMF) problem
O", V argmin|| P 1 OVT 3R(O.V).
(2)
o,v
which can be solved through an iterative minimization procedure. For the regularization R(0, V), chose the sparsity control RRO, V) Eig Oig Eij Vij.
In our experiments, hold-out cross validation (Kanagal & Sindhwani] 2010). due to its scalability and natural fit to the multiplicative update algorithm for solving Q Specifically, denoting M, and M, be the mask matrices representing the held-in and held-out entries, we optimize for
d = 70 and 3 0.3. These values were found using the two-dimensional
O", V argmin| M, @ (P- OV BR(O.V) O,V
and obtain the reconstruction error details, please refer to Appendix(A A The optimization problem (2) is NP-hard and all state-of-the-ar algorithms may converge only to a local minimum (Gillis) 4); choosing a proper initialization of O and V is crucial. We used the NNDSVD initialization (Boutsidis & Gallopoulos/ 2008). SVD-based initialization which favours sparsity on O and V and approximation error reduction.
I,, O (P- O'(V')T) + BR(O", V*). For more
Estimating the verb usage pattern for each object Each column V. of matrix V contains a pattern of verb usage for dimension k. which captures verb co-occurrence across all objects. Deriving a similar pattern for each object i, given its embedding vector O,,- = [o,, oi..... Ozyll requires combining these patterns based on the weights given each dimension. The first step in doing so
4

---

Under review as a conference paper at ICLR 2021
requires computing the cosine similarity between each embedding dimension O., and the PPMI values P.A. for each verb k in the approximated PPMI matrix P = OVT, which is
S(O.,, P.,)) = O.h P.A
(4)
|10,
Given the embedding vector for object i, Oz,: = (oig Oigs ... Osgl: we compute the pattern of verb usage for the object as S. Thus, this is weighted sum of the similarity between every O, and P.a.. We will refer the ordering of verbs by this score as the verb ranking for object i.
4 EXPERIMENTS AND RESULTS
4.1 PREDICTION OF AFFORDANCE PLAUSIBILITY
Affordance ranking task The first quantitative evaluation of our embedding focuses on the ranking of verbs as possible affordances for each object. We will use the Affordance Area Under The Curve (AAUC) relative to datasets that provide, for each object, a set of verbs known (or likely) to be affordances. Intuitively, the verb ranking for object i is good if it places these verbs close to the top of the ranking, yielding an AAUC of 1. Conversely, a random verb ranking would have an AAUC of 0.5, on average. Note that this is a conservative measure, given that a perfect ranking would still penalize every true affordance not at the top. Hence, this is useful as relative measure for comparing between our and competing approaches for producing rankings. More formally, given the K ground truth verb affordances (ou = of object i, and its verb ranking {v: ) we denote Eg to be the index such that
VE, = gb Yk. We then define AUCC for object as AUCC = K ΣK (1 4x)
Datasets We used the two largest publicly available object affordance datasets as ground truth. In the first dataset, WTAction (Wang Tarr 2020), objects are associated with the top actions label provided by human annotators in response to "What can you do with this object?". Out of 1,046 objects and 445 actions in this dataset, there are 971 objects and 433 verbs that overlap with those in our lists (~ 3.12 action labels per object) The second dataset, MSCOCO (Chao et al. 2015), scores every candidate action for an object ranging from 5.0 ("definitely an affordance") to 1.0 ("definiteyy not an affordance") marked by 5 different workers. We consider only 5.0 score as being an affordance, whereas Chao et al. (2015) used both 4.0 and 5.0. Out of 91 objects and 567 actions, there are 78 objects and 558 verbs that overlap with ours (~~ 34 action labels per object).
Baseline methods We compared the ramking of verbs produced by our algorithm with an alternative proposed in (Chao al., 2015): : ranking by the cosine similarity between word embedding vectors for each noun and those for all possible verbs in the dataset. We considered several off off-th embedding alternatives, namely Word2 Vec (Mikolov et al. (2013), 6B token corpus), Glo Ve (Pen nington et al. (2014), 6B and 840B token corpora, and our 2B corpus), Dependency-Basee Word Embedding (DBWE, Levy & Goldberg 2014), 6B corpus), and Non-negative Sparse Embedding (NNSE Murphy al. (0112). 16B corpus). Finally, included the other two methods in (Chao) ct al. 2015). LSA (Deerwester et al. 1990), trained on our corpus), and ranking by frequency of verb/noun pair in Google N-grams (Lin al. (012 The emheddings are 300-D in all cases, except for NNSE (1000-D, results are similar for 2500-D). We contrasted Word2 and Glo because they are based two different embedding approaches (negative sampling and decomposition of word co-occurrence matrix), developed on corpora twice as large as ours. We contrasted 6B and 840B versions of GloVe to see the effect of increasing dataset size, and 2B to show the effect of our corpus. In these methods, the co-occurrence considered is simply proximity within a window of a few tokens, rather than application of the verb to the noun. We included DBWE because it uses dependency parse information (albeit to define the word co-occurrence window, rather than select verb applications to nouns as we do). We included NNSE because it based on a sparse, non-negative factorization of a co-occurrence matrix of features derived from words combined with specific dependency relations. Finally, we ranked the verbs by their values in the row of the PPMI matrix P corresponding to each probed object, to see the effect of using low-rank approximation in extracting information.
Results For each dataset, we reduced our embeddings O and V according to the sets of objects and verbs available. We then obtained the verb ranking for each object, described in Section 3.3
5

---

Under review as a conference paper at ICLR 2021
Table Average AAUC of verb rankings produced by our and bascline methods
Method
NNSE (1000-D)
W2V
GloVe
Gl
Gl
Datasct
DBWE
N-gram
(6B)
(28)
(6B)
(8408)
LSA
PPMI
Ours
WTaction
14
0.65
0.60
0.70
0.79
0.75
0.81
0.77
0.88
SR
6e-3
8e-3
8e-3
7e-3
Te-3
7e-3
Ge-3 3
5e-3
6e-3
4e-3
MSCOCO
0,56
0.58
0.53
0.59
0.66
0.64
0.68
0.63
0.61
0.77
$g
8e-3
9c-3
8c-3
7e-3
7e-3
8e-3
6e-3
7e-3
8e-3
55-3 3
as well as rankings predicted with the different baseline methods in the previous section. Table ! shows the AAUC results (in terms of the mean μ and the standard error SE of AAUCs) obtained with these verb rankings on the two datasets. Our ranking is better than those of all the baseline methods, as determined from paired widdddded t-test, in both WTAction (p-values of 7.36e- 23, 4.19e-14 14, 6.82e- 174, 1.34e- 53, 1.44e-888 89, 5.21e-44 47, 8.700- 25 and 8. 399 and MSCOCO (p-values of 1.54e 23, 3.37e 21, 9.32e 36, 2.15e-20 27, 2.82e 30, 9, 64e 20, 7.93e 17 and 2. 2.13e-20 29).
The overall distributions of AAUCs for the two datasets are shown in Appendix B Our procedure yields AAUC closer to 1.0 for many more items than the other methods. This suggests that the occurrence of verb and noun within a window of a few tokens, the basis of the word embeddings that we compare against, carries some information about affordances, but also includes other relationships and noise (e.g. if the verb is in one clause and the noun in another). Results are better in the embedding trained in a corpus 280X larger, but still statistically worse than those of our procedure. Ranking based on the PPMI matrix P performs at the level of the 6B token embeddings. This suggests that our procedure is effective at removing extraneous information in the matrix P.
4.2 PREDICTION OF SPOSE OBJECT REPRESENTATIONS FROM OUR AFFORDANCE EMBEDDING
The SPoSE representation and dataset The dimensions in the SPoSE representation (Hebart) ctt al. 2020) are interpretable, in that human subjects coherently label what those dimensions are "about" ranging from the categorical (e.g. animate, building) the functional (e.g. can tie, can contain, flammable) or structural (e.g. made of metal or wood, has inner structure). Furthermore, subjects could also predict what dimension values new objects would have, based on knowing the dimension value for few other objects. The SPoSE vectors for objects are derived solely from behaviour in a "which of a random triplet of objects is the odd one out" task. The authors propose hypothesis for why there is enough information in this data to allow this: when given any two objects to consider, subjects mentally sample the contexts where they might be found or used. The resulting dimensions reflect the aspects of the mental representation of an object that come up in that sampling process. The natural question is, then, which of these dimensions reflect affordance or interaction information, and that is what experiments aim to answer.
For our experiments, we used the 49-D SPoSE embedding published with (Hebart et al. 2020). Out of these, excluded objects named by nouns that had no verb co-occurrences in our dataset and, conversely, verbs that had no interaction with any objects. We averaged the vectors for each set of objects named by the same polysemous noun (c.g. "bat"). This resulted in a dataset of 1755 objects/nouns, with their respective SPoSE embedding vectors, and 2462 verbs.
Prediction of SPoSE meaning dimensions from affordance embedding The first experiment carried out was to predict the SPoSE dimensions for object from its representation in our affordance embedding. The ability to do this tells us which SPoSE dimensions - and, crucially, which kinds of SPoSE dimensions can be explained in terms of affordances, to some extent.
Denoting the SPoSE embeddings for m objects as a m x 49 matrix Y, we solved the following Lasso regression problem for each column Y.
arg
mii
2m
||Y,, Ow 1
Allwl,, i 1,,,,... 1,2, 49,
(5)
where À was chosen based 2-Fold cross-validation, with À in [10e-7, 10c with log-scale spacing. Since both Y.,, and our embedding O represent object features by positive values, we restricted w 2 0. Intuitively, this means that we try to explain every SPoSE dimension by combination of the presence of certain affordance dimensions, not by trading them off. Figure T shows the predictions Y = Ow?

---

well (in either Under review as a conference paper at ICLR 2021
Regressed Embudsing
madeolmatal
plat
wondh
Absolitte reaidues
100
Lvou the aommena
Figure 1: Prediction of first 10 SPoSE embeddings from affordance embeddings (top) versus actual SPoSE embeddings (middle). Objects are grouped by semantic category (those with 2 15 objects). The absolute residuals of the prediction also shown (bottom). Color range is fixed across plots to show the magnitude of residuals. For full diagram with all dimensions, see Figure 7 ii Appendix C
true values Y,, and the absolute residues [ Y.al for each SPoSE dimension i. For clarity, objects are grouped by their semantic category and only plotted for categories with size 2 15. The visual resemblance of the patterns, and the range of correlations between true and predicted dimensions (top: 0.84, mean: 0.46), see Figure D for distribution), indicate that the affordance embedding contains enough information to predict most SPoSE dimensions.
Relationship between SPoSE and affordance dimensions We first considered the question of whether affordance dimensions correspond directly to SPoSE dimensions, by looking for the best match for each the latter terms of correlation. As shown in Figure a)), most SPoSE dimensions have one more affordance dimensions that are similar to them. However, when consider the cross-validated regression models to predict SPoSE dimensions from affordance dimensions, every model places non-zero weight on several of the latter. Furthermore, those predictions much more similar to SPoSE dimensions than almost any individual affordance dimension. Figure (((b) plots, for every SPoSE dimension, the correlation with its closest match (x-axis) versus the correlation with the cross-validated prediction of the regression model (y-axis). The best predicted dimensions are categorical, e.g. "animal", "plant", "tool", or functional, e.g. "can tie" or "flammable" Structural dimensions are also predictable, e.g. "made of metal", "made of wood", or "paper", but appearance-rellee related dimensions, e.g. "colorful pattern", "craft", "degree of red".. less so.
What can explain this pattern of predictability? Most SPoSE dimensions can be expressed as linear combination of affordance dimensions, where both the dimensions and regression weights are non- negative. This leads to sparse regression model since dependent variables cannot be subtracted to improve the fit where, on average, 5 affordance dimensions have 80% of the regression weight. Each affordance dimension, in turn, corresponds to a ranking over verbs. Figure 3 shows the top 10 verbs in the 5 most important affordance dimensions for predicting the "animal" SPoSE dimension. As each affordance dimension loads verbs that correspond to coherent modes of interaction (e.g. observation, killing, husbandry), the model is not only predictive but also interpretable. Whereas we could also dense embeddings predict SPoSE dimensions, they do work as
7

---

Under review as a conference paper at ICLR 2021
Correlation between SPoSE vectors matched affordance
glood
Pelliciiing
0.6
Correlation: with Regression
osject thouse)
au Saluat
body part
Correiation
0.4
string ho annnseaaaacc
2gg vulerr Viiiiba"
0.
Appearance Categorical Functional
Structural
"222 Rell ceneece match affordance
0.0
Correlation: with Best Match
(a) Correlation with outcomes
(b) Comparison
Figure 2: Correlation between each SPoSE dimension and the corresponding best match in our affordance embedding (correlation values shown in right vertical axis). b) For each SPoSE dimension, the relationship between the similarity with the best matching affordance dimension (X-axis) and the similarity with the cross-validated prediction of the regression model for it (y-axis).
accuracy interpretabiliity, see Figure 3 for GloVe). If we consider the top 5 verbs from affordance
(a)
23.51%
Zim 241
4.14%
Proposed
Bim
ein
13 64%
Glove
Din 82-
2.75%
3..0%
-ente reee ecouks FIrk, beerdir
Cheea podta
Din
54
7.02%
animal
chice Batk grain ailuanee
zkm 360
2.11%
animal
Tard de emdts, oierrat
coMES mar operata potthass
imnp.
Figure 3: Top 10 verbs in the most important affordance dimensions (proposed affordance embed- ding versus GloVe 840B) for predicting the "animal" SPoSE feature.
dimensions that are most used in predicting each SPoSE dimension, we see that "tool" has "sharpen, blunt, wield, plunge, thrust" (D50); "animal" has "spot, observe, sight, hunt, watch" (D3), "hunt, kill, cull, exterminate, chase" (D37), or "pasture, herd, slaughter, milk, tether" (D18): "food" has "serve, eat, cook, prepare, order" (D15), or "bake, leaven, ice, eat, serve" (D64); "plant" shares D2 with "food", but also has "cultivate, grow, plant, prune, propagate" (D57). The full list of affordance dimensions most relevant for predicting each SPoSE dimension is in Appendix (Feendd
13}.
These results suggest that SPoSE dimensions are predictable insofar they can be expressed as combinations of modes of interaction with objects. Using the approach in Section 3.3 we produced a combined ranking over verbs for each SPoSE dimension. We replaced the embedding O in { with
the SPoSE prediction Y and ranked the verbs for dimension h according S(Ý.,, Px). Table 2 shows, for every SPoSE dimension, ranked by predictability, the top 10 verbs in its ranking. Testing against null hypothesis of 0 correlation, the p-values obtained range from 0.0 to the maximum of 7.6le-7 for the SPoSE dimension "furry", indicating the statistical significance of correlation with our regression outcomes. Examining this table provides empirical confirmation that, first and foremost, highly predictable categorical dimensions correspond to very clean affordances. The same true for functional dimensions, e.g. "can tie" or "container" or "flammable": even though they are not "classic" categories, subjects group items belonging to them based on their being suitable for purpose (e.g. "fasten", "fill", or "burn"). But why would this hold for structural dimensions? One possible reason is that objects having that dimension overlap substantially with a known category (e.g. "made of metal" and "tool"). Another that the structure drives manual or mechanical affordance "granulated"). Finally, what the affordances for appearance dimensions that can be predicted? Primarily, actions on items in categories that share that appearance, e.g. "textured" is shared by fabric items, "round" is shared by many fruits vegetables. Prediction is worse when
(e.g. "elongated"
8

---

Under review as a conference paper at ICLR 2021
the items sharing the dimension come from many different semantic categories ((Hebart al. 2019 lists the pictures of items shown to subjects).
Pearson comelation
Dimension label
Taxonomy
Affordances (Top Ten Ranked Verbs)
0.84
animal
calegorical
kill, spot, hunt, observe, chase, feed, slaughter, sight, trip, find serve, cat, cook, prepare, taste, consume, add, mix, stir, order wear, don, match. knit, sew, fasten, rip, embroider, tear, model grow, cultivate, plant, add, cat, chap. galber, cut, dry, prume fit, invent, manufacture, incorporate design, position, attach, utilize, carry, install wield, grab, hold, carry. sharpen, swing, hand, pick, clutch, throw
0.82
food
categorical
0.75
wearable
categorical
0.71
plant
categorical
0.67
made metal
structural
0.61
tool
categorical
0.57
can tie
functional
fasten, tighten, unfasten, undo, attach, thread, lie, secure, loosen, loose contain, mix, scatter, add, gather, remove, sprinkle, dry, deposit, shovel light, extinguish, ignite, throw, carry. flash, kindle, place, manufacture, douse remove, place, hang, tear, stain, spread, weave, clean, drape, wrap grow, cultivate, pick, add, slice, place, eat, chop, throw, plant place, remove, carry. incorporate, design, contain, hring, construct, manufacture, find empty, fill, carry, place, clean, load, bring, dump, unload, leave grab, carry. wield, hold, pick, place, throw, hand, bring, drop
0.54
granulated
structural
0.48
flammable
functional
0.47
textured
appearance
0.44
round
appearance
0.40
made wood
structural
0.40
container
functional
0.38
clongated
structural
0.24
colorful pattern
appearance appcarance
manufacture, buy, design, place, remove, sell, invent, purchase, contain, bring place, bring. remove, design, hang. call, buy. pat, pull, manufacture lit, incorporatie, remove, place, design, munufacture, install, position, clean, attach place, call, add, contain, remove, find, buy, hring. introduce, sell
0.23
craft
permeable
structural
0.18
degree of red
appearance
Table 2: Affordance assignment for selection of SPoSE dimensions mentioned in the text, ordered by how well they can be predicted from the affordance embedding. Dimension labels are simplified. The full table, p-values and descriptions for each dimension are in Appendices D and E respectively.
CONCLUSIONS
In this paper, we introduced an approach to embed objects in space where every dimension corresponds to pattern of verb applicability to those objects. We view such pattern as very broad extension of the classical notion of "affordance", obtained by considering verbs that go well beyond motor actions, and objects that encompass many different categories beyond tools or household objects. We showed that this embedding can be learned from text corpus and used to rank verbs by how applicable they would be to given object. We evaluated this prediction against two separate -enerrl-purpose
human judgment ground truth datasets, and verified that our method outperforms word embeddings trained on much larger text corpora. This gave us with confidence that the patterns of applicability we identified captured information relevant to human judgments of affordance.
We used our embedding predict SPoSE dimensions for objects. SPoSE is embedding that was derived from human behavioural judgements of object similarity, and has interpretable dimensions. We used the resulting prediction models to probe the relationship between the dimensions of our embedding and the various types of SPoSE dimensions. This allowed us to conclude that our "affordance" embedding knowledge predicts 1) category information, 2) purpose, and 3) some structural aspects of the object. SPoSE dimensions to do with visual appearance were poorly predicted. Our embedding is thus sufficient for predicting most SPoSE dimensions. To be able to go further this, and conclude that our embedding is a valid model of the mental representations of objects insofar as our uses for them go - would require additional experiments. One possibility would be to run an to explicitly ask human subjects "given objects that load highly on this embedding dimension, what can you do with them",, and consider the typicality of verb answers against the weight given to those verbs by the dimension. Given that our embedding is based on language data about which verbs apply to which objects, we would expect these experiments to give verb loadings coherent with ours.
To increase prediction quality in future work, one approach will be to enrich and refine the co- occurrence matrix in larger corpora, now that the basic approach has been shown to be feasible. Another interesting future direction will be to understand how affordance could be driven by more iine-grained visual appearance properties, by considering other semantic dependencies, or jointly using text and image features such as (Wang & Tarr 2020).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christos Boutsidis and Efstratios Gallopoulos. Svd based initialization: A head start for nonnegative matrix factorization. Pattern recognition, 41(4):1350-135 2008.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911, 2014.
John A Bullinaria and Joseph P Levy. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd. Behavior research methods, 44(3):890-90)) 9070 2012.
Yu-Wei Chao, Zhan Wang, Rada Mihalcea, and Jia Deng. Mining semantic affordances of visual object categories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4259 4267, 2015.
Haonan Chen, Hao Tan, Alan Kuntz, Mohit Bansal, and Ron Alterovitz. Enabling robots to un- derstand incomplete natural language instructions using commonsense reasoning. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 1963-1969. IEEE, 2020.
Scott Deerwester, Susan Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41 (6):391-407, 1990.
Barry Devereux, Lorraine K Tyler, Jeroen Geertzen, and Billi Randall. The centre for speech, language and the brain (CSLB) concept property norms. Behavior research methods. 46(4): 1119-1127, 2014.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: bidirectional transformers for language understanding. arXiv preprint arXiv: 18.000 2018.
Pre-training of deep
Katrin Erk. A simple, ammilarity-based model for selectional preferences. 2017 Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, (June):216-223,, 2007.
Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. Introducing and evaluating WkWaC, a very large web-derived corpus of english. Proceedings of the 4th Web as Corpus Workshop (WAC-4) Can beat Google, pp. 47-54.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense? arXiv preprint 1908 02889, 2019.
James J Gibson. The ecological approach to visual perception. 1979.
Nicolas Gillis. The why and how of nonnegative matrix factorization. Regularization, optimization, kernels, and support vector machines, 12(257):257-291, 2014.
Mohammed Hassanin, Salman Khan, and Murat Tahtali. Visual affordance and function understand- ing: A survey. arXiv preprint arXiv:/807.0677 007770770077. 2018.
Martin Hebart, Charles Zheng, Francisco Pereira, and Chris Baker. Revealing the multidimensional mental representations of natural objects underlying human similarity judgments. 2020.
Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris Baker. THINGS: database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10), 2019.
Mariam Hovhannisyan, Benjamin Geib, Alex Clarke, Rosalie Cicchinelli, Roberto Cabeza, and Simon Davis. The visual and semantic features that predict object memory: Concept property norms for 1000 object images. 2020.
Lorenzo Jamone, Emre Ugur, Angelo Cangelosi, Luciano Fadiga, Alexandre Bernardino, Justus Piater, and José Santos- Victor. Affordances in psychology, neuroscience, and robotics: A survey. IEEE Transactions Cognitive and Developmental Systems, 10(1):4-25, 2016.
10

---

Under review as a conference paper at ICLR 2021
Lei Ji, Botian Shi, Xianglin Guo, and Xilin Chen. Functionality discovery and prediction of physical objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 123-130, 2020.
Bhargav Kanagal and Vikas Sindhwani. Rank selection in low-rank matrix approximations: study of cross-validation for NMFs. In Proc Conf Adv Neural Inf Process, volume 1, pp. 10-15, 2010.
Thomas K Landauer and Susan T Dumais. A solution Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211, 1997.
Daniel D Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in neural information processing systems, pp. 556-562, 2001.
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 302-308, 2014.
Dekang Lin and Patrick Pantel. DiRT-Discovery of inference rules from text. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 323-328, 2001.
Yuri Lin, Jean- Baptisse Michel, Erez Lieberman Aiden, Jon Orwant, William Brockman, and Slav Petrov. Syntactic annotations for the google books ngram corpus. 2012.
Manuel Lopes, Francisco S Melo, and Luis Montesano. Affordance-based imitation learning in robots. In 2007 IEEE/RSJ international conference on intelligent robots and systems, pp. 1015-1021. IEEE, 2007.
Li Lucy and Jon Gauthier. Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning, arXiv preprint arXiv:1705.11166 2017.
Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 3774)5547-55, 2005.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint
2013.
George A Miller, WordNet: An electronic lexical database. MIT press, 1998.
Brian Murphy, Partha Pratim, and Talukdar Tom. Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding. In International Conference on Computational Linguistics, 2012.
Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In 2015 IEE International Conference on Robotics and Automation (ICRA), pp. 1374-1381. IEFE, 2015.
Don Norman. The design of everyday things: Revised and expanded edition. Basic books, 2013.
Sebastian Padó, Ulrike Padó, and Katrin Erk. Flexible, corpus-based modelling of human plausibility judgements. SMNLP-CoNLL 2007 Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, (June): 400-409, 2007.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In P Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Michele Persiani and Thomas Hellström. Unsupervised inference of object affordance from text corpora. In 22nd Nordic Conference on Computational Linguistics (NoDaLiDa' 19), September 30-October 2, 2019, Tarku, Finland. Association for Computational Linguistics, 2019.

---

Under review as a conference paper at ICLR 2021
Matthew E Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1rxiv:180..05335, 2018.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: python natural language processing toolkit for many human languages, 2020.
Philip Resnik. Selectional constraints: An information-theoretie model and its computational realization. Cognition, 61(1-2):127-15, 1996.
Philip Stuart Resnik. Selection and information: a class-based approach to lexical relationships. IRCS Technical Reports Series, pp. 200, 1993.
Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rappoport. How well do distributional models capture different types of semantic knowledge? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 726-730, 2015.
Johann Sawatzky, Abhilash Srikantha, and Juergen Gall. Weakly supervised affordance detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2795-2804, 2017.
Karin Kipper Schuler. Verbnet: A broad-coverage, comprchensive verb lexicon. 2005.
Diarmuid Ó Séaghdha. Latent variable models of selectional preference. ACL 2010 - 48th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, (July): 435 444, 2010.
Robert Speer and Catherine Havasi. Representing general relational knowledge in conceptnet 5, In LREC, pp. 3679-3686, 2012.
Peter Turney and Michael Littman. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions Information Systems (TOIS), 21( 2003.
Peter D Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37:141-188, 2010.
Akira Utsumi. Exploring what is encoded in distributional word vectors: A neurobiologically
motivated analysis. Cognitive Science, 44(6);e 2844, 2020.
Tim Van De Cruys. A neural network approach selectional preference acquisition. EMNLP 2014 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 26-35, 2014. doi: 10.31 151115/11/ll 1/tl/1/11111000 1004.
Aria Yuan Wang and Michael J Tarr. Learning intermediate features of object affordances with convolutional neural network. arXiv preprint vxXiv:2002.08975 2020.
Philipp Zech, Simon Haller, Safoura Rezapour Lakani, Barry Ridge, Emre Ugur, and Justus Pi- ater. Computational models of affordance in robotics: taxonomy and systematic classification. Adaptive Behavior, 25(5):235-2711 2017.
Hongming Zhang, Jiaxin Bai, Yan Song, Kun Xu, Changlong Yu, Yangqiu Song, Wilfred Ng, and Dong Yu. Multiplex word embeddings for selectional preference acquisition. arXiv preprint arXiv:2001.02836, 2020.
Charles Zheng, Francisco Pereira, Chris Baker, and Martin N. Hebart. Revealing interpretable object representations from human behavior. In 7th International Conference Learning Rep- resentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
net net.//orumm
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX: HYPER-PARAMETER SELECTION FOR NON-NEGATIVE MATRIX
FACTORIZATION
Reconstruction Eiror
-
0.01
-K- 0.1
---
0.3
0.5
-
1.0
-0-
2.0
24x10
23x10'
22x10'
21x
Ellopscclel
2x16
18x10
16x10'
20
60
100
120
Figure 4: A zoom-in plot for the reconstruction errors.
Denote M., M, E {0, 1]nxm to be the mask matrices for indicating held-in and held-out entries of the input PPMI matrix in CV procedure, then optimize for O and V*:
O", 0","" argmin|| (P +P.OOV")I 0,V
(6)
To apply the multiplicative update scheme as proposed in (Lee & Seung 2001 we first consider the partial derivatives with respect O and V. Denote (), = M, we have
O +
VoF(O, (M @ OVT)V P)V + B Vv FvO(O (M, @ OVT)U (M o P) T + 3
(7)
We then have the following update rules that guaranteed be non-increasing:
(M, @ P)V()
(b+1)) + ((0)
(M, @
+
(8)
((+1) - v(i O (M, @ T)
(M, O (V
where the fraction here represents elementary wiise division. For the choice of M, and M., we follow
the same approach as proposed in (Kanagal & Sindhwani 2010). We first split the matrix into K
blocks, where the rows and columns randomly shuffled. Denote (*) and c(k) to be the index vectors for rows and columns respectively, where x column contains in block k. The mask for k-th block then be expressed ( (k) @ (4) We then randomly select out of K blocks as holdout blocks, which gives
if row contains in block k, or c = if
M, (ba) 6o (la)),
M₁ M
where kg is the index of selected block. The reconstruction error can thus be computed:
E = M|MM o (P 0 (v*)) + VV*).
(10)
Figure shows zoom-in plot of the reconstruction error under different combinations of d and For every (d, (3) setting, we perform multiple optimization since NMF is sensitive to initialization. 70 and / 0.3 accordingly. Empirically, we observe that the rank selection is
We then choose quite robust to over-fitting when there is a sufficient sparsity control, for instance, 3 > 0.1 in our dataset. We also observe that whenever d [50, 150) and 3 E (0.05, 0.5). the results are similar.
13

---

Under review as a conference paper at ICLR 2021
B APPENDIX: DISTRIBUTION OF AAUCS ON WTACTION AND MSCOCO DATASET
The following figures show the AAUC distribution of the top embeddings the WTaction and MSCOCO dataset respectively.
AAUC distributions in WTaction Dataset 14,
Proposed (0.88, 4.00e-03) LSA (0.81, 6.00c-03)
0.81
Word2Vec (0.70, 7.00e-03)
0.70
GloVe (68) (0.76, 7.00e-03)
(840B) (0.80, 6.00e-03)
10.80
PPMI (0.77. 6.00c 03)
0m 100
AAUC values
Figure 5: AAUC Distribution on WTaction Dataset
AAUC distributions in MSCOCO Dataset μ4, Se
Proposed (0.74, 6.00c: 03) LSA (0.63, 7.00e-03)
0..63
Word2Vec (0.59, 7.00e-03)
10.59
GloVe (68) (0.64, 8.00e-03)
(840B) (0.68, 68, 03)
10 00.68
PPMI (0.61, 8.00e-03)
0.5
0.7
0.8
0.9
AAUC values
Figure AAUC Distribution on MSCOCO Dataset
14

---

Under review as a conference paper at ICLR 2021
C
APPENDIX: PREDICTION OF SPOSE DIMENSIONS
SPUSE vectore
snject
Figure 7: Prediction of SPoSE embeddings from affordance embeddings (top) versus actual SPoSE embeddings (middle). Objects are grouped by semantic category (those with > 15 objects). The absolute residues of the prediction is also shown (bottom). Color range is fixed to show the magnitude of residues.
15

---

Under review as a conference paper at ICLR 2021
D APPENDIX: AFFORDANCES ASSIGNMENT FOR EACH SPOSE DIMENSION
Pearsie
Dimension
p-salue
label
Tasonumy
Affordances (Top Ten Verbs)
0.84
0.0c+00
animal
categorical
kill, spet, hum, observe, chise, feed, slaughier. sight, find serve, eat, cook. prepare, lasie, consume. add, mis, wear, don, mutch, knit, sew, fastes, tip, embroider, lear, model
0.52
0.0c+00
food
categorical
0:75
6.9e-323
wearable
categorical
0.75
317
vehicle
cutegorical
drive, hire, park, equip, rent, commanderr. crash. board, build ogerate
0.71
2.0e-271
categorical
grow, cultivate, plant, add, amplity, blrre study, tottle, install, operate, connect, activase, invens discomnett, manufacture, purchase. incosporate, design fit, invent, manafacture, incorpecate, design, position, attcch, utiiie, carry. install don, straighten, slash, bandage, spet, obseeve, eradicate, deter, trap. drink, pour, quaft, sip, guzzle, sup, swig, spill, imbiiee gulp
gother,
9.68
musical instrument
categorical
hear, sound, play,
0.08
2.5e 239
electronic device
categorical
0.47
2.5e-227
made of metal
stractural
0.66
6.2e-219
body/claching
categorical
rig, injure heal, model
0.63
bug
cutegorical
0.43
899-195
drink
categorical
0.61
183
tool
categorical
wield, grab, hold. carry, sharpen, swing, hand, pick, clunch. shrow
0.57
7.3e-152
fianctional
fasten, righten, unfastn, undo, smach, thread,
loos
0.55
3.0e-122
marine
categorical
sight, copazze, catch, contain, mis, scotier, add, guther, remove, sprinkle, dry, deposd, shovel steal, diseover, find, carve, place, design, contain, recover, cnift hide desiga, manufacture, fit, incer install, place, purchase,
spot, beach, mooc, observe.
0.54
2.ee-344
grauultted
stractural
0.52
6.76-125
valaalotokd
Fanctional
0.52
6.0-114
(honse)
categorical
0.52
1.e--22
acconautic
categorical
spot, observe. sight, destroy, built, construct. photograph. equip, design remove, construct, place, surmond, incorporate. carry. creet, design, fi. baild sprain, tractuee handage, fiex, injure, rest, bcuie,, straighten, has, hyperestend carry, hurl, drop, grah, wield, light, estinguish, ignite, throw, carry.
0.52
7.76-111
cundvors
categorical
0.50
74e-114
body part
categorical
0.50
7.
wespon
categorical
fire, hold,
0.48
3.2e-100
flammaalle
finctional
kindle, place, manufacture douse remove, place, hang, tear, stain, speead, weave, clema drape, wrap
0.47
5.5e-6
textured
agpearance
0.44
1.3e-8
mund
sppearance
caltivate, pick. add, slice, place, chop, froww plant find, place, gathes, pick, discover, recover, contain, retreve, find, place, remove, incorpoeuse wenr, romove, don, place, design, buy, call, poll, bring find
0.44
4.7e-82
accessory (heanty)
categorical
give,
0.42
1.le-7
mineraligem
categorical
0.42
2.
accesory (Tace)
categorical
0.40
5.40-88
made of wood
stractural
place, camove, carry, incorporate, design. contain, bring constaue, mamufacture find
0.40
4.
functional
empty, carry, place, clean, load, bring, dump, unload. lome grab, carry. wield, hold, pick, place, fhrow: hand, bring, drep incopprnicc construc, fil, design, ereet, position. install, mount place, boild manu facture remove, place, invent cleun, buy, apply, cuntain, design minufactere,
0.38
200-61
elongated
stractoral
0.38
1-4-60
stoucture
strnctural
0.38
3.5-60
object (buth)
categorical
0.37
1.ee-6
sting-like
sppearance
place, pull, wrap,
call, bring
0.36
wheeled
fiuctional
manufacture, hire, design, equip. fit, rent, pack, parchase, invent mane facture design, purchuse, buy, place, invent introduce, incorpeeate, lit, sell fit, manufacture, connect, design, install, incorporate, attach, millze, purchaso, invent manufacture, design,
0.36
2.0-55
mude of puper
stractural
0.31
4.4c-7
applinccomrcchne
categorical
0.32
1fe-43
speet (ball)
categorical
purchase. grah, caryy introdace. bring buy, manufacture design, purchase, bring. find, sell, istroduce, steal, place, contain, add, remove, scll, manificture combinc, fidd buy, steal place. bring,
0.33
4.34-3
object (children)
categorical
0.31
2.5e-39
agglamerate
apperrance
0.29
1.
faa surface
grab, hang. wrap, minufacturn, buy pull
0.28
circuler
spperrance
manafacture. design position, utilin remove, prsh, manufacture, buy, place, design, bring, wenr, comry invent manufacture, buy, design, place, remove, sell, invent, purchre, contain, bring invent, pliee, remove, find, contain, manufacture, design, bring, carry buy design, hang, buy, put, pull,
place, incorporate,
0.36
14c-28
accessory (block)
categorical
0.24
20x-24
colorful patiern
appeorance
0.23
40--23
object (medical)
categorical
0.23
eppearance
place, bring,
0.23
19e-21
pomocable
stractural
fit, incorperate, remove, place, design, manufacture, install, position, clean, altach place, call, remove, buy, find, bring, manufacturo. intoduce, contain, incoeponte place, call. add, comuin, remove, find, buy. bring, intoduce, sell place, call. find, remove, buy, bring, introduce, contain, ald, manufacture
0.18
6.5e-14
object (small)
categorical
0.18
8Se-11
degree of red
0.13
furry
appesrance
Table 3: Affordance assignment for SPoSE vectors ordered in terms of Pearson correlation with regression outcomes. The dimension labels vastly simplified. The full descriptions for each dimension are provided in Appendix D (Table 4 and Table
16

---

Under review as a conference paper at ICLR 2021
E
APPENDIX: FULL DESCRIPTIONS FOR EACH SPOSE DIMENSION
The abbreviation and the full description of every SPoSE dimension.
Abbreviation
Foll descriptions
accessories, beauty, black, blackness, classy, date, emphasize, fancy, hair, hard, high-class, manly, objects, picture, telescope
accessory (black)
accessorics, body parts, culture, docoration, eyes, face, face accessories, facial, goes on lead, hair, head, less appealing. senses, touches face, wearable
accessory (face)
nccessory, beautiful, beaity, color, fancyness, feminine, feminine items, floral, flowers, flowery, gentle, girly, love, muted colors, pastel. pink
accessory (beauty)
acro-nautic, air, airplanes, oviary, aviation. buoyant, flies, flight, fly, flying, flying to not, high in air, light, move, sky, swim, transportation, travel
acronautic
accumulatable, bundles, collection, colors, countable, grainy to smooth, group of similar things, groupings, groups, groups small objects, large groups, little things, many, metals, nuts, objects, patterns, piles. quantity of objects in photo, round, small, small objects in groups, small parts that look alike, symmetrical
agglomerate
unimal, animuls, animals in zue, animals that do not fly, from complex to less. fuzzy, grass, ground animals, land animals,, mammnal, mammals, natural, size, wild animals, wild human- made, wilderness
animal
upplinne/machine
building materials, construction, destructive, electric items, factory, farm tools, foundation, home tools, in groups, long. machinery, maintenance, mostly orange, processing, renovation, rocks, rope-like, thing, tool, tools
body, body parts, esp extremities, extremities, extremities of body. feet, feet to hands, lingers, found on people, hand, hands, human, legs. limbs, lower body,
body part
bodies, body, body accessories, body maintenance, body part related, body parts, body parts with hair, face, how much skin showing. human, human body parts, part of body, parts, people, skin, touched by skin animals that stick onto things, ants, bug. bugs, can burt you, dangerous, gardening, insects, interact with bugs. small unimals, small to large, wild
body//lothing
bug
bands, bondages, can chained, circles, coils, construction, fasteners, knotting, long, rope, ropes, round, string-like, strings, tensile, thing can tie around, tied, tics, trapped, violent, wires, wrap, wrapped around to what gets wrapped
can tic
circalar
circles, circular, cylindrical, discs, flat, round, shape. targets
colorful pattern
artistic, bright, hright colors, color, color variety, color vihrancy, colorful, colors, many colors, patterss
able to something in it, boxes, huckets, can put things inta, carts, container, containers, containers for liquids, containing, covering, cylinders, diverse, drums, enclosed objects, hold other things, hollow tubes, paints, shapes, storage, unknown
container
a lot of patterns, art, artisinal, aris, candles, circles, color, crafts, detailed dois, do it yourself, grandma, grandparent-like like, handmade, home patterns, home-making related, housewark, in grandma's home, intricacy, quilt, rectangles, sewing, specks, stitching twine, unknown, weaving, wood, woven, yarn
craft
degrre of red.
color, colors, degree of rednness red, red (bright)
3 d-dimensional, beverage, containers, containers for liquids, drinks, edible, gluss, glasses, hold liquid. liquid. liquids, other things, things that in containers, things that hold liquids, vessels liquids, viscusity
drink
digital, digital devices, digital media, electric, electronic, electronies, hard, hard to understand, media, old technology, wecnolooical, technology, telephones, typing instruments
electronic dovice
able to be held, cane, cylinders, cylindrical. darts, grouped. long. long narrow, long objects, long-shaped, narrow, pen like, pencils, pens, shupe, sharp, skinny rectangles, stick like, sticks, straight, straight to curved, symmettrccal, thin
elongated
flammable
fire, flammability, flammable, beat, hot, light, outdoors, warm
attaching. breakable, clean, eloth, convenience, disposable, flat coverings, gathering, grated pattern, handle everyday, helpful, hold things in, multi- shaped, noe smooth to smooth, paper, paper-like, sheets, stick- thin, things that roll, tissue, white
flat surface
baked food, baked goods, carbs, cheesy, comforting, cooked, deliciousness, edible, entrees, food, made dish. natural products, nutrients, pastry, propared food. processed, salt. where it comes from
food
furry
fuufy, furry, more of one color, white, white and fluffy, winter
a lot of items, ash, color, dirt, elements, grain looking, grains, grainy. grainyness, gramular objects, ground, ground (grinded), homogencity, lots same, many, not colorful, parricles, rocky, shope, size particles, small, small particles, stones, tiny groupings, tone, unknown mincrals or drugs
granulated
buckle, build, building, gray, hurd, metal, metal lools, metallic, metallic tools, metals, shiny, silver, tools, use with hands
made of metal
books, card, classroom, collections, flatness, found office, groups, has text, note takentaking, affice, paper, paper (colorful), papers. printed reading materials, rectangles, school, square, squares, stacks, striped, work brown, made of wood, natural, natural resources. orange, wood, wood-colored, yellow aquamarine life, aquatic activities, cruise, fish, in water, marine, nautical, ocean, outdoor, outside water, paradise, sca, ships, vacation, water
made of paper
made of wood
marine
Table 4: Descriptions of abbreviation (A)
17

---

Under review as a conference paper at ICLR 2021
Abbreviation
Foll descriptions
beauty, minerals, crystal, deriivderived, gems, in artistic way, inspecting. intricate, joweeyy jewels, metallic, natural, natural minerals, prized, pure, rare, reflective, round, roundish, sharp, shiny, shinyness, sterile, translucent, valuable
mineral/gem
contrel noise, hearing. instruments, listen, listening, loud, make noise, music, music instruments, musical. musical instrumenis, recreational instruments
musical instrument
bathroom, cleaning, essential everyday, gray, home (inside home)., household items, hygiene, self--cure, stap. toiletries, water, white
object (bath)
object (children)
baby, baby toys,child, child, child-like, children, dolls, toys. young. youth
bland to colorful, chairs, cloth, common in houschold, everyday household, flat, furniture, house, house essentials, house surfaces, household commonness, household furniture, house, living furniture, main component room
object (bouse)
health, health--cnceerinn
hospina, hygiene, injury, medical, medical instruments, medical supplies, medicine,
object (medical)
sick, to good health, unhealthy, water, wellness
7. amish, appealing, candles, circular, color? Unsure, colorful, covers, cylinders, cylindrical, laat fat cylinders, hands on, jewelty, lids, saving, shape, similar-haped, things you grab, twine, unknown, yellow
object (small)
backyard, blue. hrown colored, columns, commen in outdoor, din, gurden, landscape, made, momuments nature, natural, nature, not colorful, outdoor ohjects, outdoorsy, park, pavemennt pristine, public, quiet, rocks, rough, rows, sccnery, separaters, stand their own, statues, stone, tools, wood, woodsy,
outdoors
can pass through, doi patterned, grates, boled, knit, little holes, mesh, metal, net, nets, octagonal, patlern, patterned, patterns, patterns (holes), repeated pattems, repeating, repeating patterns, repetitive, shiny, silver, small, strainer
permeable
green. green leaves, green plants, green plants and herbs, greenery, greenness, greens. grow, natural, nature, plant like. plants, things that grow from earth, vegetable, vegetables
plant
urtistic, ball, balls, circular, circular and colorful, circus, contrasting circles, fruits, kid, pictures, round, rund, shape, spherical
round
sport (ball)
athlctic, ball toys, balls) to less active, competition, recreation, round, sport, sports, sports (active, sporty
amount of rigid cads, confetti, diffcrent shapes, clongated, hay, high-density, knots, lines, lines jutting out, long- long things mashed up, look like sticks, mesh, netting, patterns, prickly. protuding. repeating in an ordered way. rope, ropes, skinny, spiky. stacks, strings, stringy. symmetrical, tangled, twirled aroond
string-like
amusement, antenna, big, cagod, common to humans, complete, disordered. electrical. elongated, enclosures. found outside, grand, high. high in air, industrial. ladder. large, multiple cylinders, multiple similar things. narrow, ouldoor, part circle, stupes, stacks, structural, tall, things that go up, that hang. trapped, wires
structure
appealing, carpets, flat coverings, fractals, lay flat, mesh, pattem, patterned, patterns, pieces, rectangles, repeating shapes, repetitive. rugs, sheets, small repeating patterns, squares, textured
textured
clongated, hand tools, houschold, jaggod, long. pointy tools, pole, saws, scrape, sharp. sharp tools, small instruments, straight, tools, use with hands, utility wedges
tool
Fnglish royalty, antiquity, bottles, bronze, fine things, gaudiness, gold, high-cluss, high-quality, history, important, jowelry, jewels, monarchy, old, ornaments, procious metals, pristine, royal, royalty, shiny, silver. trophy, valucable
valuablefold
can be moved, car, cars, complex vehicles, construction, efficiency of transportation, fast, ground motorized vehicles, machinery, muhility, move, speed of movement, transportation, transporation vehicles, travel with, truck, vehicles, wheeled vehicles, wheels
vehicle
black, danger, dangerous, equipment, masculine, military, repaively-associatee ormaents, risky, self-defense, somber, violence, violent, war, warlike, weaponry, weapons
weapon
uccessories, blue, can wear or carry or put on, clothes, clething, cotlon clothing, covering body. shirt, things to wear, things you wear, touch hody, touch person, utilities, warm clothes, wearable
wearable
able to hold, bicycle, caged, can sit on, can stand or drive, carrier, chair, destinations, holds objects, light movement, mobility, motion, move someone, playful, round, thing, things wheels, trapped, wagon, wheel, wheeled structures, wheels
wheeled
Table 5: Descriptions of abbreviation (B)
18

---

Under review as a conference paper at ICLR 2021
F APPENDIX: COMPONENTS OF SPOSE DIMENSION APPROXIMATION
The following consecutive Figures 8 13 show the component information each SPoSE dimension. The percentage is calculated based the portion w; - -IOO,iil, where w; is the regression coefficient corresponding to our i-th embedding dimension. The right ticks show the dimension affordances.
animmal
23.518
sight.
photograph, find, chase, butcher,
Dim
15.89%
hunt, cull exterminste, stalk,
Bim 18
11.6455
pasture, herd, slaughter, milk tether, immolate, impound, saddle, bleat dediaw, leash, wor,, feud, groom, outhanize, pet, crate, sled, yelp eradicate. deter, swat. combat, discourage, exterminate, prevent, kill, elminate, trap
Dim 36
9.02%
Dim 54
7.02%
food
Dim 25
27.48% serve, eat, cook, prepare, order, consurre, taste, accompany, sell, bake bake, leaver, serve, cook, crumtle, tap, kneed, freeze
Dim 64
10.96
10.4995
blend, combine.
7.58%
chop, grate, quarter, grow,
cultivate, crow, plant, prune, propagate, gather, ripen, dry, pick, graft
Dim
drink, sip, sup, swig, quaff, guzzle, imbibe, gulp, brew, chug
Gim
6. 6.04%
munch, scoft, eat, gobble, nibble, consume, devour; toast, wolf, serve
Dim
5.3%,
wirnow, mill, parboll, grind, sift, dehusk, stockpile, infest, hoard
..04
blanch, boll. drain,
evercook, addi braise
wearable
9
weat, den, unbutton, match, button, model, iron, knit, tug. unfold
19.03%
embroider. hang, unfasten, tighton, fasten, undo, lossen, locse, sticken, tie buckie, undee
Dim
vehicle
46.49%
stop,
17.01%%
plot,
mount, position, incorporate, fix, install, destroy
9.16%
8.61%
beach, raft, paddli, deice, whir, fit, swagger, quiver, pinion, flip. flounce, futteer,
Dim
3.01%
plant
Cim 57
31.03% cultivate, grow, plant, prune, propagate, gather, ripen, dry, pick, graft grass, upreot. muich, smother. dothe, rost, prunt, mew, water. crop blanch, boil, steam, drain, cook, fry, overcook, add, braise sprinide,
Dim 8
17.45%
Dim 2
14.7%
9.11%
blend, combine.
96%
spot, sight.
photograph, find, eradicate, deter, swat, combat, discourage, exterminate, prevent, eliminate, trap slice, peal, chop, grate, mash, quarter, grow, saute, cultivate
5.88%
Dim
fleck, scallop, strew, emanate, pluck, suffuse, scatter. notch, tatter, crystallize
musical Instrumant
Dlm 24
64.8% sound, hear, play, blare, amplfty. learn, toctle, toot, study, tinide
wield, grab, carry, hand, swing, wave, clutch, pick, manufacture, purchase, sell, design, introduce, pmoduce, stock, choose
Dim
59%
connect,
activate, operate, plug, integrate, trigger. wire, incorporate
13.61%
manufacture, buy,
design, introduce,
D0m
9.8%
invent, introdace,, manufacture, develop, design, openate, employ, constract, sell determine, compute, plot, ascertain, underline, whter, erase, even, highlight, express
Dim 17
5.68%
made of metal
10.0%
position, incorporate, fix, install,
8.41%
grab, carry,
wave, pick,
8.27N
openate, plug, integrate, trigger, incorporate
7.45%
unscrew, screw, slacken, disengage. tighton, weld, undo,
7.34%
introdiaee manufacture, develop, dasign, operate, employ, buy, construct, sell sound, hear, play, blare, ampllfy, warn, tootle, toot, study, tinde sharpen, blunt, wield, plunge, thrust ply. forge, grasp, hold. heft
Dim
Figure 8: Components of SPoSE dimension approximation (A)
19

---

Under review as a conference paper at ICLR 2021
body/klething
wear, don, unbutton, match, button, model, Iron, tug,
Dim
sprain, hyperextend, flex, frecture. Injure, bandage, bruse, rest, hurt, straigtten
Gim 9.07%
rouge, twindl,, flinch, twitch, snser, thumb, crinkle, redden, writkle, regale
Dim
7.0.%
cloister, remanry. ostracioe, unianize, intermarry. uldate, lytch, jail, famish, pillery thud.
6.84%
swathe,
6.59%
chaperone
moisturize, spritz, dehair, deflesh, redden, rabbit, bronze, glare
Dim
alcerate, distend, decompress, sicken, repopulate, unhinge, gyrate, tone, chafe, pat gabble, cluck, bridle, loll lisp, trill, ukeraae,, grash, bifurcate, rant
Dim
5.18%
bug
30.00%
eradicalte, deter, combat, discoura exterminate, prevent, kill, eliminate, trap
29.54%
degut,
10.30%
watch, kill, photograph, find, chase. feed orsity, defiesh, caloffy, masticate, jumble. fra cture weaken, strengthen, roughen, grash
Dim 56
6.87%
drink
Cim
34 12%
drinkk, sip, sup, swig. quaff, guzzle, imbibee, gulp, brew, chug emply. fill, clean, dutterr, line, label, tip, overload, clear, seal squirt, formulate, dilute, smear, dissolve, inject, evaporate, ocee, drip. apply
Gim
19.07%
Dim 6
Dim
thicken, spoen, reheat, stir, simmer, season, pour, sieve, ladie, drizzle invent, introduce, manufacture, develop, design, operate, employ, buy, construct,
Dim
sharpen, blunt, wield, plunge, thrust, ply. forge. grasp, hold, heft wield, grab, carry,
Dim
carry.hald, hold, hand, swing, wave, clutch, pick snatch unscrew, screw, slacken, disengage, lighten, weld, solder, introduce, manufacture, develop. design, operate, employ, buy, construct, moiston, rinse, soak, reuse, dip, wipo, saturate, insert,
insert
6.5%
Cim
57.55%
unfasten, lighten, fasten, undo, loasen, loose, slacken, tie, buddle untie coll, splice, braid, sever, thread, loep, soktier, shorten, lengthen, snip
Glm 20
42.45%
marine
34 82%
capsize.
skipper.
22.38%
degut,
17.01%
sight, kill, photograph,
10.42%
rearm, station, drydock, charter, arm, destroy
8.%
block, clog. dam, choke, obstruct, repair, aweesh, leak, tramp, whoosh, heap, compact, promenade, rrapso, shingle, accumulate
5.01%
granulated
78.
heap. compact, promenade, poe, shingle, accumulate quarr, vii, petrify, deforest, siicify, mine, virify, carbonize, hew,
tramp,
Dim
winnow, mill porboll, grind, reap, sift, dehusk, stockplle, infest, hoard sprinile, mix, add, sti, blend, combine, taste, spread, seatter, contain
Cim 29
10.34%
Dim
7.75%
meoe, fisuree furrww, burtle, decoct, bawl, macerate, gnaw. exfollate, ream
Dim 26
1.20%
steal, recover, retrieve, discover, hide, pawn, smuggle, regain, snatch, find
Dim
6. 6.68%
deflesh, calefly. masticate, jumble. fracture, strengthen,
5.14%
eradictte,
eliminate, trap
valuable/bld
Dim 26
25.87%
steal, recover, retrieve, discover, hide, smoggle,, snatch, find
Dim 62
25 25.0%
erect carve, flank, adom, construct, surmount, desecrate, souipt, demolish, decorate sound, hear. play, blans, amplify, lear, tootle, tost, study, tinkee wear, don, unbutton, match button, model, iron, knit, tug, doff
Dim 24
12.05%
Dim 9
8.36%
Dim 60
motorcycle, zigzag, crisscress, roll, span, pulsate, scour, roam, traverse, repopulate
object (house)
Dim
empty, fill, clean, dutter. line, label, tip, overload, clear, seal
Dim 20
14.85%
weave, drape, embroider, knit, sew, hang, stitch, wrap. wave, unfold
Dim 31
13.34%
manufacture, buy, purchase, sell, design, Introduce, produce, stock, choose
Dim
9.32%
open, padlock, unlatch,
widen, unlock, brick
70%5
install, disconnect,
operate, plug, integrate, trigger,
erect, carve, adom, construct, surmount, desecrate, soulpt, damolish, decorate
Dim
nationalize, telephone, reorganize, liquidate, federate, quake, industrisize, privatize, disband.
merge
Dim
5.44%
iptoe, totter, leer, youl, mosey, todkle, sledge, troop, descend ascend
Dim 66
5.28%
fit mount, position, incorporate, attach, fix, lower, Instail, utilize, rotate
Figure Components of SPoSE dimension approximation
20

---

Under review as a conference paper at ICLR 2021
aeronautic
26.94%
destroy
56.18%
observe, sight, hurt, photcgraph. find, feed
Dim
10.74%
deice, whir, fiit, quiver, pinion, flap, flounce, flutter imp Hlank, adom, construct, lour, eddy, soud, glower, taxi, dissipate, exhale, ionize, bekc
Dim
B.45
desecrate, demolish,
Dim 55
6.60%
outdoors
24.16%
erect, flank, adom, construct,
desecrate, demolish,
Dim 8
11.17%
uproot, mulch, smother, clothe, root, now, water, crap lignify, disterpeer, burl, sturco, block, clog, dam, thoke, flood, obstruct, repair, swossh, leak, wodcle scuffle, tramp, shovel, w wheesh, heap. compact, promerade, traipse, shingle, accumulate
Dim 41
6.85%
whilewash, clam, abut, panel, mottle
Dim 65
6.09%
Dim 69
6.06%
body part
Dim
40 92%
sprait, hypereatend, flee, fracture, bandage, bruise, hurt, straighten
Dim
13.42%
rouge, twinkle, flinch, twithh, snear, thumb, arinid, redden, wriklee, regale
Dim 9
12.21%
wear, don, unbuttion match, button, model, iron, knit, bug, dorf gabile, cluck, bridie, lell, lip, trill, ulcerate, gnash, bifuncate, rant casity, deflesh, caldry, masticate, Jumble, fracture, weaken, strangthen, roughen, gnash
Gim
6.97%
Dim 56
6.69%
weapon
24.27%
huri, fire, explode.
chuck, drop hold, heft
15.82%
wield,
Dim
15.01%
pilnt, beard, rearm, crew, station, drydeck, charter, commandeer, destroy wield, carry, hold, hand, swing, wave, clutch, snatch deflesh, calofy, masticate, jumble, fracture, weaken, strengthen, unbuttion, match, button, model. knit, doff
11.72%6
5.75%
5.3%
Dim
extingubh, light, kindle, rekindle, flare, ignite, shine, dim, flash, douse
18.08%
fire, throw, explode, detenate, toss, snipe, chuck, drop
block, clog, flood, obstruct. repair,
6.91%
quarryy patrify, silkify, vitrify, carbonice, drydock,
5.3%%
beard,
textured
25.83%
embroider,
unfold
Dim 69
22.56%
scuffle, tramp, shovel, whoosh, heap, compact, promenade, traipse, shingle, accumulate
Cim 41
13 34.4%%
fignify, distempper, burl, stucco, paper, whilewash, clam, abut, panel, mottle block, clog, dam, choke, floed, obstruct, repair, swoosh, lesk, wadele
Dim
33.69%
strangulats, fumille, glove, punt, bunt, belt, vulcarize, hitt slog, ballcon cultivate, grow, plant, prans, propagats, gather, ripen, dry, pick, graft slice, peol chop. dice, grate, mash, quarter, grow saute, cultivate lbb, hurl, fire, throw, explode, detonate, toss, snipe, chuck, drop quarry, ry, vein, patrify, deforest, silicily, mine, vitrily, carbonize, hew, forest steal, recover, retrieve, discover, hide, pawn, smaggle, regain, snatch, find
Dim
21.29%
Dim
23.22%
Dim
133.21%6
Cim
10.07%
Cim
6.02%
accessory (beauty)
23.63%
scallop,
crystallize
steal, recover, retrieve, discover, hide, pawn, smuggle, regain, wear,
66.81%.
bug, grow, plant, prune, propagato, gather, ripen. dry, graft
weave, drape, embroider, knit hang, wrap, wave, sound, hear, play, blane, amplity, learn, tootle, toct, study, tinkle
mineraligem
30.96% steal, recover, retrieve, discover, pawn, smuggle. regain,
lour, eddy, soud, glower, dissipate, roil, exhals,
Dim
extinguth, light, kindle, rekindie, flare, lonite, shine, dim, flash, douse fit, mount, posktion incorporate, attach, fix, lower, install, utiliz, rotate
Dim
7.62%
Dim 6
6.48%
squirt, formulate, diute, smear. cissolve, Injest, evaperate, oaze, drip, apply
Dim
6. 0.0%%
scuffle, tramp, shovel, wheesh, compact, promerade, traipse, shingle, accumulate energize, implant, cool, interconrect, short, carbonize, embed,
Dim
5.27%
quarry, deforest, silicfly
Figure 10: Components of SPoSE dimension approximation (C)
21

---

Under review as a conference paper at ICLR 2021
accessory (face)
rouge, twintle, flinch, twitch, sneer. thumb, crinkle, redden, wrinkle, regale
24.57%
pomade,, gel. tweeze, primp,
model, shampon,
20.57%
doff birch, remonsirate, billet, room, feminize, estracize, wince, stammer, falter, chaperone
Dim
14.44%
made of wood
75%
souipt, quary, ven, petrify, deforest, sitcily, mine, virifyy carbonize, hew, meow, fissure, furrow. burble, decect, bawl, macerate, gnaw, exfollatts ream
erectt carve,
surmount,
Dlm
9.34%
Gim
unload, wheel, lug, load, transport, haul, stow, dump, offload, carry nationalize. telephone, reorganize, liquidate, federate, industrialize, disband. merge
Dlm 44
7.76%
Dim
6.91%
unscrew. slacken, disengage. lighten, weld, soider, undo, insert
Dim
6.89%
tittoe
todole,
descend,
wiekl, grab, carry. hand,
container
Dim
47.01%
emply. fill, clean, dutterr line, label, tip, ovveroad, clear, seal
Dim
unload, wheel, lug. load, transport, haul, stow, dump, offieed, carry
12.87%
nationalzze, telephone, reorganize, liquidate, federate, quake, industrialize, privatize, disband. merge
Dim
6.11%
thud,
elongsted
wield, grab, carry. hand, swing,
sharpen, plunge, ply,
splice, thread, feage.grasp, grasp,
13.3%
unscrew, strew, disengage, lightenn lengthen, snip
8.76%
study, play, amplify. surmount, desecrate, souipt, demolish, erect, carve,
5.83%
structure
Dim 62
27.73%
orect, carve, flank, adom, construct, surmount, desecrate, souipt, demolish, decorate ftt, mount, position, incorporate, attach, fix, lower, Install, utilize, rotate tiptoe, totter, leer, you!, money, toddlle, sledge, troop, descand, ascend
Dim 66
21.61%
Dim 14
14.13%
Dim 58
9.12%
piko,, board, ream, crew, station, drydock, charter, commandeer, destroy
Dim 28
7.96%
close, open, padlock, unlatch,
lock, wider, uniock, brikk
Dim
5.2%
sound, hear, play, blare, amplffy, learn, tootle, tose, tinkle
abject (bath)
22.4%
moiston, rinse. spal, reuse, wipe, saturate, insert, inject, eveporate, drip, apply
97%
squirt, formulate,
9.30%
block, manufacture. purchase, introduce,
Dim
8.69%
stock, cheose
5.72%
unfix,
string- like
23. 52%
coll, splice, brakd, sever, thread, so soldier, shorten, langthen, snip moc, pomade, gel, tweeze, primo, comb, dehair, unfix, shampoo,, peart weave, drape, ambroider, sew, hang, stitch, wrap,, wave, grass, upract, mulch, smather, dathe, root, prune, mow, water, crop
17.1
3..9%%
energize, implant, cool, interconnect. couple, short, compansate, pulse, embed, wot, moiston, rinse, soak, reuse, dip, wipe, saturate, insert, dry
wheeled
Dim 4
31.99% park, drive, hife, crash, rent, commandeerr ride, boand, shunt
Dim
14.23%
unload, wheel log, load, haul, stow, dump, offlead, carry position, incorporate, fix, lower, instail, utilize, rotate
11.39%
6.8%
tptoe,
toddlee invent, introduce, manufacture,
descend,
5.70%.
5.66%
manufacture,
design. introduce,
cheose
Figure 11: Components of SPoSE dimension approximation (D)
22

---

Under review as a conference paper at ICLR 2021
made paper
Dim 27-
24.5%
determine, compute, plat, ascertain, underline, alter, erase, even, highiight, express manufacture, purchase, sell, design, introduce, own, produce, stock, cheose insail. cornect. disconnect, activate, plug,
Dim 31
14.13%
8.03%
6.84%
dry
6.51%
emply, clean,
wield, grab, carry,
swing, wave, dutch, pick, wenve, drape, embroider, kiit, sew, hang. stirch, wrap, wave, unfold steal, recover, retriove, discover, hide, pawn, smugole. regain, snatch, find
Dim
5.0.9%
Dim 68
15.08%
unlead, wheel, lug, load, transport, haul, stow, dump, offlcad, carry coll, splice, braid, sever, thread, loop, soldier, shorten, longthen, snip
Dim
14, 86%
Dim
12.81%
install, cornect, disconnect, activate, operate, pluy, integrate, trigeer, wire, incorporate
Dim 65
5.725
block, clog, dam, choke, flood, obstruct, repair, swoosh, leak, wedele
7.8%
implant interconnect, manufacture, purchase,
embed,
5.59%
introduca, crew, drydeck, arm,
5.33%
destray
sport (ball))
Dim
strangulate, fumble, glove, punt, bunt, belt, vulccaizz,, hit, slog, ballcon
Gim 31
10.94%
manufacture, buy, purchase, sell, design, introduce, own, produce, stock, choose
Dim
9.91%
park, drive, hire, crash, rent, commandeer, ride, stop, board, shunt
9.64%
wield grab. carry, hold, hand,
9.64%
capsize,
skipper,
sound, play, blare, amplify, Iearn, tootle, toot, study, tinkle
wear, don, unbutton, match, button, model, iron, knit, hug,
Gim
61.9
fit, mount, pasition, incorporate, attach, fix, lower, install, uttlize, rotate
Dim
5.29%
invent, introduce, manufacture, develop, design, operate, employ, buy, construct, sell
oblect (children)
25.92% meel, remonstrate, room,
breastfeed, swaddle,
eutharize
wince, stammer, chaperone
produce, stock, erect, carve, adorn, construct, surmount, desecrate, sculpt, demolishh deecorate
manufacture, buy, purthase, sell, gn.
7.38%
steal, recover, retrieve, discover, hide, pawn, smuggle, regain, snatch, find
Dim 9
7.28%
wear, don, unbutton, match, button, model. iron, knit, tug, determine, compute, plat, ascertain, underline, alter, crase, even, highlight, express
Dim
5.67%
agglomerate
steal, recover, retrieve, discover, pawn, smugole, regain, snatch, find
wet, meisten, rinse, soak, reuse, wipe, saturate, insert, dry
quarry, vein, petrify, deforest, silicffy, mine, virfffy. carbonize, hew, forest sprinkle, mix, add, blend, combine, taste, spread, scatter, contain winnow, mill parboll, grind, reap, sift, dehusk, stockpile, infest, hoard
Dim
Dim
cultivate, grow, plant, prune, propagate, gather, ripen, dry, pick, graft
flat surface
Dim
38,00% weave, drapo, embroider, knit sew, hang, stichh, wrap, wave, unfold
wet, meisten, rinse, soak, reuse, wipe, saturate, insert, dry wield, grab, carry, fill, clean, dutterr, line. label, tip, everload, clear, seal
11.21%
swing,
Dim
9.71%
circular
Dim
13.90%
strangulate, fumille glove, punt, bunt, belt, vulicaize,, hitt slog, ballcon fit, mount, pasition, incorporate, attach, fix, lower, install, utilze, rotate metortyde,
Dim 66
11.14%
8.83%
roll, pulsate,
7.39%
top,
6.78%
noneycomo, etch, lacquer, embess, sonsitize, fabricate, bisect, steal, recover, retrieve, discover, hide, pawn, smuggle, repain, snatch, find block clog, dam, choke, flood, abstruct, repair, swoosh, leak, waddle
Dim
Dim
6.1/%
Dim
5.88%
invent, Introduce, manufacture, develop, design, operate, empley, buy, construct, sell thicion,
5.34%
ladie, drizzte
Figure 12: Components of SPoSE dimension approximation (E)
23

---

Under review as a conference paper at ICLR 2021
accessory (black)
16.
tug, wield, grab, carry, hold, hand, swing, ware, clurch, pick, snatch
don, match, button,
13.00%
6
squitt, formulatee, dilute, smear, dissolce, inject, evaporate, ocee, drip, apply
Dim
drink, sip, sap, swig, quaft, quzzle, imbibe, qulp, brew. chug
Gim
cleistar, remanry, ostradae, urionize, intermarry, ululate, lynch, jail, famiis, pillory
tim
7.17%
steal, recover, retrieve, discover, hide, pawn, smuggle, regaii,, snatch, find
Dim 66
5,22%
fit mount, position, incorporate, attach, ch, fix. lower, install,, utilize, rotate
colorful
Gim
27.78% determine, compute, plot, ascetain, underline, alter, ersse, even, highlight, express
Gim 20
15.89%
weave, drape, embrokter, knit, sew, hang, stitch, wrap. wave, unfoid
Dim 6
15.5%
squirt, formulate, dilute, smear, dissolve, inject, evaporate, ocas, drio, apply manufacture, buy, purchase, sell, design, introduce, own, produce, stock, choose unbutton, button, model,
Dim 31
13.107%
8.00%
doff
Dim
6.0%
discover, hide, coll, braid, thread,
56%
lengthen,
object (medical)
24.1% assity, caleify,
strengthen, raughen, grash
wet, moiston, rinse, soak, reuse, dip, wipa, saturate, insert, dry
Dim 1
invent, introduce, manufacture, develop, design, operate, employ, buy, construct, emply, fill, clean, dutter, lhe, label, tip, overload, clear, seal steal, reorver, retrieve, discaver, hide, pawn, smupgle, rega snatch, find gabble, cluck, bridle, loll lisp, trill, ulcerate, gnash, bifurcate, rant freckle, moittrize, dehair, deflesh, redden, rabbiit exfoliate, bronze, glane unload, load,
Dim
Olm 26
1.93%
Olm 45
7.39%
Dim 27
6.15%
5.39%
offload.
craft
Dim 20
75.14% weave, drappe, embroider, knnt, sew, hang, stttch, wrap, wive, unfoldd todolee stedge, troop, descend,
13889%
Dim 66
37.82%
fit, mount, position, incorporate, atta ch, fix, lower, Install, utilize, rotate black, dag, dam, choke, food, obstruct, repair,, sweesh, leak, padlock, unlatch, reapen, lock, wider, unlock, brick
Dim
27.89%
object (small)
Dim
65, extinguish, light, kindle, retindle, flane, ignite, shine, dim, flash, douse
quarry, petrify, deforest, slidfy, mine, virily, carbonize,
Dim
9.43%
padiock, unlatch, coll, splice, brakd, sever, thread, loop, sokieer, shorten, lengthen, snip
widen, unlock, brick
Dim 10
9.26%
degree red
Dim 57
42.78% cultivate, grow, plant, prune, propogate, gather, ripen, dry, pick, graft squirt, formulate, dilute, smear, dissolve, inject, evaporate, ocze, drip, apply
Dim 6
26.25%
Dim
extinguish, light kindle, relindle, flare, ignito, shine, dim, flash, douse strangulate, fumble, glove, punt bunt, bek, vulcanize, hit slog, balloon weave, drape, embroider, knitt sew, hang, stitch, wrap, wave, unfold
Dim
turry
Dim 55
100.0% lour, eddy, scud, glower, taxi, dissipate, roil, cahale, lonize, betch
Figure 13: Components of SPoSE dimension approximation (F)
24


Reviews:
[{"review_id": "o3brdGr9oln", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper attempts to learn embeddings for objects based on their affordances i.e., verbs that could be applied to them to realise their meaning. Here each dimension corresponds to an affordance or an aspect of meaning shared by actions, thus allowing a correspondence between nouns (objects) and verbs (their affordances) based on co-occurrences in text corpora in which they exist. Empirical results show that these embeddings allowing prediction of a “mental representation” of objects (i.e., in comparison to human-given annotations of dimension “semantics” in embeddings) and a qualitative analysis attempts to show how interpretable the objects are.\n\nReason for score:\n\nThis paper approaches an interesting problem, but is not well-placed in literature and has missed previous work that attempts to do almost the same thing; however from a different angle. I thought the question and problem was interesting enough, but given the missed references and existing embedding-learning work, there is limited novelty in this approach. However, I think the results are interesting enough and the authors did a really nice job of qualitatively analysing the representations (and additionally, it would be good to see further discussion of use-cases of this e.g., instead of just a verb-ranking task), so my score is fairly positive overall. \n\nPositive points + questions:\n\n1. This paper is well written and the methodology is clearly explained. \n\n2. The empirical results show that verb rankings obtained by these embeddings outperform all previous embeddings (however those were not learned with this objective in mind, but just tested on the verb-ranking task). \n\n3. The results on the SPoSE task (predicting which dimensions correspond to which affordances) shows that the correlations between true and predicted dimensions are high.\n\n4. The qualitative figures and examples are very insightful and highlight te promise of this object-verb objective to learn embeddings that highlight affordances and useful semantic properties of objects. \n\n5. On that note, it would be really interesting to see if this helps downstream tasks e.g., not just an object-verb ranking task, but a more general task (however one that does require reasoning about objects and verbs together in order to correctly solve some decision making/classification problem)\n\n\nNegative points + questions:\n\n1. Missing references: https://roboticsconference.org/program/papers/80/\n\n2. The approach of attributing/grouping together verbs for specific dimensions seems less intuitive (and more restrained) than just allowing the embedding space to be learned by trying to make verb-object embedding representations more similar based on objective optimised for similarity/distance between embeddings. It is also further restrained given that novel verbs (unseen during training) may not be accounted for.\n\n3. How reliable is the processing of the corpora (e.g., tokenising, bigrams, using Stanza, dependency parsing); were these sanity-checked to assess if they were correct and the amount of noise that exists? Previous work that has attempted to do this from e.g., CommonCrawl/Wikipedia data found a range of errors/inconsistencies because of the domain shift and differences in the style/type of language commonly found on the internet, thus resulting in object-verb pairs that were too noisy for predicting/training purposes. It would be helpful to see examples + a manually annotated portion for correctness.\n\n4. The correlation results don’t have statistical significance tests/metrics and that would be helpful to see.\n\n5. It would be interesting to see/discuss if the bigrams cause any change in performance (e.g., maybe only allowing unigrams changes the distribution of words and therefore object, verb pairs that might affect results slightly).\n\n6. More importantly, given how prevalent subword embeddings are now, it would be interesting to see if that could be incorporated here---for e.g., if a byte pair encoding algorithm was first run over the corpus and nouns/verbs were split according to those, does this still hold? This seems important given that subword embeddings are now used (and perform better) for most of the best performing models, so if a method that allows better object-verb disambiguation could be used to kickstart tasks that require subword embeddings, that would be helpful to see! This seems very doable within this framework with a few minor changes..\n\n\nAdditional minor comments:\n\n1. It is interesting that the object/verbs mined from datasets have the number of verbs nearly twice those of the objects (previous work/datasets seem to have a smaller number of verbs given the overlap of similar verbs for the same object). This begs the question of whether or not similar verbs can be collapsed into one another/and also a qualitative analysis of whether these are grouped together and can be predicted alongside.\n\n2. It would be good to see statistical significance tests for metrics.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gh-kPC8iQPh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General comments\n---\n\nThis paper uses a factorisation of a verb-object co-occurrence count matrix to predict which verbs are applicable to which objects. This idea is related to the classical notion of an \"affordance\" from Gibson. The method is evaluated on a number of recent affordance datasets, obtaining better performance than a number of baseline systems.\n\nThe main problem I have with the paper as it stands is that it's not clear what the overall goal of the work is. A minor problem is that the method used appears to be entirely standard, so it's unclear what the technical contribution is. A further minor problem is that there is a whole related sub-field of computational linguistics which has been investigating a similar problem for decades which is ignored in the discussion.\n\nThe main problem: is the goal to develop a psychologically plausible cognitive model? Are we doing cogsci here? Or is it to build a knowledge base that can be used by an AI system (so more on the engineering side)? But if the latter, how would the knowledge be used, and by what sort of AI system? Is the knowledge to be used by a text-based system (if so, how) or by a situated agent interacting with an environment (in which case it needs explaining how the knowledge could be grounded in the agent's environment)?\n\nThe minor problem is that factorisation of the co-occurrence matrix appears just as standard as the other methods compared against, eg in sec. 4.1. So why are the other techniques any more baselines than yours?\n\nThe further minor problem is that the sub-field of acquiring selectional preferences in computational linguistics looks to be solving the same problem as what you have here. Classic references are Wilks from the 1970s and Resnik from the 1990s. \n\nMore specifically, there's a lot of existing work on taking a set of verb-object pairs and clustering the data in some way. This paper from 2010 is a good one to look at, and has lots of relevant references:\n\nLatent variable models of selectional preference\nDiarmuid O Seaghdha\nACL 2010\n\nMore specific comments\n--\n\nwe show that the dimensions can be used to predict a state-of-the-art\nmental representation of objects - it's not clear that the\nrepresentation itself is s-o-t-a; I suspect you mean that you obtained\ns-o-t-a performance on an existing object-representation dataset.\n\n\"Gibson (2014) coined the term “affordance” to describe what the\nenvironment\" - the term was coined by Gibson was much earlier, 1979?\n\n\"We will refer to objects and the nouns naming them interchangeably.\" -\nnot sure what you mean here: is it that you either say \"someone can\nstroke a cat\" or the verb \"stroke\" can apply to the noun \"cat\"? what's\nthe significance of the difference?\n\nwhen describing the various datasets, eg sec. 3.1, some examples would\nhelp.\n\nTypos etc.\n--\n\nThe labels in Figure 1 are too small to read.\n\nto a particular ”mode of interaction” - left quotes\n\n defined as ”affordance mining” - left quotes\n\nIn this paper, we use the list of 1854 object concepts - not sure why\nthe number is in bold.\n\n The resulting list has 2541 verbs - not sure why\nthe number is in bold.\n\nV is the verb loading for each of the d dimensions - \"loading\" is an\nodd term to use here, maybe \"weighting\"?\n\n 2)purpose\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eo1XfstqSgH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors design a distributional word embedding method inspired by Gibsonian theories of perception. They use matrix factorization techniques to derive low-rank object representations in what they call an \"affordance space,\" linking each object to aspects of meaning shared among different types of physical actions. They argue that the learned representations are interpretable, and that this affordance space \"underlies the mental representation of objects.\"  \n\nI unfortunately found the paper both conceptually and methodologically flawed. These criticisms fall mainly under the \"Quality\" and \"Significance\" categories, expanded below. First, a summary in pros/cons:  \n\nPros: Designs cognitively-motivated knowledge representations; leverages a diverse set of experiments to better understand and defend these representations.\n\nCons: Conceptual flaws about the content of the derived representations; evaluations are insufficient to support the claims of the paper.  \n\nQuality\n\nThis paper suffers from both conceptual and methodological issues.\n\n1. The claimed \"affordance space\" is not falsifiably \\*about\\* affordances in any deep sense. While the original data matrix linking words and their associated attested verb combinations clearly gets at possible event--object interactions, the factorized affordance space doesn't necessarily have this property. The lower-dimensional basis may span the space according to \"modes of interaction\" as claimed, but equally likely may describe coherent categories of contexts/places in which the actions occur, or categories of agents which perform the action, for example.I actually see three facts reported in the paper that make me think the derived data isn't about affordances per se. First, figure 2b actually shows that some of the dimensions of the affordance space best correlated with SPoSE dimensions are object-taxonomic properties. Second, the evaluation based on the raw affordance matrix (called \"PPMI\" in Table 1) underperforms the full model by a substantial amount, suggesting that the factorization introduces information not captured in the actual affordance data. Third and possibly most importantly, table 2 confirms that \"structural\" and \"appearance\" features are some of the best predicted features from the affordance space.The authors may argue that the set of English verbs used in the raw matrix are not the right basis for affordance knowledge, and that the factorization leads to a better abstract/conceptual affordance knowledge representation less tied to linguistic productions. But this claim about the content of the factorized representation needs to be articulated and substantiated with tests of alternative hypotheses.As a quick analogy in case my point isn't clear: you might learn word embeddings on a Wikipedia dump by factorizing a matrix of word--Wikipedia topic co-occurrence counts. The resulting low-dimensional representations aren't \\*about Wikipedia topics\\* in any deep sense, no matter the factorization method --- we simply talk about them as distributional meaning representations.\n2. Regarding the methodology of evaluation 2: is your aim to demonstrate the necessity and sufficiency of affordance knowledge for object feature knowledge? The evaluation demonstrates a rough sort of sufficiency, but not necessity. Demonstrating necessity would require testing alternate representations, I think, which isn't reported. What do you think about this premise/issue?\n3. Evaluation 1 doesn't seem very meaningful to me. It seems self-evident that representations constructed on the basis of verb--object co-occurrence data will perform well in predicting object--action co-occurrences, and probably better than representations which are not specifically tuned exclusively for that language task. (I agree that it's nontrivial that a corpus-derived representation would suffice here, but I don't find it interesting that it outperforms other more general / less task-specific corpus-derived representations.)  \n\nSignificance\n\nThe aim of this paper is not clear to me. It cannot argue for a superior system of word representation, since it does not evaluate these representations on any broad evaluation tests. It also doesn't make a convincing cognitive argument about the content of mental representations, given the conceptual and methodological issues in evaluation 2, discussed above. (A convincing cognitive argument would also need to draw on data from human behavior beyond the sort gathered on AMT, or possibly neural evidence; see Mitchell et al. (2008) as an example.)  \n\nOriginality\n\nBecause I haven't closely followed the relevant literature, I can't speak to the originality of the embedding method. That being said, it doesn't seem like a substantial conceptual innovation to me. I would be more motivated to let this slide if the paper were stronger on the experimental / analytic side.  \n\nClarity\n\nThe paper is clearly written and the authors provide plenty of supporting supplemental information. Some minor comments on this front:\n\n* Figure 1 is not very useful, either for assessing success of the method or for understanding its shortcomings. For the latter purpose, maybe consider showing the \\*residuals\\* of the regression, so we can understand where affordance information performs relatively better/worse across SPoSE dimensions?\n* Gibson (2014) citation should be Gibson (1979).  \n\nMitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. science, 320(5880), 1191--1195\\.\n\n## Post-rebuttal response\n\nI have read the other reviews and the authors' extremely thorough responses — much appreciated! See the thread below for some brief responses to the rebuttal sections in turn.\nI regret posing far too high a standard in my original review. The authors' rebuttals have helped to quiet my doubts a bit, and better understand the utility of this paper as a product for cognitive science. I have accordingly revised my judgment quite a bit upward.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YNLwLe5sqOe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors develop object representations based on the concept of affordances, making use of a dependency parsed text corpus and a factorization of the PPMI matrix of noun-verb pairs. They show the proposed approach is able to predict human judgements of object affordances better than distributional methods and LSA. They further show that the novel representations correlate well to a set of interpretable representations that were obtained via human judgements of object similarity.\n\nMain contributions:\n1. A method of learning an embedding of objects from an unannotated text corpus which is infused with a degree of knowledge of object affordance.\n2. An analysis of the relationship between the proposed embeddings and SPoSE embeddings.\n\nStrengths:\n1.  The proposed approach is simple and does not require any form of complex annotation. This enables the consideration of a large set of verbs compared to approaches which are based on manually created datasets.\n2. The analysis of interpretability is well thought out. The proposed embeddings display high predictive abilities for a majority of SPoSE dimensions, suggesting that this method might offer a good model of the mental representation of objects.\n\nWeaknesses:\n1. One crucial aspect which I feel the authors neglected to adequately address is the aim of the work, and its practical applicability. What is the significance of the proposed method, beyond its ability to predict a different set of representations? If these representations are meant to achieve a new state of the art, the evaluation is too limited and fails to include common methods in the literature, such as contextual embeddings. [This has since been addressed]\n2. Given the existence of methods that make use of visual features to predict affordances, I would have liked to see some form of comparison between image- and text-based methods.\n\n## Response to comments\n\nI thank the authors for their comments and their revision, which have clarified the aim of this work. Having read the authors' responses to this and other reviews, I realize that in my initial assessment I had misjudged the nature of the manuscript. After careful consideration, I have therefore increased my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a computational linguistic study of the semantics that can be inferred form text corpora given parsers (which are trained on human data) are used to infer the verbs and their objects in text. The reviewers agreed that the work was well executed, and that the experiments comparing the resulting representations to human data were solid. The method employed has little or no technical novelty (in my opinion, not necessarily a flaw), and it's not clear what tasks (beyond capturing human data) representations could be applied to (again, not a problem if the goal is to develop theories of cognition). 

The first draft of the work missed important connections to the computational linguistics literature, where learning about 'affordances for verbs' (referred to as 'selectional preferences') has long been an important goal. The authors did a good job of setting out these connections in the revised manuscript, which the reviewers appreciated. 

The work is well executed, and should be commended for relating ideas from different sub-fields in its motivation and framing. But my sincere view is that it does not meet the same standards of machine-learning or technical novelty met by other papers at this conference. It is unclear to me what the framing in terms of 'affordance' adds to a large body of literature studying the semantics of word embeddings, given various syntactically and semantically-informed innovations.  It feels to me like this work would have been an important contribution to the literature in 2013, but given the current state of the art in representation learning from text and jointly learning from text and other modalities, I would like to have seen some attempt to incorporate these techniques and bridge the gap between the notion of affordance in text/verbs (selectional preference) and Gibson's notion of object affordance (what you can do physically with an object) in experiments and modelling, not just in the discussion. Such a programme of research could yield fascinating insights into the nature of grounding, and the continuum from the concrete, which can be perceived and directly experienced, to the abstract, which must be learned from text. I encourage the authors to continue in this direction. An alternative is to consider submitting the current manuscript to venue where the primary focus is cognitive modelling, and accounting for human, behavioural data, and where there is less emphasis on the development of novel methods or models.

For these reasons, and considering the technical scope of related papers in the programme, I cannot fairly recommend acceptance in this case. 
Decision: Reject