Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper attempts to learn embeddings for objects based on their affordances i.e., verbs that could be applied to them to realise their meaning. Here each dimension corresponds to an affordance or an aspect of meaning shared by actions, thus allowing a correspondence between nouns (objects) and verbs (their affordances) based on co-occurrences in text corpora in which they exist. Empirical results show that these embeddings allowing prediction of a “mental representation” of objects (i.e., in comparison to human-given annotations of dimension “semantics” in embeddings) and a qualitative analysis attempts to show how interpretable the objects are.

Reason for score:

This paper approaches an interesting problem, but is not well-placed in literature and has missed previous work that attempts to do almost the same thing; however from a different angle. I thought the question and problem was interesting enough, but given the missed references and existing embedding-learning work, there is limited novelty in this approach. However, I think the results are interesting enough and the authors did a really nice job of qualitatively analysing the representations (and additionally, it would be good to see further discussion of use-cases of this e.g., instead of just a verb-ranking task), so my score is fairly positive overall. 

Positive points + questions:

1. This paper is well written and the methodology is clearly explained. 

2. The empirical results show that verb rankings obtained by these embeddings outperform all previous embeddings (however those were not learned with this objective in mind, but just tested on the verb-ranking task). 

3. The results on the SPoSE task (predicting which dimensions correspond to which affordances) shows that the correlations between true and predicted dimensions are high.

4. The qualitative figures and examples are very insightful and highlight te promise of this object-verb objective to learn embeddings that highlight affordances and useful semantic properties of objects. 

5. On that note, it would be really interesting to see if this helps downstream tasks e.g., not just an object-verb ranking task, but a more general task (however one that does require reasoning about objects and verbs together in order to correctly solve some decision making/classification problem)


Negative points + questions:

1. Missing references: https://roboticsconference.org/program/papers/80/

2. The approach of attributing/grouping together verbs for specific dimensions seems less intuitive (and more restrained) than just allowing the embedding space to be learned by trying to make verb-object embedding representations more similar based on objective optimised for similarity/distance between embeddings. It is also further restrained given that novel verbs (unseen during training) may not be accounted for.

3. How reliable is the processing of the corpora (e.g., tokenising, bigrams, using Stanza, dependency parsing); were these sanity-checked to assess if they were correct and the amount of noise that exists? Previous work that has attempted to do this from e.g., CommonCrawl/Wikipedia data found a range of errors/inconsistencies because of the domain shift and differences in the style/type of language commonly found on the internet, thus resulting in object-verb pairs that were too noisy for predicting/training purposes. It would be helpful to see examples + a manually annotated portion for correctness.

4. The correlation results don’t have statistical significance tests/metrics and that would be helpful to see.

5. It would be interesting to see/discuss if the bigrams cause any change in performance (e.g., maybe only allowing unigrams changes the distribution of words and therefore object, verb pairs that might affect results slightly).

6. More importantly, given how prevalent subword embeddings are now, it would be interesting to see if that could be incorporated here---for e.g., if a byte pair encoding algorithm was first run over the corpus and nouns/verbs were split according to those, does this still hold? This seems important given that subword embeddings are now used (and perform better) for most of the best performing models, so if a method that allows better object-verb disambiguation could be used to kickstart tasks that require subword embeddings, that would be helpful to see! This seems very doable within this framework with a few minor changes..


Additional minor comments:

1. It is interesting that the object/verbs mined from datasets have the number of verbs nearly twice those of the objects (previous work/datasets seem to have a smaller number of verbs given the overlap of similar verbs for the same object). This begs the question of whether or not similar verbs can be collapsed into one another/and also a qualitative analysis of whether these are grouped together and can be predicted alongside.

2. It would be good to see statistical significance tests for metrics.


review_title: An interesting problem, but limited novelty (and missed similar, related work) for a focused problem. However, qualitative and quantitative results look good and if this could be proven useful for more than just the verb-object prediction task evaluated in the paper, this could be a good contribution!

final_score: rating: 7: Good paper, accept
review_content: Summary: The authors develop object representations based on the concept of affordances, making use of a dependency parsed text corpus and a factorization of the PPMI matrix of noun-verb pairs. They show the proposed approach is able to predict human judgements of object affordances better than distributional methods and LSA. They further show that the novel representations correlate well to a set of interpretable representations that were obtained via human judgements of object similarity.

Main contributions:
1. A method of learning an embedding of objects from an unannotated text corpus which is infused with a degree of knowledge of object affordance.
2. An analysis of the relationship between the proposed embeddings and SPoSE embeddings.

Strengths:
1.  The proposed approach is simple and does not require any form of complex annotation. This enables the consideration of a large set of verbs compared to approaches which are based on manually created datasets.
2. The analysis of interpretability is well thought out. The proposed embeddings display high predictive abilities for a majority of SPoSE dimensions, suggesting that this method might offer a good model of the mental representation of objects.

Weaknesses:
1. One crucial aspect which I feel the authors neglected to adequately address is the aim of the work, and its practical applicability. What is the significance of the proposed method, beyond its ability to predict a different set of representations? If these representations are meant to achieve a new state of the art, the evaluation is too limited and fails to include common methods in the literature, such as contextual embeddings. [This has since been addressed]
2. Given the existence of methods that make use of visual features to predict affordances, I would have liked to see some form of comparison between image- and text-based methods.

## Response to comments

I thank the authors for their comments and their revision, which have clarified the aim of this work. Having read the authors' responses to this and other reviews, I realize that in my initial assessment I had misjudged the nature of the manuscript. After careful consideration, I have therefore increased my rating.
review_title: Clear exposition of a method with unclear applicability

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors design a distributional word embedding method inspired by Gibsonian theories of perception. They use matrix factorization techniques to derive low-rank object representations in what they call an "affordance space," linking each object to aspects of meaning shared among different types of physical actions. They argue that the learned representations are interpretable, and that this affordance space "underlies the mental representation of objects."  

I unfortunately found the paper both conceptually and methodologically flawed. These criticisms fall mainly under the "Quality" and "Significance" categories, expanded below. First, a summary in pros/cons:  

Pros: Designs cognitively-motivated knowledge representations; leverages a diverse set of experiments to better understand and defend these representations.

Cons: Conceptual flaws about the content of the derived representations; evaluations are insufficient to support the claims of the paper.  

Quality

This paper suffers from both conceptual and methodological issues.

1. The claimed "affordance space" is not falsifiably \*about\* affordances in any deep sense. While the original data matrix linking words and their associated attested verb combinations clearly gets at possible event--object interactions, the factorized affordance space doesn't necessarily have this property. The lower-dimensional basis may span the space according to "modes of interaction" as claimed, but equally likely may describe coherent categories of contexts/places in which the actions occur, or categories of agents which perform the action, for example.I actually see three facts reported in the paper that make me think the derived data isn't about affordances per se. First, figure 2b actually shows that some of the dimensions of the affordance space best correlated with SPoSE dimensions are object-taxonomic properties. Second, the evaluation based on the raw affordance matrix (called "PPMI" in Table 1) underperforms the full model by a substantial amount, suggesting that the factorization introduces information not captured in the actual affordance data. Third and possibly most importantly, table 2 confirms that "structural" and "appearance" features are some of the best predicted features from the affordance space.The authors may argue that the set of English verbs used in the raw matrix are not the right basis for affordance knowledge, and that the factorization leads to a better abstract/conceptual affordance knowledge representation less tied to linguistic productions. But this claim about the content of the factorized representation needs to be articulated and substantiated with tests of alternative hypotheses.As a quick analogy in case my point isn't clear: you might learn word embeddings on a Wikipedia dump by factorizing a matrix of word--Wikipedia topic co-occurrence counts. The resulting low-dimensional representations aren't \*about Wikipedia topics\* in any deep sense, no matter the factorization method --- we simply talk about them as distributional meaning representations.
2. Regarding the methodology of evaluation 2: is your aim to demonstrate the necessity and sufficiency of affordance knowledge for object feature knowledge? The evaluation demonstrates a rough sort of sufficiency, but not necessity. Demonstrating necessity would require testing alternate representations, I think, which isn't reported. What do you think about this premise/issue?
3. Evaluation 1 doesn't seem very meaningful to me. It seems self-evident that representations constructed on the basis of verb--object co-occurrence data will perform well in predicting object--action co-occurrences, and probably better than representations which are not specifically tuned exclusively for that language task. (I agree that it's nontrivial that a corpus-derived representation would suffice here, but I don't find it interesting that it outperforms other more general / less task-specific corpus-derived representations.)  

Significance

The aim of this paper is not clear to me. It cannot argue for a superior system of word representation, since it does not evaluate these representations on any broad evaluation tests. It also doesn't make a convincing cognitive argument about the content of mental representations, given the conceptual and methodological issues in evaluation 2, discussed above. (A convincing cognitive argument would also need to draw on data from human behavior beyond the sort gathered on AMT, or possibly neural evidence; see Mitchell et al. (2008) as an example.)  

Originality

Because I haven't closely followed the relevant literature, I can't speak to the originality of the embedding method. That being said, it doesn't seem like a substantial conceptual innovation to me. I would be more motivated to let this slide if the paper were stronger on the experimental / analytic side.  

Clarity

The paper is clearly written and the authors provide plenty of supporting supplemental information. Some minor comments on this front:

* Figure 1 is not very useful, either for assessing success of the method or for understanding its shortcomings. For the latter purpose, maybe consider showing the \*residuals\* of the regression, so we can understand where affordance information performs relatively better/worse across SPoSE dimensions?
* Gibson (2014) citation should be Gibson (1979).  

Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. science, 320(5880), 1191--1195\.

## Post-rebuttal response

I have read the other reviews and the authors' extremely thorough responses — much appreciated! See the thread below for some brief responses to the rebuttal sections in turn.
I regret posing far too high a standard in my original review. The authors' rebuttals have helped to quiet my doubts a bit, and better understand the utility of this paper as a product for cognitive science. I have accordingly revised my judgment quite a bit upward.
review_title: Interesting motivation, but conceptually unclear and methodologically flawed

final_score: rating: 5: Marginally below acceptance threshold
review_content: General comments
---

This paper uses a factorisation of a verb-object co-occurrence count matrix to predict which verbs are applicable to which objects. This idea is related to the classical notion of an "affordance" from Gibson. The method is evaluated on a number of recent affordance datasets, obtaining better performance than a number of baseline systems.

The main problem I have with the paper as it stands is that it's not clear what the overall goal of the work is. A minor problem is that the method used appears to be entirely standard, so it's unclear what the technical contribution is. A further minor problem is that there is a whole related sub-field of computational linguistics which has been investigating a similar problem for decades which is ignored in the discussion.

The main problem: is the goal to develop a psychologically plausible cognitive model? Are we doing cogsci here? Or is it to build a knowledge base that can be used by an AI system (so more on the engineering side)? But if the latter, how would the knowledge be used, and by what sort of AI system? Is the knowledge to be used by a text-based system (if so, how) or by a situated agent interacting with an environment (in which case it needs explaining how the knowledge could be grounded in the agent's environment)?

The minor problem is that factorisation of the co-occurrence matrix appears just as standard as the other methods compared against, eg in sec. 4.1. So why are the other techniques any more baselines than yours?

The further minor problem is that the sub-field of acquiring selectional preferences in computational linguistics looks to be solving the same problem as what you have here. Classic references are Wilks from the 1970s and Resnik from the 1990s. 

More specifically, there's a lot of existing work on taking a set of verb-object pairs and clustering the data in some way. This paper from 2010 is a good one to look at, and has lots of relevant references:

Latent variable models of selectional preference
Diarmuid O Seaghdha
ACL 2010

More specific comments
--

we show that the dimensions can be used to predict a state-of-the-art
mental representation of objects - it's not clear that the
representation itself is s-o-t-a; I suspect you mean that you obtained
s-o-t-a performance on an existing object-representation dataset.

"Gibson (2014) coined the term “affordance” to describe what the
environment" - the term was coined by Gibson was much earlier, 1979?

"We will refer to objects and the nouns naming them interchangeably." -
not sure what you mean here: is it that you either say "someone can
stroke a cat" or the verb "stroke" can apply to the noun "cat"? what's
the significance of the difference?

when describing the various datasets, eg sec. 3.1, some examples would
help.

Typos etc.
--

The labels in Figure 1 are too small to read.

to a particular ”mode of interaction” - left quotes

 defined as ”affordance mining” - left quotes

In this paper, we use the list of 1854 object concepts - not sure why
the number is in bold.

 The resulting list has 2541 verbs - not sure why
the number is in bold.

V is the verb loading for each of the d dimensions - "loading" is an
odd term to use here, maybe "weighting"?

 2)purpose

review_title: Review for UNDERSTANDING MENTAL REPRESENTATIONS OF OBJECTS THROUGH VERBS APPLIED TO THEM

Meta-review: This paper is a computational linguistic study of the semantics that can be inferred form text corpora given parsers (which are trained on human data) are used to infer the verbs and their objects in text. The reviewers agreed that the work was well executed, and that the experiments comparing the resulting representations to human data were solid. The method employed has little or no technical novelty (in my opinion, not necessarily a flaw), and it's not clear what tasks (beyond capturing human data) representations could be applied to (again, not a problem if the goal is to develop theories of cognition). 

The first draft of the work missed important connections to the computational linguistics literature, where learning about 'affordances for verbs' (referred to as 'selectional preferences') has long been an important goal. The authors did a good job of setting out these connections in the revised manuscript, which the reviewers appreciated. 

The work is well executed, and should be commended for relating ideas from different sub-fields in its motivation and framing. But my sincere view is that it does not meet the same standards of machine-learning or technical novelty met by other papers at this conference. It is unclear to me what the framing in terms of 'affordance' adds to a large body of literature studying the semantics of word embeddings, given various syntactically and semantically-informed innovations.  It feels to me like this work would have been an important contribution to the literature in 2013, but given the current state of the art in representation learning from text and jointly learning from text and other modalities, I would like to have seen some attempt to incorporate these techniques and bridge the gap between the notion of affordance in text/verbs (selectional preference) and Gibson's notion of object affordance (what you can do physically with an object) in experiments and modelling, not just in the discussion. Such a programme of research could yield fascinating insights into the nature of grounding, and the continuum from the concrete, which can be perceived and directly experienced, to the abstract, which must be learned from text. I encourage the authors to continue in this direction. An alternative is to consider submitting the current manuscript to venue where the primary focus is cognitive modelling, and accounting for human, behavioural data, and where there is less emphasis on the development of novel methods or models.

For these reasons, and considering the technical scope of related papers in the programme, I cannot fairly recommend acceptance in this case. 
Decision: Reject