SlowGuess/NAACL_2022_OCR huggingface dataset:
Perceptron. The perceptron approach evaluates the representational capacity of raw trajectory data as-is, with a lower bound of random stratified and soft upper bound of fully supervised. The perceptron performs relatively well for its simplicity, being only 7 points below the fully supervised upper bound. This suggests that the trajectory data itself encodes a significant amount of verb meaning, but leaves plenty of room for improvement.
Figure 6: Comparison of each approach by verb. As we are primarily interested in the effect of pretraining, all bars are shown relative to the performance of the fully supervised model. I.e., the fully supervised model (which represents a soft upper bound on how well one can do given raw state information without any pretraining) is visualized as 0, and all other models are visualized relative to that. Human performance is inner-annotator agreement (\% of workers who agree on the majority label).
Self-supervised pretraining. The pretraining + probe approach evaluates the ability of self-supervised models to encode verb meaning from trajectory data. This is equivalent to the perceptron approach, but with learned hidden representations as input rather than raw trajectory data. The pretrained model does outperform the perceptron, as well as the fully supervised approach. Fine-tuning only improves on this slightly, highlighting that self-supervised pretraining can yield representations that successfully encode verb meaning.
Breakdown by verb. Figure 6 shows a comparison of average precision for each verb. There are some patterns worth highlighting. In particular, we can categorize verbs into three main groups: trivial, tractable, and hard.
Trivial verbs are verbs that can are well-represented by trajectory data as-is, i.e. those with high performance with the perceptron approach. These include fall, fall off, fall over and pick up.6 Many of these have high agreement, and may be explained by the object's change in height.
Tractable verbs are those that see significant benefit from pretraining, including slide, roll, throw,
toss, put down, turn, flip, and stop. An intuition behind this is that these verbs involve manner distinctions, and in particular, rotations of the object relative to itself. Such information doesn't fall directly out of raw state descriptions, but is likely to be well modeled by a pretraining objective that tries to predict the object's future position.
Hard verbs are those with low performance that don't benefit much from pretraining. These include bounce, drop, tip, topple, and spin. Many of these are verbs which have lower agreement. Bounce, slap and spin appear to benefit a bit from both pretraining and fine-tuning, suggesting that they may be tractable with similar but more robust pretraining. Tip and topple have fairly high performance, and may almost be categorized as trivial, perhaps being explained by the object's change in rotation. However, they are noticeably lower than other trivial verbs, despite seeing no benefit from pretraining, suggesting that there is nuance to their meaning in the dataset, which isn't captured by any approach. Finally, drop is a great example of a hard verb, as it is similar to trivial verbs like fall. However, drop involves interaction between the agent and object that is highly agreed upon by annotators, but doesn't appear to be captured by our approaches, despite the model receiving both object and agent data. More challenging examples may be able to
<sup>6</sup>We exclude hit, push, and bump from trivial verbs, as these have high average precision with random stratified, showing that they are very positively skewed, not necessarily well-represented.
unveil a similar story for other verbs of interaction like pick up and put down.
We test the hypothesis that verb meanings can be grounded in 3D trajectories, i.e., the position of objects over time. Specifically, we investigate the extent to which representations of object trajectories, learned without any linguistic supervision, naturally encode concepts that align to English verb semantics. Our primary contributions are twofold. First, we build a procedurally generated agent-object-interaction dataset for which we collect crowdsourced annotations. This is the first dataset of its kind, and provides a rich inventory of human judgments about the extensions of 24 verbs of motion. Second, we compare a variety of representation learning approaches, specifically contrasting approaches which operate directly on perceptual inputs to approaches which learn abstractions over the raw perception (via pretraining). We find that some verbs meanings (e.g., fall and push) are captured easily by the raw state information, while others (e.g., roll and turn) require additional processing to be represented well.
This work is a first step toward exploring ways to capture fine-grained distinctions in grounded verb semantics that are trivial for humans, but challenging for models. Recent benchmarks at the intersection of NLP, vision and robotics (Deitke et al., 2020; Shridhar et al., 2020b) illuminate unsolved challenges in AI that demand a more robust understanding of verb semantics and spatial reasoning. As these benchmarks continue to be developed, and rich multimodal datasets from technologies like virtual reality become increasingly abundant, we envision that future work in this vein will be especially relevant.
In the future, we plan to explore more sophisticated models for self-supervised pretraining, and evaluate how well these models transfer to more naturalistic language learning settings (Ebert and Pavlick, 2020). Beyond this, there is a large body of related research questions to be explored. For example, can representations of trajectory data be fused with visually-grounded representations to yield better encodings of verb semantics? Collaborative efforts will be key to addressing these next milestones in natural language understanding.
This work was supported by the DARPA GAILA program. Thank you to George Konidaris, Carsten Eickhoff, Roman Feiman, Jack Merullo, Charles Lovering, Gabor Brody, and the members of the Brown LUNAR lab for helpful discussion and feedback.
Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021. Prost: Physical reasoning of objects through space and time. arXiv preprint arXiv:2106.03634.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Benjamin K Bergen. 2012. Louder than words: The new science of how the mind makes meaning. Basic Books.
Benjamin K Bergen, Shane Lindsay, Teenie Matlock, and Srini Narayanan. 2007. Spatial and linguistic aspects of visual imagery in sentence comprehension. Cognitive science, 31(5):733-764.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics.
Anna M Borghi and Lucia Riggio. 2009. Sentence comprehension and simulation of object temporary, canonical and stable affordances. *Brain Research*, 1253:117-128.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165.
Elia Bruni, Giang Binh Tran, and Marco Baroni. 2011. Distributional semantics from text and images. In Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics,
pages 22-32, Edinburgh, UK. Association for Computational Linguistics.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. 2018. Learning to detect human-object interactions.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. 2020. Robothor: An open simulation-to-real embodied ai platform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3164-3174.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bonnie Dorr and Mari Broman Olsen. 2018. Lexical conceptual structure of literal and metaphorical spatial language: A case study of "push". In Proceedings of the First International Workshop on Spatial Language Understanding, pages 31-40.
Dylan Ebert and Ellie Pavlick. 2020. A visuospatial dataset for naturalistic verb learning. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 143-153, Barcelona, Spain (Online). Association for Computational Linguistics.
Jerome Feldman. 2008. From molecule to metaphor: A neural theory of language. MIT press.
Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243. Springer.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. 2020. Threadworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954.
Peter Gärdenfors. 2019. Using event representations to generate robot semantics. ACM Transactions on Human-Robot Interaction (THRI), 8(4):1-21.
Francesco Giuliani, Irtiza Hasan, Marco Cristani, and Fabio Galasso. 2021. Transformer networks for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 10335-10342. IEEE.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2019. Action genome: Actions as composition of spatio-temporal scene graphs.
Sinan Kalkan, Nilgün Dag, Onur Yürüten, Anna M Borghi, and Erol Şahin. 2014. Verb concepts from affordances. Interaction Studies, 15(1):1-37.
Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. 2017. Learning visually grounded sentence representations. arXiv preprint arXiv:1707.06320.
Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. pages 153-163.
Claudia Mazzuca, Chiara Fini, Arthur Henri Michalland, Ilenia Falcinelli, Federico Da Rold, Luca Tummolini, and Anna M Borghi. 2021. From affordances to abstract words: The flexibility of sensorimotor grounding. *Brain Sciences*, 11(10):1304.
James Pustejovsky and Nikhil Krishnaswamy. 2014. Generating simulations of motion events from verbal descriptions. In Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM* 2014), pages 99-109, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
Benjamin Radford. 2020. Seeing the forest and the trees: Detection and cross-document coreference resolution of militarized interstate disputes. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 35-41, Marseille, France. European Language Resources Association (ELRA).
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36.
Kumar Shridhar, Harshil Jain, Akshit Agarwal, and Dennis Kleyko. 2020a. End to end binarized neural networks for text classification. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 29-34, Online. Association for Computational Linguistics.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020b. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749.
Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423-1433, Jeju Island, Korea. Association for Computational Linguistics.
Jeffrey Mark Siskind. 2001. Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. Journal of artificial intelligence research, 15:31-90.
Mark Steedman. 2002. Plans, affordances, and combinatory grammar. Linguistics and Philosophy, 25(5):723-753.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766.
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. 2018. Actor-centric relation network.
Emre Ugur, Erol Sahin, and Erhan Oztop. 2009. Predicting future object states using learned affordances. In 2009 24th International Symposium on Computer and Information Sciences, pages 415-419. IEEE.
Lei Wang, Yangyang Xu, Jun Cheng, Haiying Xia, Jianqin Yin, and Jiaji Wu. 2018. Human action recognition by learning spatio-temporal features with deep neural networks. IEEE access, 6:17913-17922.
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Yan Zhang, Siyu Tang, He Sun, and Heiko Neumann. 2018. Human motion parsing by hierarchical dynamic clustering. In BMVC, page 269.
A Dataset parameters
The following tables describe the session-level and action-level parameters for our procedural data generation protocol described in Section 3.2.
Parameter	Description	Random Values
Start Location	Initial agent location	random from waypoint set
Start Rotation	Initial agent rotation in degrees	(0,360)
Target Mesh	Shape of the object	cube/sphere/capsule/cylinder
Target Position	Initial object location	random location on counter
Target Rotation	Initial object rotation in degrees	(0,360)
Target Mass	Mass in kg of the object	(0.1,10)
Target Drag	Hinders object motion	(0,2)
Target Angular Drag	Hinders object angular motion	(0.1,1)
Dynamic Friction	Friction when object is moving	(0,1)
Static Friction7	Friction when object is not moving	(0,1)
Bounciness	Energy retained on bounce	(0,1)
Table 2: Session-level parameters, which add variety between 3-minute sessions.
Parameter	Description	Values
Pick Speed	Hand velocity for Pick motion	(1,3)
Put Speed	Hand velocity for Put motion	(1,3)
Push Speed	Hand velocity for Push motion	(1,3)
Throw Force	Object force for Throw motion	(25,125)
Hit Force	Object force for Hit motion	(25,125)
Table 3: Action-level parameters, which add variety in the execution of each action primitive in the dataset.