SlowGuess/NAACL_2022_OCR huggingface dataset:
To summarize this experiment, given the same two referents, people interpreted one referent as correct in one context and the other as correct in another context, each according to specific norms that are activated in that context. This suggests that social norms activated by the context had enough modulatory influence to determine the interpretation of an ambiguous referring expression favored by the norm. As a consequence, not knowing the norms that apply in these context will likely lead to incorrect interpretations of referential expressions as other factors not necessarily congruent with the norm-based interpretation will be used for reference resolution, as the next section on current NLP tools will demonstrate.
4.1 Evaluating NLP Tools
To complement our empirical study, we evaluated several coreference and natural language processing tools on our experimental scenes to determine if they achieve human performance for norm-guided reference resolution tasks. These include Neural
NLP Tool	Context	C	Answer
NeuralCoref	Dining Room	A	coordination X
Dining Room	B	coordination X
Leather Shop	A	[the worktable] X
Leather Shop	B	[the worktable] X
Bookstore	A	coordination X
Bookstore	B	coordination X
Bathroom	A	non-referential X
Bathroom	B	non-referential X
CoreNLP	Dining Room	A	[dinner] X
Dining Room	B	[a shoe] ✓
Leather Shop	A	[the room] X
Leather Shop	B	[the room] X
Bookstore	A	[a toothbrush] X
Bookstore	B	[a magazine] ✓
Bathroom	A	[a magazine] X
Bathroom	B	[a toothbrush] ✓
AllenNLP	Dining Room	A	coordination X
Dining Room	B	[a shoe] ✓
Leather Shop	A	coordination X
Leather Shop	B	coordination X
Bookstore	A	coordination X
Bookstore	B	coordination X
Bathroom	A	coordination X
Bathroom	B	coordination X
GPT-3: Curie	Dining Room	A	coordination X
Dining Room	B	coordination X
Leather Shop	A	coordination X
Leather Shop	B	coordination X
Bookstore	A	non-referential X
Bookstore	B	coordination X
Bathroom	A	coordination X
Bathroom	B	coordination X
GPT-3: Davinci	Dining Room	A	coordination X
Dining Room	B	coordination X
Leather Shop	A	coordination X
Leather Shop	B	coordination X
Bookstore	A	[toothbrush] X
Bookstore	B	[toothbrush] X
Bathroom	A	[toothbrush] ✓
Bathroom	B	[toothbrush] ✓
Table 4: Evaluation of coreference tools on contexts that use a definite reference. The dining room scene and leather shop scene both use the referring expression remove it; the bookstore scene and home bathroom scene, similarly, use the referring expression pick it up. We report whether these tools can detect if it is referential and refers to the correct object.
Coref, an extension of SpaCy (Honnibal and Johnson, 2015) and based on (Clark and Manning, 2016), Stanford CoreNLP (Finkel et al., 2005), AllenNLP (Lee et al., 2017). We also evaluated the GPT-3 base models, Davinci and Curie, by OpenAI, (Brown et al., 2020) designed for text generation and question-answering tasks. For this experiment, we specifically focus on the scenes that use a definite referring expression such as pick it up, in the bookstore and home bathroom scenes, and remove it, in the dining room and leather shop scenes. For the GPT-3 models, we prompt the Davinci model with the phrase which one but do not prompt the Curie model with a question. We made this decision to probe the different capabilities of these models; for the Curie model, we chose not to prompt it to see if would coherently generate the rest of the text and the resolve the correct referent likely to follow from the referring expression in the command (e.g. pick it up, remove it); for the Davinci model, we tested the question-answering capabilities by providing a question. This evaluation did not use a similar question as the main study—Was this the correct object?—as understanding a yes-no response is more opaque and we wanted to see if it could return the referents. We also note that all of these models were used off the shelf without fine tuning.
Results are summarized in Table 4 where we list the referents that the tools selected to match the referring expression. We represent an entity in brackets and also note when the referent is the coordination of the two referents (e.g. a shoe and a spoon) or the referring expression was interpreted as non-referential. A check mark  $\checkmark$  denotes the correct referent and an  $\times$  denotes the incorrect referent.
The experimental conditions have a role, here, since the recency or distance of the referent serves as a traditional feature for coreference models in NLP. Swapping the ordering of the referents ensures that if a model resolves the correct referent, it is consistent and is more likely taking into account the context then the surface structure. This swapping method is similar to (Emami et al., 2018)'s evaluation of BERT on the KnowRef test set for consistency.
In the results, there are only three cases where the coreference models choose the right referent. CoreNLP and AllenNLP both correctly link [a shoe] and it in the dining room scene. For this condition (condition B), shoe is mentioned first in
the text. Although, once the objects are switched, the models choose the wrong object—CoreNLP selects [dinner] (none of the candidate referents) and AllenNLP selects both the shoe and the spoon in a coordination. CoreNLP seems to do the best by selecting another correct referent: [a magazine] in the bookstore scene. But it fails yet again once the objects are swapped.
The GPT-3 models perform poorly overall but the Davinci model, prompted by which one, gets closer to the right answer by picking out individual referents more often than the Curie model. Davinci is consistently incorrect in the bookstore scene but consistently correct in the bathroom scene, yielding the only correct result when the referents are switched. The correct referent selected in the bathroom, the toothbrush, was also selected in the bookstore for both conditions. This suggests that the model is biased towards picking the toothbrush over the magazine more generally.
For most of the tools doing reference resolution on these scenarios, we see a theme of referring back to the coordination of the two referents, when only one referent should be selected. Therefore, it is clear these results do not match human intuition for this specific reference resolution task and, more importantly, fail to understanding social norms in order to consistently infer the correct referent.
5 Discussion & Future Implementations
The coreference task performed by humans and the NLP tools show a striking difference in outcomes. Given the same context and text, people tended to agree on the correct referent. Since the examples were stripped of linguistic cues that would give away the referent, people relied on context and social norms based on the reasons they selected and the written explanations they provided. Notably, however, the inconsistent agreement across all scenes can be reconciled with the fact that social norms and conventions are not equally shared across all people. This is supported, in part, by the written responses too. Additionally, these results also suggest that not every norm is weighed the same; the deontic force—how strongly the norm is to be followed—potentially influences how the norm guides a behavior or interpretation and competes with other norms. Admittedly, a limitation of our study is that we do not explicitly categorize our hypothesized social norms and conventions in a gradable fashion, but future work will consider
deontic force for a more fine-grain understanding of social norms.
NLP tools, on the other end, tell a different story. Many of the tools specifically designed for coreference resolution failed to consistently select the correct referent. The more powerful NLP engines, such as GPT-3 model, also performed poorly. This shows that relying on such a system to resolve references in these contexts would be problematic. The Davinci model when prompted by the question which one? justifies its response with an explanation of grammatical appropriateness: If we use the noun that appears in the context, it is clear that the speaker is referring to the toothbrush. There is no other "it" in the sentence... We would never say, "Pick up the magazine." This is why it's important to know whether the noun is the subject or object of a sentence. This explanation echoes something meaningful about grammar, yet is faulty and unclear. Rather, this argument is produced from statistical correlations the system extracted from large corpora. Furthermore, the system has no understanding of norms or how to apply them. The potential danger, here, is that simply employing deep learning systems without giving them a sense of norms will lead such systems to also violate norms. While the consequences of breaking norms can range in severity, at the most extreme end, they can include harm to other people.
A norm aware reference resolution system, therefore, will not only help to disambiguate referents but help a system know what not to do. This is especially important with embodied agents whose actions in the real world will be influenced by its reference resolution capabilities and natural language understanding.
5.1 Implementation in Embodied Agents
Inspired by our experimental results, we outline a potential methodology for robots to use social norms and conventions in performing situated reference resolution. In order to make the inferences necessary for selecting the correct referent in our scenarios, a novel pragmatic component must be tightly integrated with vision and natural language processing in a robotic cognitive architecture. All three inputs will simultaneously contribute to the interpretation of a referring expression.
A pragmatic component will serve as a knowledge base specifically for social norms and it would require a baseline representation of norms, which
can be collected experimentally for a particular domain (Malle et al., 2020). Upon hearing natural language input from a co-located speaker, a robot will begin incrementally processing the natural language and look for a referring expression. At the same time, the visual system will scan the environment for two purposes: to search for perceptually salient objects that potentially match the referring expression and to trigger the setting to activate a set of norms. For example, spotting a fork, plate, or table, the robot can infer with greater probability that it is located in a dining room and cue an inventory of social norms operationalized as prescriptions and prohibitions. Some of these prescriptions, informally, might be: food or drinks are allowed on the table or you are allowed to sit at the dinner table. Alternatively, some prohibitions might be  $X$  items should not be on the dining table or food and drink should be contained on the dinner table.
Incremental processing will allow the robot to gradually look for potential referents in the scene and, if it finds potential candidates to match the referring expression, it will also consider the joint probability of each referent given the social norms. The social norms activated from the setting should contribute to the interpretation from the start, not only when an ambiguous situation arises, since they can modulate the interpretation at any point; as seen from our experimental results, regardless of the linguistic form of the referring expression, the social norms can flip the interpretation of the referent when everything else is constant. An advantage to using an inventory of social norms in this way is that they can eliminate potential referents right away. The strength of the norm, roughly corresponding to their deontic force, must be considered for a fine-grained application of norms as some norms will compete with each other. Additionally, it will be critical to understand what norms may or may not be overruled as not to cause harm to human users. While norm activation begins early on, it can continually update through visual and natural language input. If the robot is uncertain about the setting, for instance, it can ask clarifying questions to gain more information. This approach seems applicable in preventing harm where it might be better in many instances to ask questions in uncertain contexts than to overstep boundaries.
To walk through a situated reference resolution scenario, and use a scenario from our experiment, imagine someone commanding the robot: remove it.
Even if the speaker pauses after the verb remove, incremental processing begins to parse the utterance and the robot visually scans the environment for the setting and salient objects. The robots activates the norms stored in the social norm knowledge base and continues processing the input. Once the utterance is completely processed, the expression, it, is linked to either a shoe or a spoon. With no other cue from the linguistic input, the prohibition of shoes on dining room tables pushes the interpretation towards removing the shoe. The norm is determined to be strong enough for the robot to act and so it proceeds to remove the shoe. Thus, the robot successfully uses its norm knowledge base in tandem with its vision and natural language processing abilities, to handle what appears on the surface to be an underspecified referring expression.
We conducted a human subjects study to demonstrate how social norms can guide reference resolution. Given a text vignette and a referring expression stripped of linguistic cues, the majority of subjects confirmed the intended referent in each context and relied on knowledge of conventions to make their decision. In contrast, several NLP tools evaluated on the same examples consistently failed to select the correct referent. We argue that these NLP tools critically lack an understanding of conventions and social norms and should not be completely relied on for reference resolution as they can also violate norms.
Finally, we integrate our findings into designing a methodology for teaching robots to use social norms and conventions to perform situated reference resolution. In future work, we experiment with using visual scenes for activating norms and evaluate larger NLP models with fine-tuning to our task. Lastly, we will implement our methodology into a cognitive architecture and look more closely at how the gradability of social norms influences reference resolution.
We are grateful to James Pustejovsky for assisting with our experimental contexts, the anonymous reviewers for their helpful feedback, and the SMART scholarship for funding the first author.
Henk Aarts and Ap Dijksterhuis. 2003. The silence of the library: environment, situational norm, and social behavior. Journal of personality and social psychology, 84(1):18.
Cristina Bicchieri. 2005. The grammar of society: The nature and dynamics of social norms. Cambridge University Press.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Joyce Y Chai, Pengyu Hong, and Michelle X Zhou. 2004. A probabilistic approach to reference resolution in multimodal user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces, pages 70-77.
Jon Chamberlain, Massimo Poesio, Udo Kruschwitz, et al. 2008. Phrase detectives: A web-based collaborative annotation game. In Proceedings of the International Conference on Semantic Systems (I-Semantics' 08), pages 42-49.
Robert B Cialdini, Carl A Kallgren, and Raymond R Reno. 1991. A focus theory of normative conduct: A theoretical refinement and reevaluation of the role of norms in human behavior. In Advances in experimental social psychology, volume 24, pages 201-234. Elsevier.
Kevin Clark and Christopher D Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667.
Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2018. The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. arXiv preprint arXiv:1811.01747.
Jenny Rose Finkel, Trond Grenager, and Christopher D Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 363-370.
Joy E Hanna and Michael K Tanenhaus. 2004. Pragmatic effects on reference resolution in a collaborative task: Evidence from eye movements. Cognitive science, 28(1):105-115.
Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373-1378, Lisbon, Portugal. Association for Computational Linguistics.
Ronnie Janoff-Bulman, Sana Sheikh, and Sebastian Hepp. 2009. Proscriptive versus prescriptive morality: Two faces of moral regulation. Journal of personality and social psychology, 96(3):521.
D. Jurafsky and J.H. Martin. 2009. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall Series in Artificial Intelligence. Pearson Prentice Hall.
Doruk Kicikoglu, Richard Bartle, Jon Chamberlain, and Massimo Poesio. 2019. Wormingo: a true gamification approach to anaphoric annotation. In Proceedings of the 14th International Conference on the Foundations of Digital Games, pages 1-7.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. arXiv preprint arXiv:1707.07045.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Bertram F Malle, Eric Rosen, Vivienne B Chi, Matthew Berg, and Peter Haas. 2020. A general methodology for teaching norms to social robots. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 1395-1402. IEEE.
Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, and Dieter Fox. 2014. Learning from unscripted deictic gesture and language for human-robot interactions. In Twenty-Eighth AAAI Conference on Artificial Intelligence.
Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems (TiiS), 3(1):1-44.
Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), volume 1, pages 1778-1789. Association for Computational Linguistics.
Michael J Spivey, Melinda J Tyler, Kathleen M Eberhard, and Michael K Tanenhaus. 2001. Linguistically mediated visual search. Psychological science, 12(4):282-286.
Michael K Tanenhaus, Michael J Spivey-Knowlton, Kathleen M Eberhard, and Julie C Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217):1632-1634.
Luis Von Ahn. 2006. Games with a purpose. Computer, 39(6):92-94.
David Whitney, Miles Eldon, John Oberlin, and Stefanie Tellex. 2016. Interpreting multimodal referring expressions in real time. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3331-3338. IEEE.