Grounding Language in Multi-Perspective Referential Communication
Authors: name: Zineng Tang, name: Lingjun Mao, name: Alane Suhr

We introduce a task and dataset for referring expression generation and
comprehension in multi-agent embodied environments. In this task, two agents in
a shared scene must take into account one another's visual perspective, which
may be different from their own, to both produce and understand references to
objects in a scene and the spatial relations between them. We collect a dataset
of 2,970 human-written referring expressions, each paired with human
comprehension judgments, and evaluate the performance of automated models as
speakers and listeners paired with human partners, finding that model
performance in both reference generation and comprehension lags behind that of
pairs of human agents. Finally, we experiment training an open-weight speaker
model with evidence of communicative success when paired with a listener,
resulting in an improvement from 58.9 to 69.3% in communicative success and
even outperforming the strongest proprietary model.