Understanding Embodied Reference with Touch-Line Transformer
Authors: Yang Li
Xiaoxue Chen
Hao Zhao
Jiangtao Gong
Guyue Zhou
Federico Rossano
Yixin Zhu
Venue: ICLR 2023 poster (2023)

Abstract: We study embodied reference understanding, the task of locating referents using embodied gestural signals and language references. Human studies have revealed that, contrary to popular belief, objects referred to or pointed to do not lie on the elbow-wrist line, but rather on the so-called virtual touch line. Nevertheless, contemporary human pose representations lack the virtual touch line. To tackle this problem, we devise the touch-line Transformer: It takes as input tokenized visual and textual features and simultaneously predicts the referent’s bounding box and a touch-line vector. Leveraging this touch-line prior, we further devise a geometric consistency loss that promotes co-linearity between referents and touch lines. Using the touch line as gestural information dramatically improves model performances: Experiments on the YouRefIt dataset demonstrate that our method yields a +25.0% accuracy improvement under the 0.75 IoU criterion, hence closing 63.6% of the performance difference between models and humans. Furthermore, we computationally validate prior human studies by demonstrating that computational models more accurately locate referents when employing the virtual touch line than when using the elbow-wrist line.

Published as a conference paper at ICLR 2023
UNDERSTANDING EMBODIED REFERENCE WITH TOUCH-LINE TRANSFORMER
Yang Li Xiaoxue Chen', Hao Zhao
Jiangtao Gong', Guyue Zhou', Federico Rossano", Yixin Zhu 303
Institute for Al Industry Research, Tsinghua University
2 Department of Cognitive Science, UCSD Institute for Artificial Intelligence, Peking University
Liyang@air.tsiighuaa edw.ghhaaaauuuuueooom cn, zhachaolaair.tsinghuut com,
yixin.zhuépku.edu.
bttps://yang-1i 1000000000000000..-- w.t....../rooc---Leeeaaaaaaaaaa
ABSTRACT
We study embodied reference understanding, the task of locating referents using embodied gestural signals and language references. Human studies have revealed that, contrary to popular belief, objects referred to pointed to do not lie on the elbow-wrist line, but rather on the so-called virtual touch line. Nevertheless, contemporary human pose representations lack the virtual touch line. To tackle this problem, we devise the touch-line Transformer: It takes as input tokenized visual and textual features and simultancously predicts the referent's bounding box and a touch-line vector. Leveraging this touch-line prior, we further devise a geometric consistency loss that promotes co-linearity between referents and touch lines. Using the touch line as gestural information dramatically improves model performances:
Experiments on the YouReflt dataset demonstrate that our method yields a +25.0% accuracy improvement under the 0.75 IoU criterion, hence closing 63.6% of the performance difference between models and humans. Furthermore, we computationally validate prior human studies by demonstrating that computational models more accurately locate referents when employing the virtual touch line than when using the elbow-wrist line.
INTRODUCTION
Understanding human intents is essential when intelligent robots interact with humans. Never- theless, most prior work in the modern learning community disregards the multi-modal facet of human-robot communication. Consider the sce- nario depicted in Figure wherein person instructs the robot to interact with chair behind the table. In response, the robot must compre- hend what humans referring before taking action (e.g., approaching the object and clean- ing it). Notably, both embodied gesture signals and language reference play significant roles. Without the pointing gesture, the robot could not distinguish between the two chairs using the utterance the chair that is occluded. Likewise, without the language expression, the robot could not differentiate the chair from other objects in that vicinity (e.g., bags on the table). To address this deficiency, we investigate the embodied ref- erence understanding (ERU) task introduced by Chen et al. (2021). This task requires algorithm to detect the referent (referred object) using (i) an image/video containing nonverbal communication signals and (ii) sentence verbal communication signal.
"The chair that is occluded.
Figure 1: To accurately locate the referent in plex scenes, both nonverbal and verbal expressions are vital. Without nonverbal expression (in this case, the pointing gesture), the verbal expression ("the chair") can- not uniquely refer to the chair because multiple chairs present this context. Conversely, cannot distin- guish the intended referent "the chair" from other nearby objects with only nonverbal expressions.

---

Published as a conference paper at ICLR 2023
The first fundamental challenge in tackling ERU is the representation of human pose. The de facto and prevalent pose representation in modern computer vision is defined by COCO (Lin et al. 2014) graph consisting of 17 nodes (keypoints) and 14 edges (keypoint connectivities). Existing models for ERU (Chen et al., 2021) assume pre-extracted COCO-style pose features to be the algorithm inputs. However, rethink the limitations of the COCO-style pose graph in the context of ERU and uncover counter-intuitive fact: The referent does not lie on the elbow-wrist line (i.e., the line that links the human elbow and wrist). As shown in Figure 2 this line (in red) does not cross the referred microwave, exhibiting a typical misinterpretation of human pointing (Herbort & Kunde 2018).
A recent developmental study (O Madaang et all1,,2011) presents compelling evidence sup- porting the above hypothesis. It studies how hu- mans mentally develop pointing gestures and argues that it virtual form of reaching out to touch. This new finding challenges conven- tional psychological views (McGinn| 1981; Kital 2003) that the pointing gesture is mentally a be- havior of using the limb as an arrow. Inheriting the terminology in O' Madagain we coin the red line in Figure 2 as an elbow- wrist line (EWL) and the yellow line (which
al. (2019).
Figure 2: Virtual touch line (VTL) (in green) vs. elbow-wrist line (EWL) (in red). VTLs affords more accurate location of referents than EWLs.
connects the eye and the fingertip) as a virtual touch line (VTL). Inspired by this essential ob- servation that vTLs are more accurate than EWLs in embodied reference, we augment the existing CoCo-style pose graph with an edge that connects the eye and the fingertip. As validated by a series of experiments in Section 4] this augmentation significantly improves the performance on YouReflt.
The second fundamental issue in tackling ERU is how to jointly model gestural signals and language references. Inspired by the success of multi-modal Transformers (Chen et al. 2020 Li et al.. [020 Tan
& Bansal 2019; Lu et al. 2019 Kamath et al. |021 in multi-modal tasks (Hudson & Manning [019
Antol et al. 2015 Zellers et al. 2019). we devise the Touch-Line Transformer. Our Transformer- based
model takes as inputs both visual and natural- natural-Ianguage modalities. Our Touch-Line Transformer jointly models gestural signals and language references by simultaneously predicting the touch-line vector and the referent's 's bounding box. To further help our model utilize gestural signals (i.e., the touch-line vector), we integrate geometric consistency loss to encourage co-linearity between the touch line and the predicted referent's location, resulting in significant performance improvements.
By leveraging the above two insights, our proposed method achieves +25.0% accuracy gain under the 0.75 IoU criterion on the YouRefIt dataset compared to state-of-the-an methods. Our approach closes 63.6% of the gap between model performance and human performance.
This paper makes four contributions by introducing (i) novel computational pose representation, VTL, (ii) the Touch-Line Transformer that jointly models nonverbal gestural signals and verbal references, (iii) a computational model leveraging the concept of touch line by novel geometric consistency loss that improves the co-linearity between the touch line and the predicted object, and (iv) new state-of-the tfte-of-thhearr performance on ERU, exceeding the 0.75 IoU threshold by +25.0%.
2 RELATED WORK
Misinterpretation of pointing gestures Pointing enables observers and pointers to direct visual attention and establish references communication. Recent research reveals. surprisingly. that observers make systematic errors (Herbort & Kunde 2016): While pointers produce gestures using VTL., observers interpret pointing gestures using the "arm-finger" line. O' Madagain et al. (2019) founded the VTL mechanism: Pointing gestures orient toward their targets as if the pointers were to touch them. In neuroscience, gaze effects occur for tasks that require gaze alignment with finger pointing (Bédard et al. 2008). The preceding evidence demonstrates that eye position and gaze direction are crucial for understanding pointing. Critically, Herbort & Kunde (2018) verify that directing human observers extrapolate the touch-line vector reduces the systematic misinterpretation during human hummn communication. Inspired by these discoveries, we incorporate the touch-line vector enhance the performance of pointing gesture interpretation.
2

---

Published as a conference paper at ICLR 2023
Detection of gaze targets The gazing gesture itself can refer to particular objects in the absence of language expressions. The detection of the gaze targets is the localization of the objects people gaze at. |Recasens et al. (2013) and Chong al. (2018) introduce two-stage methods that build saliency maps and estimate gaze directions before combining them to predict gaze targets. Tu et al. (2022) devise a one-stage end-to-end approach for simultaneously locating human heads and gaze targets for all individuals in an image. Fang et al. (2021) develop a three-stage method that predicts first the 2D and 3D gaze direction, then the field of views and depth ranges, and finally the gaze target. (Zao et al. (2020) assume that humans find gaze targets from salient objects on the sight line, designing a model to validate this hypothesis in images and videos. [i al. (202T) extend the detection of gaze target to 360-degree images. Chong et al. (020) and Recasens et al. (0177) extend the detection of gaze target into videos, wherein the gazed object may not be the same frame as the gazing gesture.
Referring expression comprehension Language expressions can also refer to particular objects. Referring expression comprehension is to locate referents in an image using only language expressions. Multiple datasets (e.g.. RefCOCO Yu et al. 2016), RefCOCO+ (Yu et al. 2016), RefCOCO-g (Mao) ct al. 2016), and Guesswhat (De Vries et al. (017)) ()) serve as the benchmark. To encode images and sentences, Du al. (2022) develop Transformer- basee framework and use the language to guide the extraction of discriminative visual features. Rohrbach et al. (2016) recognize the bidirectional nature between language expressions and objects in images, obviating the need for bounding box annotations. Yu et al. (2018) devise a modular network, wherein three modules attend to subjects, their location in images, and their relationships with nearby objects, respectively. While the above efforts locate referents using language expressions, they rarely use human-like nonverbal gestures.
Language-condition imitation learning Robots must identify target objects when completing goal-conditioned tasks (Stepputtis et al. 2020). Counterintuitively, SSteppttiis er al. (2020) argue against describing the target object using (one-hot) vectors due to inflexibility to support continued learning in deployed robots. Instead, SStppuutiis et al.] (202)) employ natural language to refer to target objects in goal-conditioned manipulation tasks. Specifically, leveraging semantic module for understanding what the referents referred to by language expressions, their architecture helps robots locate and attend to the target object when the natural language expression unambiguously refers to an object. However, this method struggles to resolve ambiguities (e.g., in cases with pointing gestures). Recent work on language-condinng
imitation learning (Stepputtis et al. 2020 Lynch & Sermanet 2021 Mees et al., (022)) may therefore benefit from our VTL by more accurately locating the target object, especially when the natural language alone is ambiguous.
METHOD
3.1 NETWORK ARCHITECTURE
Our framework, illustrated in Figure 3 consists of a multi-modal encoder, a Transformer decoder, and prediction heads. We describe each component in detail below.
Multimodal encoder We generate a visual embedding vector by extracting visual features from input images with a pre-trained ResNet (He et al. (016)) backbone, flattening them, and then adding them set of position embeddings (Parmar et al. 2018 Bello al. 2019). Mcanwhile, we generate a textural embedding vector from input texts using a pre-trained BERT (Liu et al., 2019). After obtaining visual and textural embedding vectors, we concatenate and feed them into a Transformer encoder to learn multi-modas representations.
Transformer decoder We feed the above modill representations our Transformer decoder, which additionally takes as input a set of learnable object queries and gestural key point queries. With molltllmodallii representations and input queries, our Transformer decoder (pre-trained MDETR (Kamath et al. 2021)) generates object output embeddings and gestural output embeddings.
Prediction heads Object and gestural output embeddings from the Transformer decoder are the inputs for our prediction heads (MLPs), which predict bounding boxes (for referents) and gestural key points. We keep one object bounding box and one pair of gestural key points with the highest scores as the final prediction. Specifically, we define the score of bounding box as 1 minus the non-object class column of the softmax of the predicted object logits. For a pair of gestural key points, the score is the is-a-V'I --a--VT/EEWL. column of the predicted arm logits' softmax.
3

---

Published as a conference paper at ICLR 2023
The on the table.
BERT
language Input
Language Embeddings
Object
Head
Concatenate
Transformer
Transformer
Encoder
Decoder
Eeceretiv Eanoiteryy
Positonnll Embreidings
Gesture
Head
CNN
Object Queries
Gesture
Queries
Visual
Embeddings
Figure 3: Overall network architecture. Language and visual inputs are first encoded by the text encoder and visual encoder to obtain language and visual embeddings, respectively, Next, these embeddings are concatenated and fed into the Transformer encoder learn multimodal representations. The Transformer decoder and prediction heads output the predicted bounding box and VTL/EWL. geometric consistency loss integrated to encourage the use of gestural signals.
3.2 EXPLICIT LEARNING OF NONVERBAL GESTURAL SIGNALS
Measure co-lineaerity Ideally, a referent should have a high co-linearity with the VTL. We measure this co-linearity using cosine similarity:
cos sim = CosineSimilarity((t) ze. Ve), (xo Te. Yo - ye)],
where (xy, My), (Ie, Me), and (æo. Vo) the x-y coordinates of the fingertip, the eye, and the center of the referent bounding box, respectively.
Encourage co-lineaerity We encourage model to predict referent bounding boxes that highly co-linear with VTLL using a referent alignment loss, which a geometric consistency loss:
Latign ReLUU(cos simgt cos simprea).
We compute cos simgt using the ground-truth referent boxes and compute cos simpprd using the predicted referent boxes. Ground-truth VTTLs used for both cos_simgt and cos_sim!pred Of note, the design of the offset cos_sing ensures the predicted object is as co-linear as the ground-trute object the VTL. In other words, the loss in Eqquaiion (2) minimized zero when the predicted object box is as co-linear as the ground-truth one to the VTL.
Modify for EWLs In experiments using EWLs instead of VTLs, we replace the fingertip (xy. My) and the eye (æe, Ve) in Equation (1) using the wrist (Zw, yw) and the elbow (I), respectively.
3.3 IMPLICIT LEARNING OF NONVERRAL GESTURAL SIGNALS
We eliminate the postural signals (by removing humans) from images to investigate whether our model implicitly learn utilize nonverbal gestural signals without being explicitly asked to learn. Please refer to Appendix A for details of our procedure.
3.4 ADDITIONAL LOSSES
We define the total loss during training as
Liotal Cbox A Agggssture + A Caliign + XgCroken + As
(3)
where λ;s are the weights various losses, Lbox is the weighted sum of the L1 and GloU losses for predicted referent bounding boxes, and Lgesture is the L1 loss for the predicted VTLs EWLs. The soft token loss Lroken and the contrastive loss Ceontrastive follow those in Kamath et al. (202T): these two losses help our model align visual and textural signals.
4

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASET, EVALUATION METRIC, AND IMPLEMENTATION DETAILS
We use the YouRefIt dataset (Chen 2021 with 2, 950 training instances and 1, 245 test instances. YouReflt has two portions: videos and images; we only use the image portion. The inputs are images without temporal information. Each instance contains an image, a sentence, and the location of the referent. We provide additional annotations of VTLs and EWLs for the YouReflt dataset.
For fair comparisons with prior models, we follow Chen al. (2021) and report precision under three different IoU thresholds: 0.25, 0.50, and 0.75. A prediction is correct under threshold if its loU with the ground-trun truth box is greater than that threshold. Additionally, we choose our best models by the precision under the GloU (Rezatofighi et al. (219)) threshold of 0.75. Compared with IoU, GIoU is an improved indicator of the model's 's ability to locate objects accurately. We report the results using GloU thresholds in Seccion 4.5
During training, we use the Adam variant, AMSGrad (Reddi et al. 2018). and train our models for 200 epochs. We use the AMSGrad variant because we observe slow convergence of the standard Adam optimizer in experiments. We set the learning rate to 5e-5 except for the text encoder, whose learning rate is le-4. We do not perform learning rate drops because we rarely observe demonstrable performance improvements after dropping them. We use NVIDIA A100 GPUs. The sum of the batch sizes on all graphic cards 56. Augmentations follow those in Kamath al. 12021). The total number of queries is 20; 15 for objects, and for gestural key points.
4.2 COMPARISON WITH STATE-OF-THE-AN METHODS
Our approach outperforms prior state-of-the-an art methods by 16.4%, 23.0%, and 25.0% under the IoU threshold of 0.25, 0.50, and 0.75, respectively; see Table Specifically, model performs better than visual grounding methods (Yang et al., 2019; 2020), which do not explicitly utilize nonverbal gestural signals. Our approach also performs better than the method proposed in YouReflt (Chen et al. 2021) which did not leverage the touch line or the Transformer models for molti-modal tasks.
Table I: Comparison with the state-of-the-ar methods.
1oU=.25
IoU=.50
IoU=.75
FAOA (Yang et al.
2019
44.5
30.4
8.5
ReSC
Yang el al.
2020)
49.2
34.9
10.5
YouRellI Pol-onty (Chen et al.
2021
52.6
37.6
12.7
YouReflt Full (Chen al. 2021
54.7
40.5
14.0
Ours (Inpainting)
59.1 (+4.4)
51.3 (+10.8)
32.4 (+18.4)
Ours (No explicit gestural key points)
64.9 (+10.2)
57.4 (+16.9)
37.2 (423.2)
Ours (EWL)
69.5 (+14.8)
60.7 (+20.2)
35.5 (+2155)
Ours (VTL)
71.1 (+16.4)
63.5 (+23.0)
39.0 (+25.0)
Human
94.2
85.8
53.3
4.3 EXPLICITLY LEARNED NONVERBAL SIGNALS
We report performances of models that explicitly pre- dict either the VTLs or the EWLs.
Table 2: Effeett of learning two different types of postural key points.
Results Overall, the model trained to predict the VTL. performs better than the one trained to predict the EWLs under all three IoU thresholds; see details
loU
None
EWL
VTL
0.25
64.9
69.5 (+4.6)
71.1 (+6.2)
0.50
57.4
60.7 (+3.3)
63.5 (+6.1)
in Table 2 The model trained explicitly predict the EWLs performs even worse than the one not trained to explicitly predict any gestural signals under the IoU threshold of 0.75.
0.75
37.2
35.5 (-1.7)
39.0 (+1.8)
Analysis Under the IoU threshold of 0.75, the unreliability of the arm's orientation and the stringent precision requirement can partly explain the worse performance of the model with explicit EWL predictions. As we observed in Figure 4 the EWLs are unreliable for predicting objects' locations simply because they frequently do not pass through the referents. This mismatch prevents the model
5

---

Published as a conference paper at ICLR 2023
VTLs The de
EWLs
VTLs
EWLS
VTLS
EWLs
The pho Eorrggt
(a)
(b)
(c)
Mytonntt cogo
My Cays
Picba Surger
Pebia
bett
Gat AEA
(h)
Charper
Durger
(k)
(I)
Figure 4: The model explicitly trained to predict vTLs more accurately locates referents than the one trained to predict EWLs. We draw green arrows (different from the predictions) illustrate eyes and fingertips more accuratcly indicate object locations. (Green sentence: natural language inputs; Green box: ground-truth referent location; Red box: predicted referent location; Red numbers near predicted box: GloU between the predicted box and the ground-truth box.)
from correctly determining the referents' locations using the EWLs' orientations. For instance, in Figure 4a (right), the EWL fails to pass the glass cup in the yellow box; it passes ceramic cup instead. The model struggles to identify the referent correctly with contradictory nonverbal (the ceramic cup) and verbal (the glass cup) signals.
In contrast, VTLS partly explains the model's improved performance. For example, in Figure 4b the VTL passes the comb in the green box while the EWL fails. Similarly, in other subtigures in Figure 4, VTLs passes the referent while EWLs do not, or VTLs are closer the referents' box centers than EWL are. The higher consistency between verbal and nonverbal signals increases the chance of successfully identifying the referents.
Under the lower IoU thresholds of 0.25 and 0.50, the rough orientations provided by the EWLs and the more lenient precision requirements can partly explain the improved model performances. Specifically, the rough orientations provided by the EWLs might help the model eliminate objects that significantly deviate from this direction. Hence, the model can choose from fewer objects, leading to higher probability of correctly locating referents.
4.4 IMPLICITLY LEARNED NONVERBAL SIGNALS
We investigate whether our model can implicitly learn nonverbal gestural signals without being explicitly asked to predict the VTLLs or the EWLs.
Table 3: Effects of implicitly
learned nonverbal signals.
IoU
Original
Inpainting
To investigate this hypothesis, we conducted two sets of experiments:
0.25
64.9
59.1 (-5.8)
original images and inpainted images (human removed), In either
0.50
57.4
51.3 (-6.1)
case, we do not provide our annotated gestural key points (eyes, fingertips, elbows, and wrists) to our model or ask our model to predict any of these gestural key points.
0.75
37.2
32.4 (-4.8)
The set of experiments using original images without providing or predicting gestural key points corresponds the "Original" column in Table 3] the "No explicit gestural key points" column in Table the "No exp. ges. k. poi." column in Table 7 and the "None" column in Table
Our model performs much worse when nonverbal gestural signals absent. Specifically, removing humans from input images results in 5.8%, 6.1%, and 4.8% performance drop under the IoU threshold of 0.25, 0.50, and 0.75, respectively; see details in Table

---

Published as a conference paper at ICLR 2023
original
inpainted
original
inpainted
original
inpainted
Notebook
(b)
Lruck
(c)
(d)
Thatia pi
Fiends
thandd
ofine
The booksi froet clim
GloL: 0.871
painting
painting
(e)
(1)
poper lovsl
The paper
Plats
CHU S1A5
GRAE
(h)
(i)
Figure S: Training with inpainted images leads a performance drop. Removing human contexts increase the difficulties in determining which object is referned to by the natural language expression (a. b, c, i) or even makes it impossible (d, e. f, g. h, j). We call it impossible to determine the referent using natural language input when multiple objects in the image satisfy the descriptions the natural language input sentence. (Green sentence: natural language input: Green box: ground-trurt referent location; Red box: predicted referent location; Red numbers near boxes: GloU between the predicted box and ground-truth box.)
These performance drops indicate that the model lacks some helpful information when only inpainted images are given. Specifically, removing humans in the scenes eliminates useful communicative signals, including pointing, poses, and the gaze direction (Sebanz et al. (006). The lack of human context introduces additional ambiguities in identifying the referents, especially when the natural language input alone cannot uniquely refer to an object in the scene.
For example, Figure 5t] given the natural language input "the books in front of me," determining which piles of books this person refers to is difficult: one on the table and one in the scene's 's lower right corner, thus need of nonverbal gestural signals, such as pointing. Similarly, Figure 5e) we cannot distinguish which picture frame the person refers to without considering the pointing gesture.
attention
prediction
attention
prediction
These are paper towels
Blue coclur tryy
GRU 0.88
0:81
door handle
This is fam
besket
the bai on the floor
OHAB
Figure 6: Attention weights visualizations. Atlention visualization of models trained to learn VTLs explicitly. (Blue: attention from object tokens; Yellow: attention from gestural key point tokens; Green sentence: natural language input; Green box: ground-trute -ruth referent location; Red box: predicted referent location; Red numbers near boxes: GloU between the predicted box and ground-trure box.)
7

---

Published as a conference paper at ICLR 2023
The glass bottle with lid
The ipad on the desk
body lotion
BILL UED
GIOUE 0.43
CROLE POAII-
GHOUE ob3 black pol
(d)
Figure 7: Examples of failure cases. Green sentence: natural language input; Green box: ground-truth referent location; Red box: predicted referent location; Red numbers near boxes: GloU between the predicted box and ground-truth box.
GloU: 0012
candle
coffee pot
M
a black chair
GloU: 00767
GloU:-0.042
MOPPO
(a)
(b)
(c)
Figure 8: Rare cases where the human heads are invisible.
4.5 ADDITIONAL. RESULTS
Attention weights visualizations We visualize the attention weights of our model trained with VTL (Figure 6). We yellow to visualize the attention weights of matched gesture keypoint queries and blue for matched object queries. Visualizations show that the attention of object queries can attend to the regions of target objects, whereas the attention of keypoint queries primarily focuses humans' heads and hands. These results indicate that our model successfully learns gestural features that boost performance.
Failure cases While the VTL effectively helps the model leverage the nonverbal gestural signals, the performance of our model still could not surpass 74% human performance. The gap between the model and human performance can be partly attributed to the model's inability to distinguish the subtle differences among similar objects. In other words, while explicitly predicting the VTLs helps our model utilize gestural signals to determine the object's direction. it does not help the model recognize the minor differences among multiple objects in the same direction.
While the VTL indeed helps narrow down the options to these two bottles, does not help our model recognize which one the body lotion. Similarly, in Figure 7b] the VTL indicates two bottles-or with red lid and the other with yellow However, does not help the model distinguish the subtle differences between lid colors.
For example, the VTL passes two bottles in Figure 7a
Additionally, in very rare cases (Figure 8), failure to detect the human head in the image prevents the application of our VTL.
Computed cosine similarities The computed cosine similarity is much higher when using the VTLS Table 4), indicating the VTLs are more co-linear to the referents. This result verifies our hypothesis that the VTLs more reliable in referring to the objects' locations.
Referent alignment loss An ablation study (Table 5) shows that our reference alignment loss plays significant role in leveraging nonverbal gestural signals.
Table Computed cosine similar- ities. (tgt: computed using ground- truth box centers; pred: computed using predicted box centers.)
Table 5: Effects of the referent alignment loss Latign Removing the reference alignment loss leads to performance drops in locating referents.
gesture
ref. align. loss ToU=:25 25
IoU=.50
IoU=,75
cos sim pred
gesture cos sim tgt cos sim pred
VTL VTL
True False
71.1
63.5
39.0
0.9878
EWL 0.9580
0.9579
67.1
59.0
36.4
0.9815
VTI.
0.9901
0.9878
(-4.0)
(-4.5)
(-2.6)
(-0.0063)
8

---

Published as a conference paper at ICLR 2023
Effects of object sizes We evaluate the model performance w.r.t the object detection of different object sizes (Table 6). We define small (S), medium (M), and large (L) objects following the size thresholds in Chen et al. 2021). Compared to the relatively consistent human performance, deep learning models' performances are significantly lower when detecting diminutive objects than when detecting more prominent objects, especially under the less stringent loU thresholds of 0.25 and 0.50. The degraded performance indicates that future artificial intelligence models need to improve the performance of small object detection.
Table 6: Model performances w.r.t. the detection of different object sizes, Numbers red and blue denote the highest and the second highest performance, respectively.
loUs
0.25
0.50
0.75
Object Sizes
All
S
All
All
FAOA YYng et al.
2019
44.5
30.6
48.6
30.4
15.8
36.2
39.3
8.5
1.4
14.4
ReSC Yang
2020
49.2
32.3
54.7
60.1
34.9
14.1
42.5
47.7
10.5
0.2
10.6
20.1
Yourefft PAF nen ala
2021
52.6
35.9
60.5
61.4
37.6
14.6
49.1
49.1
12.7
1.0
16.5
20.5
YouReflt Full Chen el al.
2021
54.7
38.5
64.1
61.6
40.5
16.3
54.4
51.1
14.0)
1.2
17.2
23.3
Ours (Inpainting)
59.0
41.3
59.3
75.8
51.3
32.1
54.6
66.7
32.4
9.4
33.3
53.4
Ours (No explicit gestural key points)
64.9
49.6
76.7
57.4
40.8
62.1
37.2
14.4
56.7
Ours (EWL)
69.5
56.6
71.7
80.0
60.7
44.4
66.2
71.2
35.5
11.8
38.9
55.0
Ours (VTL)
71.1
55.9
75.5
81.7
63.5
47.0
70.2
73.1
39.0
13.4
45.2
57.8
Human (Chen et al. 2021
94.2
93.7
92.3
96.3
85.8
81.0
86.7
89.4
53.3
33.9
55.9
68.1
Precision computed using GIoU thresholds
We provide the performance of our models when
evaluated using GloU instead of IoU thresholds; see (Table 7
Table 7: Model performances when evaluated using GloU thresholds.
IoU=.
IoU=.50
IoU=.75
Ours (Inpainting)
$7.9
50.9
31.4
Ours (No exp. ges. k. poi.)
63.7
56.5
36.2
Ours (EWL)
67.9
59.7
34.8
Ours (VTL)
70.0
62.5
38.2
Language inputs with more ambiguities We investigate the performance of our model when using language inputs with more ambiguities by changing each input sentence to the word "object." Our results Table 8) show that language inputs with more ambiguities impair the performance of our models severely.
Table 8: Model performances when using language inputs with more ambiguities.
More ambiguous
Re-training
IoU=
IoU=,50
IoU=:75
Ours (No Exp. Ges. Poi.) Ours (VTL)
True
False
16.0
True
False
31.4
25.3
13.1
Ours (No Exp. Ours (VTL)
Poi.)
True
True
49.2
42.1
23.4
True
True
49.4
41.7
23.9
Ours (No Exp. Ges. K. Poi.) Ours (VTL)
False False
False False
64.9
57.4
37.2
71.1
63.5
39.0
CONCLUSION AND LIMITATIONS
We presented effective approach, Touch-Ling Transformer, to utilize the simple but effective VTLs to improve an artificial agent's ability to locate referents referred to by humans in the wild. Our approach is inspired by recent findings in human studies on the touch-line hypothesis, which further revealed that people frequently misinterpret other people's 's referring expressions. Our proposed archi- tecture, combined with the proposed VTL, significantly reduced the gap between model performance and human performance.
Some limitations exist in our work. First, resizing operations before and after inpainting might influence model performance. Next, primarily study the eyes' location and the upper limbs' orientation regarding nonverbal signals, leaving the study of other types of nonverbal signals to future works, such gazing direction, the direction of the finger, and the orientation of the lower limbs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank Miss Chen Zhen (BIGAI) for making the nice figures and three anonymous reviews for constructive feedback. This work is supported in part by the National Key R&D Program of China (0022)D0114900,, the Beijing Nova Program, and Beijing Didi Chuxing Technology Co., Ltd.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), 2015.
Patrick Bédard, Arul Thangavel, and Jerome N Sanes. Gaze influences finger movement-related and visual-related activation across the human brain. Experimental Brain Research, (88(1): 2008.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In International Conference on Computer Vision (ICCV), 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), 2020.
Yixin Chen, Qing Li, Deqian Kong. Yik Lun Kei, Song-Chun Zhu, Tao Gao, Yixin Zhu, and Siyuan Huang. Youreñit: Embodied reference understanding with language and gesture. In International Conference on Computer Vision (ICCV), 2021.
Eunji Chong. Nataniel Ruiz, Yongxin Wang. Yun Zhang, Agata Rozga, and James Rehg. Connect- ing gaze, scene, and attention: Generalized attention estimation via joint modeling of gaze and scene saliency. In European Conference on Computer Vision (ECCV), 2018
Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. Detecting attended visual targets in video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. Visual grounding with transformers. In International Conference on Multimedia and Expo (ICME), 2022.
Yi Fang, Jiapeng Tang, Wang Shen, Wei Shen, Xiao Gu, Li Song, and Guangtao Zhai. Dual attention guided gaze target detection the wild. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Oliver Herbort and Wilfried Kunde. Spatial iitteppttertttnnooonn of pointing gestures distal referents. Journal of Experimental Psychology: Human Perception and Performance, 42(1):78, 2016.
Oliver Herbort and Wilfried Kunde. How to point and to interpret pointing gestures? instructions can reduce pointer-observer misunderstandings. Psychological Research, 82(2):395-406, 2018.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for worl-world visual reasoning and compositional question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
10

---

Published as a conference paper at ICLR 2023
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-mi molti--oddal understanding. In International Conference Computer Viston (ICCV), 2021.
Sotaro Kita. Pointing: Where language, culture, and cognition meet. Psychology Press, 2003.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vi: universal encoder for vision and language by cross-modal pre- pre-training. In AAAI Conference on Artificial Intelligence (AAAT), 2020.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yunhao Li, Wei Shen, Zhongpai Gao. Yucheng Zhu, Guangtao Zhai, and Guodong Guo. Looking here there? gaze following in 360-degree images. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X.::907.11192, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostic visiolinguistic representations for vision-and-lange language tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In Robotics: Science and Systems (RSS), 2021.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Francisco Massa and Ross Girshick. of instance segmentation and object detection algorithms in pytorch, 2018.
Fast, modular reference implementation
Colin McGinn. The mechanism of reference. Synthese, pp. 157-186, 1981.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters (RA-L), 2022.
Cathal O'Madagain, Gregor Kachel, and Brent Strickland. The origin of pointing: Evidence for the touch hypothesis. Science Advances, 5(7)xeaav2558, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICML), 2018.
Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Adria Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze in video. In International Conference Computer Vision (ICCV), 2017.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations (ICIR), 2018.

---

Published as a conference paper at ICLR 2023
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision (ECCV), 2016.
Natalie Sebanz, Harold Bekkering, and Günther Knoblich. Joint action: bodies and minds moving together. Trends in Cognitive Sciences, 10(2):00-76, 2006.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
Simon Stepputtis, Joseph Campbell, Mariano Phiclipp. Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditione imitation learning for robot manipulation tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Annual Conference on Empirical Methods Natural Language Processing (EMNLP), 2019.
Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. End-to-end human-gaze-tarre detection with transformers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
thengyuan Yang. Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. fast and accurate one-stage approach to visual grounding. In International Conference on Computer Vision (ICCV), 2019.
Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In European Conference on Computer Vision (ECCV), 2020.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), 2016
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Rowan Zellers, Yonatan Bisk, Ali Farhadi. and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Hao Zhao, Ming Lu, Anbang Yao, Yurong Chen, and Li Zhang. Learning draw sight lines. International Journal of Computer Vision (ICV), 12866):006--1100, 2020.
12

---

Published as a conference paper at ICLR 2023
A INPAINTING
We hypothesize that our model may learn to use postural signals without being explicitly asked to. To test our hypothesis, we remove postural signals (by removing humans) from the images mainly using the Mask R-CNN (Massa & Girshick] 2018) and MAT (Li et al. 2022). We investigate whether our model performs worse when trained using images without gestural signals.
We hypothesize that Transformer models may learn to use postural signals without being explicitly asked to learn these types of signals. To test hypothesis, we conduct two groups of experiments: the inpainting group and the control group. In the inpainting group, we remove postural signals the input image. In the control group, we do not modify the input image.
In the inpainting group, we modify input images. Specifically, we remove postural signals from input images by removing humans and filling in missing portions of the image (initially occupied by humans) using MAT (Li et al. 2022). Specifically, we first use Mask R-CNN X-101 (Massa & Girshick 2018) to produce human masks. After that, we expand the human masks created by the mask renn to both the left and right sides to cover the edge of humans entirely. We ensure that the expanded mask never encroaches on regions occupied by the ground truth bounding box for the referent. After that, we feed the expanded masks into MAT. With input masks, MAT removes the regions covered by the masks and fills these regions. Examples of masks, expanded masks, and inpaintings are in Figure AI
original Imagee
expanded mask
impainted imauge
Figure A1: Illustration of the inpainting process. We remove gestural signals from input images before feeding images into model study the effects of implicitly learned postural signals.
We remove gestural signals (through inpainting) from images when studying our model's 's ability to learn these signals implicitly. Before generating inpaintings. by 50 pixels. We reshape masks and images to 512 x 512 before feeding them into the MAT model because the checkpoint produced by MAT only works for inputs of size 512 x 512. Outputs of the MAT model are reshaped to their original sizes before feeding into our model. We observe that, for tiny number of images, human masks cannot be generated by F-RCNNs. In these sporadic cases, we use the original image instead.
expand the human mask to both sides
B ADDITIONAL LOSSES
B. LOSSES FOR PREDICTED REFERENT BOUNDING BOXES
We use weighted sum of L1 and GloU losses for predicted bounding boxes. Each bounding box B = (x, y, w, h) is represented using the x and y coordinate the box center (x and y), width (w), and height (h). We denote the predicted box as B, and its ground truth B,.
L1 Loss
For each pair of predicted box Bn, and ground truth box B,,, the L1 loss is:
LL1, = BB, Br, 1xp, xz,1 + 1p,, Vt.l ||wp - we,l + I/p.. heg
(AI)
Each time, there pairs of predicted and ground truth boxes. The total L1 loss for all pairs is:
1 n
LI
Leng
(A2)
GIoU loss Before computing the GloU Loss RRzzatffigii et al. 2019 is transformed to B- = (Emins Umin, T Tmarx Umaz), where I'in
each box B (x. y, w. h) w Fmaz = x + Umin
h
h
y 2. Vmaa
y 2-
A1

---

Published as a conference paper at ICLR 2023
The area of predicted box Bp and ground truth box B, are computed as: Areap,
(Ppmaxi Pmmss (Hpmaz, Tymin Feminee )
(A3)
Area;, (Dimane
(Vtmans Pimss
The IoU and Union of Bp and B, are computed as:
Steft, = max (2 Pming L1 minn
Ytop, max max(Upmm (ppmingg Mtming
rightt min( Pmane, Flmuxe
Ybottom, = min (ypmaz . 3tmmax
(A4)
Intersection; (Fright, Bleft.) x (Ubottom; Ytop:)
Union; Areap; Area, Intersection;
IoU; Intersection;
Union;
For each pair of By, and Br. the GloU is:
cleft. min( x pminm Itminn
klop. min Upmin Mtmmin
Lright. max(Tp Ttmanz
Mottom. max(UPmax; 1 : (itmazs)
(A5)
Area;- (Fright: Flefte) (Hottoom: Weops)
LoU,- Area Union;
GloU,
Area
The GloU loss for one pair of the predicted box and target box is:
LGIOU, GIoU,.
(A6)
The GloU loss for all pairs of predicted and target boxes is:
LGIOU 1
LGIOL.
(A7)
B.2 LOSSES FOR GESTURAL KEY POINTS
For the predicted eyes and fingertips elbows and wrists, LI loss. Specifically, each time, our model predicts m pairs gestural key points. Each pair denoted as pairp. (Feyepy Neyep, F Vfstp,) (for VTL) or pairp, (Felbpp Velby, Twsstp Dwst,,) ) (for EWL). The ground truth gestural key points are represented pair;; (Neyer, Doyes, Tigh, Mfgte) (for VTL) or pairt, (Ielhr, Belb., Twste, ywat,,) (for EWL).
The loss for each predicted gestural pair is defined as:
Lggettre Keyer. - Ieyee, + [Beyen. Beyer, + | - fote, + Iysors, - -Vggte, (A8)
for VTL, or Lgesture L1, for EWL.
Telbp, Feibr, (eelop Betler + [wsst Dwoete + yywetp Iuuote (A9)
The L1 gestural key point loss is:
Ligesture,111 min I L E (1....,..
(A10)
A2

---

Published as a conference paper at ICLR 2023
Additionally, apply entropy loss Lgesture_C' for predicted gestural key points with two classes: "are gestural key points" and "are not gestural key points."
The total loss for gestural key points is:
Lgesture Agesture.11 Lggesure.L1 + Agesture.CE LegestureCC..
(A11)
where Oggesture..l = 6 and L.gesture.CB = 1.5.
B.3 SOFT TOKEN LOSS
For each object, we predict token spans produced by the BPE scheme (Sennrich et al.) 2016), instead of object categories, and set the maximum number of tokens to 256 following Kamath et al. (2021). Following Kamath et al. (202T), we use soft token loss Trooken for the predicted token spans. The soft token loss is a cross-entrope loss. Specifically, an object may correspond to k token locations & & 256), and the ground-truth probability for each of the token locations x
B.4 MATCHING STRATEGY
We match prediction and ground truth using the Hungarian algorithm (Kuhn 1955 Carion al. 2020 by minimizing the cost C:
ami LL + aGGoU LATOU + atoken
Ltokens
(A12)
where 0Li 5, aGIoU 2, and atoken 1.
B.5 CONTRASTIVE ALIGNMENT LOSS
The contrastive alignment loss encourages alignment between the object feature from the Transformer decoder and the text feature from the Transformer encoder.
The number of predicted objects is n, and the number of text tokens is l. Let Fa, and Fty denote the feature of the th object and the feature of the th token, respectively. At the same time, let T,* denote the set of tokens to be aligned to by the th object, and let O denote the set of objects to be aligned to the th token. Mcanwhile, 0.07 is the temperature.
For all objects, the contrastive alignment loss is:
log EXp(FTFFF,
Leontrastive,
Σ JET Ek_1 exp(1'' Fi./T)
(A13)
For all text tokens, the contrastive alignment loss is:
exp(FI (F Fo, Fo5/T)
Leontrastive, = Σ O
Σ
llog
(A14)
I exp F? Fou/T)
JEO
The final contrastive alignment loss is the average of Leontrastion, and Leontrastive;
Lcontrastive, + Leontrastive,
Leontrastive
(A15)
2
C COMPUTING COSINE SIMILARITIES USING DIFFERENT LINES
The cosine similarity in Equation (1) can be computed using different lines. Specifically, the eye, the fingertip, and the object can form a triangle, and there are three ways choosing two lines from triangle. In Equation (1) use the two lines connected by the eye.
We investigate the effects of using different lines for cosine similarity computation. Specifically, we conducted an additional experiment using two lines connected by the fingertip. In other words, use the following two vectors: one from the fingertip to the eye, and the other from the object center to the fingertip.
A3

---

Published as a conference paper at ICLR 2023
Our results Table Al show that using the lines connected by the eye and using the two lines connected by the fingertip can regarded as fungible.
In specific, using the two lines connected by the fingertip, the model's performance is 69.0. 60.8, and 37.3 under the IoU threshold of 0.25, 0.50, and 0.75, respectively. Compared to the EWL model, it obtains +1.5 performance boost under the IoU threshold of 0.75. Compared to the No Explicit Gestural Key Points Model, obtains a +4.1 and +3.4 performance boost under the IoU threshold of 0.25 and 0.50, respectively.
Table A1: Using different lines for cosine similarity computation in the VTL model.
IoU=
IoU=,50
IoU=,75
Ours explicit gestural key points)
64.9
57.4
37.2
Ours (EWL) Ours (VTL, cos sis Vertex = Fingertip)
69.5
60.7
35.5
69.0
60.8
37.3
A4


Reviews:
[{"review_id": "r_4Rr82Ijt_", "reviewer": "Reviewer_iwkJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to automatically locate referents in scenes using a combination of embodied gesture signals and natural language descriptions. Their key contribution is the implementation of the \"virtual touch-line\", which is the extended line connecting the eye and the fingertip to localize objects in scenes. The authors propose a transformer-based network and a geometric consistency loss to combine the language and the gesture signals and predict the object bounding boxes. They demonstrate the benefits of their approach through experimental evaluations and ablation studies.", "summary_of_the_review": "Overall, the paper solves a challenging problem using an intuitive and sound approach. The technical descriptions are clear and the experiments highlight the quality of the work. This method sets a new benchmark in embodied reference understanding and can be reused in associated applications.", "strength_and_weaknesses": "**Strengths**\n\n1. The proposed design of the virtual touch-line is sound, intuitive, and well backed up by prior work.\n\n2. The geometric consistency loss is also sound.\n\n3. The ablation studies show the benefits of the individual components of the proposed end-to-end network.\n\n4. The discussion of the limitations helps to understand the scope of the work.\n\n\n**Weaknesses**\n\n1. Looking at Eqn. 1, the authors take the reference line to be the one connecting the eye and the object. Have the authors considered the alternate, the line connecting the fingertip and the object? Is there any specific reason to choose one over the other, e.g., the noises in eye and fingertip detection?\n\n2. I did not find an ablation study that uses only the gestural signals but no language signals. This could be an interesting ablation to observe, especially in the real-world context where the language input is noisy or corrupted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QA1nW91mcYm", "reviewer": "Reviewer_BH2V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a transformer based technique for embodied reference understanding. In this problem, the task is to determine the object being referred to (by the human in the image and the text) in an image. The model takes in visual and text features (from a CNN and BERT respectively) and outputs 1) the bounding box of the referent (the object being referred to) 2) touch-line, the vector from the human (referrer) to the referent (referred object). The motivation for including the touch line is that this should help the model locate the referred object. The model is trained to predict both of these outputs. For the touch-line, the authors annotate the datasets themselves to provide this information. They try two different ways to define the touch-line. One is the vector from elbow to wrist (EWL), and the other is the one from the eye to fingertip (virtual touch line, VTL). Citing studies from psychology, the authors argue that VTL is a better choice and in fact show that predicting VTL leads to better referent detection than using EWL. The model uses an additional geometric loss to encourage the predicted touch line and the predicted referred object to lie on the same line. Overall, their model is able to reach SoTA performance on the YouRefIt embodied reference understanding dataset.", "summary_of_the_review": "Given the significant increase in performance on the embodied reference understanding task and the novel idea of coupling virtual touch line prediction with referent detection, I think this would be a good addition to the conference. However, I'd have liked to see a more detailed analysis of the performance of the model.", "strength_and_weaknesses": "Overall, I think this is an interesting paper that shows a few modifications to a multi-modal transformer based setup leads to a significant increase in performance in embodied reference understanding. Note that I'm not familiar with this literature very well (embodied reference understanding seems to be a problem recently introduced), so my assessment of novelty/originality might not be 100% accurate. As far as I understand, the main contributions here are the multi-modal transformer and the predicting virtual touch line alongside the bounding box of referent. I like the virtual touch line idea (and the argument/results for why you would prefer it to EWL) and this seems to increase performance. Overall, the results are pretty strong but it is not fully clear where this increase in performance comes from (more on this below). I'd say the writing is overall pretty clear but some model and evaluation details are missing, which makes it difficult to judge the full merit of the approach (again more on this below).\n\n- What exactly leads to better performance from this model? Or to put it another way, what makes this model work much better than the Chen et al.'s model? Looking at the results, even without any pose information or humans in the image, the method in this paper does much better than Chen et al's? Why is this the case? I think it'd be great to have an ablation analysis that shows where this increase in performance comes from.\n- The requirement of pose annotation (and VTL) limits the applicability of the approach slightly, but the model seems to do better than previous ones even without any pose information. So it'd be nice to know why.\n- An alternative could be to use a pre-trained pose estimation network and use the features from this model as additional inputs. Have the authors tried this?\n- What parts, if any, are pre-trained in this model? It looks like the model builds on a previous object detection model. Is this pre-trained? This has to be mentioned in the paper.\n- None of the results tables have standard deviations. Even though the performance gap between models is pretty large, it'd be nice to have these std devs.\n- In Table 2, when you train with EWL, do you still have the geometric consistency loss? If so, wouldn't this hurt performance since EWL usually doesn't lie on a line with the referent.\n- What happens if you don't have the geometric consistency loss (but still predict VTL)? How much does this hurt performance?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NvtGzUS9R8t", "reviewer": "Reviewer_qAaB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a solution to “embodied reference understanding”, which aims to locate so-called “referents” which are objects referred to by text and gestures. The core observation of this work is that referents do not lie on the elbow-wrist line but rather on the eye-wrist line, which they call the “virtual touch line (VTL)”. By learning to predict the VTL using a multi-modal hybrid transformer architecture and by encouraging the co-linearity of object center, eye, and fingertip, the proposed method is shown to perform well on the YouRefIt dataset.", "summary_of_the_review": "The paper was a good read, and the proposed solution is well packaged and delivered. However, the text could benefit from a mild revision, and the results on a single dataset might not necessarily warrant acceptance. Yet, I lean towards the positive side as the overall concept is simple, intuitive, and executed well.\n\n**Post rebuttal comment.**\nI thank the authors and the reviewers for their diligent comments. After having reviewed all available written materials, I feel that while the authors have made a good effort to respond to the various concerns, and their results do show meaningful improvements to the baseline of YouRefIt (Chen et al., 2021), it is difficult to enthusiastically recommend the paper's acceptance for the following reasons:\n- the task is incredibly niche and this is self-evident by the lack of datasets to run evaluations on (and therefore show within-research-domain generality)\n- no indications are made on how to extend the proposed concepts (hand-designed reference vector based on pre-trained keypoint network) to other areas of ML and CV (to show out-of-research-domain generality).\n\nGiven that the scope of NeurIPS is focused in the ML domain with allowance for competitive applications to CV, it is important for papers to provide knowledge that goes beyond a very niche CV topic. Yet, the paper is solid enough to be published at a top-tier CV+ML venue, so I retain my rating of borderline/weak accept.", "strength_and_weaknesses": "The introduction motivates the VTL well, and general approach is easy to understand, though the architecture section is somewhat brief.\n\nWhile it is written well, the “Related Work” section is much too verbose and contains parts that are typically considered out-of-scope or tangential, in particular: “Gaze estimation”, “Saliency estimation”, and “HRI and collaboration”. Less relevant work could be removed to make the section more compact and to-the-point.\n\nOverall, the results are quite impressive. It is clear that the VTL objective out-performs the use of EWL. The visual information provided by the imaged human is also useful (vs inpainting), and the proposed contribution overall out-performs recent works. It is also nice to see that the VTL-EWL discussion is further validated by seeing the cosine similarities defined in Eq. (1).\n\nHowever, one must wonder whether there are additional ways to demonstrate the value of the contributed method. That is, are there other object-referring datasets that could be used where body keypoints are not annotated, but are visible and thus permit a VTL? I am not a domain expert so cannot make any recommendations, but the method and evaluation here seems far too specific and lack possibilities to generalize.\n\nMiscellaneous:\n- It is never really explained what the “(No Pose)” experiments are.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster