Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a transformer based technique for embodied reference understanding. In this problem, the task is to determine the object being referred to (by the human in the image and the text) in an image. The model takes in visual and text features (from a CNN and BERT respectively) and outputs 1) the bounding box of the referent (the object being referred to) 2) touch-line, the vector from the human (referrer) to the referent (referred object). The motivation for including the touch line is that this should help the model locate the referred object. The model is trained to predict both of these outputs. For the touch-line, the authors annotate the datasets themselves to provide this information. They try two different ways to define the touch-line. One is the vector from elbow to wrist (EWL), and the other is the one from the eye to fingertip (virtual touch line, VTL). Citing studies from psychology, the authors argue that VTL is a better choice and in fact show that predicting VTL leads to better referent detection than using EWL. The model uses an additional geometric loss to encourage the predicted touch line and the predicted referred object to lie on the same line. Overall, their model is able to reach SoTA performance on the YouRefIt embodied reference understanding dataset.
strength_and_weaknesses: Overall, I think this is an interesting paper that shows a few modifications to a multi-modal transformer based setup leads to a significant increase in performance in embodied reference understanding. Note that I'm not familiar with this literature very well (embodied reference understanding seems to be a problem recently introduced), so my assessment of novelty/originality might not be 100% accurate. As far as I understand, the main contributions here are the multi-modal transformer and the predicting virtual touch line alongside the bounding box of referent. I like the virtual touch line idea (and the argument/results for why you would prefer it to EWL) and this seems to increase performance. Overall, the results are pretty strong but it is not fully clear where this increase in performance comes from (more on this below). I'd say the writing is overall pretty clear but some model and evaluation details are missing, which makes it difficult to judge the full merit of the approach (again more on this below).

- What exactly leads to better performance from this model? Or to put it another way, what makes this model work much better than the Chen et al.'s model? Looking at the results, even without any pose information or humans in the image, the method in this paper does much better than Chen et al's? Why is this the case? I think it'd be great to have an ablation analysis that shows where this increase in performance comes from.
- The requirement of pose annotation (and VTL) limits the applicability of the approach slightly, but the model seems to do better than previous ones even without any pose information. So it'd be nice to know why.
- An alternative could be to use a pre-trained pose estimation network and use the features from this model as additional inputs. Have the authors tried this?
- What parts, if any, are pre-trained in this model? It looks like the model builds on a previous object detection model. Is this pre-trained? This has to be mentioned in the paper.
- None of the results tables have standard deviations. Even though the performance gap between models is pretty large, it'd be nice to have these std devs.
- In Table 2, when you train with EWL, do you still have the geometric consistency loss? If so, wouldn't this hurt performance since EWL usually doesn't lie on a line with the referent.
- What happens if you don't have the geometric consistency loss (but still predict VTL)? How much does this hurt performance?

clarity,_quality,_novelty_and_reproducibility: Overall, this work has a few novel ideas (e.g., virtual touch line, using pose) and makes a significant improvement in performance on embodied reference understanding. The writing is overall clear and the results are strong. However, it is not fully clear where the performance increase comes from so the paper would benefit from a more in depth discussion and analysis of the model and results.
summary_of_the_review: Given the significant increase in performance on the embodied reference understanding task and the novel idea of coupling virtual touch line prediction with referent detection, I think this would be a good addition to the conference. However, I'd have liked to see a more detailed analysis of the performance of the model.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a method to automatically locate referents in scenes using a combination of embodied gesture signals and natural language descriptions. Their key contribution is the implementation of the "virtual touch-line", which is the extended line connecting the eye and the fingertip to localize objects in scenes. The authors propose a transformer-based network and a geometric consistency loss to combine the language and the gesture signals and predict the object bounding boxes. They demonstrate the benefits of their approach through experimental evaluations and ablation studies.
strength_and_weaknesses: **Strengths**

1. The proposed design of the virtual touch-line is sound, intuitive, and well backed up by prior work.

2. The geometric consistency loss is also sound.

3. The ablation studies show the benefits of the individual components of the proposed end-to-end network.

4. The discussion of the limitations helps to understand the scope of the work.


**Weaknesses**

1. Looking at Eqn. 1, the authors take the reference line to be the one connecting the eye and the object. Have the authors considered the alternate, the line connecting the fingertip and the object? Is there any specific reason to choose one over the other, e.g., the noises in eye and fingertip detection?

2. I did not find an ablation study that uses only the gestural signals but no language signals. This could be an interesting ablation to observe, especially in the real-world context where the language input is noisy or corrupted.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

What does the "no pose" ablation mean in Table 1 and how is it different from "inpainting"?


**Quality and Novelty**

The proposed method and implementation using the virtual touch-line are novel. The experimental results show the quality of the work.


**Reproducibility**

The authors have clearly explained all the components of their method, making it reproducible. They also provide their source code.
summary_of_the_review: Overall, the paper solves a challenging problem using an intuitive and sound approach. The technical descriptions are clear and the experiments highlight the quality of the work. This method sets a new benchmark in embodied reference understanding and can be reused in associated applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a solution to “embodied reference understanding”, which aims to locate so-called “referents” which are objects referred to by text and gestures. The core observation of this work is that referents do not lie on the elbow-wrist line but rather on the eye-wrist line, which they call the “virtual touch line (VTL)”. By learning to predict the VTL using a multi-modal hybrid transformer architecture and by encouraging the co-linearity of object center, eye, and fingertip, the proposed method is shown to perform well on the YouRefIt dataset.
strength_and_weaknesses: The introduction motivates the VTL well, and general approach is easy to understand, though the architecture section is somewhat brief.

While it is written well, the “Related Work” section is much too verbose and contains parts that are typically considered out-of-scope or tangential, in particular: “Gaze estimation”, “Saliency estimation”, and “HRI and collaboration”. Less relevant work could be removed to make the section more compact and to-the-point.

Overall, the results are quite impressive. It is clear that the VTL objective out-performs the use of EWL. The visual information provided by the imaged human is also useful (vs inpainting), and the proposed contribution overall out-performs recent works. It is also nice to see that the VTL-EWL discussion is further validated by seeing the cosine similarities defined in Eq. (1).

However, one must wonder whether there are additional ways to demonstrate the value of the contributed method. That is, are there other object-referring datasets that could be used where body keypoints are not annotated, but are visible and thus permit a VTL? I am not a domain expert so cannot make any recommendations, but the method and evaluation here seems far too specific and lack possibilities to generalize.

Miscellaneous:
- It is never really explained what the “(No Pose)” experiments are.
clarity,_quality,_novelty_and_reproducibility: The work is introduced, explained, and validated well. It is of relatively high quality and clarity.

I believe the concept of VTL and its incorporation into a multi-modal transformer architecture is rather novel and original.
summary_of_the_review: The paper was a good read, and the proposed solution is well packaged and delivered. However, the text could benefit from a mild revision, and the results on a single dataset might not necessarily warrant acceptance. Yet, I lean towards the positive side as the overall concept is simple, intuitive, and executed well.

**Post rebuttal comment.**
I thank the authors and the reviewers for their diligent comments. After having reviewed all available written materials, I feel that while the authors have made a good effort to respond to the various concerns, and their results do show meaningful improvements to the baseline of YouRefIt (Chen et al., 2021), it is difficult to enthusiastically recommend the paper's acceptance for the following reasons:
- the task is incredibly niche and this is self-evident by the lack of datasets to run evaluations on (and therefore show within-research-domain generality)
- no indications are made on how to extend the proposed concepts (hand-designed reference vector based on pre-trained keypoint network) to other areas of ML and CV (to show out-of-research-domain generality).

Given that the scope of NeurIPS is focused in the ML domain with allowance for competitive applications to CV, it is important for papers to provide knowledge that goes beyond a very niche CV topic. Yet, the paper is solid enough to be published at a top-tier CV+ML venue, so I retain my rating of borderline/weak accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of embodied reference understanding (locating referents in an image using embodied gestural cues and language references). The main idea is the design of the touch-line transformer, which takes tokenized visual and textual features to simultaneously predict the referent’s location and the touch-line vector. Additionally, a geometric consistency loss that promotes the co-linearity between referents and touch lines is applied. The experiments are conducted on the YouRefIt dataset.

The main strengths of this paper are the interest of the main idea (which takes inspiration from psychology studies that show the relevance of the virtual touch-line -considered in this work and ignored by previous works); the obtained results (which significantly outperform previous methods on YouRefIt dataset); and the ablation studies, that show the contribution of each component and help understanding the results. Some concerns of the reviewers (e.g. need for additional details in the evaluation) were addressed by the authors in their feedback. 

Overall, this paper proposes an interesting new technical solution to the problem of embodied reference understanding, which is an important problem in HRI and HR collaboration.
justification_for_why_not_higher_score: The main reason to accept the paper as a poster instead of as a spotlight is the fact that the paper does not propose a technique that can be applied to different problems or domains. As pointed out implicitly or explicitly by the reviewers, this paper addresses a very specific Computer Vision problem and proposes a very interesting and tailored solution. Thus, the ICLR audience interested in this work might be not as broad as for more general ML approaches.
justification_for_why_not_lower_score: As discussed in the metareviews summary, all the reviewers consider that the paper has significant contributions, and they agree on accepting the paper.
Decision: Accept: poster