Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce CAESAR, an embodied simulator for generating multimodal referring expression datasets. The generated referring expressions can include both verbal utterances and nonverbal gestures (pointing + gaze), and require resolving spatial relations between objects in the scene. The simulator can also be used to generate contrastive examples, where the referred object from the verbal utterance does not match the object referenced by the nonverbal gesture. The authors use their simulator to generate two datasets, and benchmark various existing methods on these generated datasets.

I am mostly happy with this work, but I would like weakness #1 to be addressed since this seems like a glaring blindspot.
strengths: 1. Table 1 in the paper is extremely well designed, and the information is succinctly conveyed. Everything I need to know about this work in comparison to existing referring expression datasets is neatly summarized in this one table. The related work is also very well covered.

2. The methodology for designing the simulator is described in great detail, and very easy to follow. All the design decisions are motivated well. It is clear the authors put a great deal of thought into possible issues with the dataset.

3. The dataset analysis is very well done, and gives useful insights.

4. The authors release not only their generated datasets but also their simulator.

5. The paper is well-written, and very easy to follow.

weaknesses: 1. This seems to be a glaring omission: for the two generated datasets, CAESAR-L and CAESAR-XL, no train/val/test splits are reported. Does that mean the models were trained and evaluated on the full data?? If so, I can't take any of the results in Section 6 at face value.

2. While it is great that the authors are releasing not only their generated datasets but also the simulator, it's not clear how the authors think this simulator can be useful to other researchers. What can the simulators be used to design that is not already covered by the generated CAESAR datasets? This is not a hard requirement, or even a real weakness, but would be useful to motivate releasing the simulator, or as Future Work discussion.

3. The paper could use more vision-language pre-trained model baselines, such as UNITER or ViLT, or any other VL transformer model that jointly takes vision and language inputs and processes them in the Transformer together.
correctness: It appears to be constructed in a sound way - lack of train/val/test splits notwithstanding.
clarity: Yes
relation to prior work: Yes
additional feedback: -
review_title: Official Review of CAESAR

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces a new embodied multi-model referring simulator (CAESAR) for understanding human verbal utterance, pointing, and gaze. 
Two datasets of varying sizes have been collected from CAESAR containing resourceful information for referential tasks. 
Besides, several baselines about configuring the grounding tasks are performed to illustrate the different impacts of different modalities. 
strengths: I do appreciate the dataset synthesizing method: CAESAR bridges the simulation in a virtual engine with motion captures in reality, making it possible to run experiments on testing the embodied AI's performance in real indoor settings. 

The data collecting process is clear but non-trivial. Compared with previous related work on collecting/synthesizing pointing and gazes (e.g. ReferIt3D, CLEVR-Ref), the CAESAR-L/XL dataset contains thorough descriptive information for the agent, as well as for the scene/object understanding.

As a dataset plus simulator, the dataset is well-organized. The website for holding the benchmark is well-designed. 
weaknesses: The thing I worry about most about the paper is the generalizability of the dataset. (a) Even though the dataset is of huge size, it is based on few scene types. I am wondering if is easy for the dataset to be synthesized in photo-realistic indoor scenes. (b) For the 80 objects in the dataset, in my opinion, they mostly are small items in the kitchen or living room. How can we believe they are representative of embodied AI tasks. (c) The dataset brings human gestures and gazes, but I don't see this work describing the variability when humans are pointing and looking at an object. 

For the experiments, this work runs several end-to-end experiments on grounding the referential task given inputs of multi-modalities. However, I want to know (a) What is the insight when we bring gaze, pointing, language, and views together? (b)How do pointing and gaze cooperate to bring or remove ambiguity from language descriptions? 
correctness: The paper presents the main contribution as datasets. The process of collecting the data is demonstrated in detail. Statistics from exploratory data analysis are well presented in the paper and appendix. The experiments and evaluation methods are straightforward and clearly defined.
clarity: The paper is well written, except that the figures and tables are surrounded by texts (looks crowded). 
relation to prior work: The embodied AI setting (egocentric view) makes this work different from most previous visual understanding works (e.g. RefCoCo, Clever, YouRefIt). As far as my knowledge goes, the motion tracking + synthesizing data collecting strategy is original and unique in the related fields. 
additional feedback: The dataset size is extremely large. Do you plan to release some sample data of a small size to help users to get started?
review_title: Elementary but crucial work toward human science pedagogy and recognition

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 1. The CAESAR developed by the authors break the limitations of the previous dataset for capturing embodied multimodal referring expressions by providing the awareness of multiple perspectives.
2. The data generation system is easy to use and can provide large convenience for other researchers.
3. The use of this data generation system for multi-perspective awareness is an enlightening breakthrough for current research in the field.
4. Benchmarks with SOTA methods are done with the provided datasets for grounding embodied spatial relations.
strengths: 1. The data generation system developed in this work can help increase the model's robustness with the awareness of multiple perspectives.
2. The proposed data generation system is easy to use and user-friendly, and open to researchers.
3. A brief summary of the limitations and future works are given, which provides new inspiration for subsequent work in this area.
weaknesses:  In the experiment result shown in Table 3, adding the result of combining two different perspectives and analyzing the results will be more convincing for this part.
correctness: The claims made in the submission correct and the datasets are constructed in a sound way.
clarity: The paper is well written with detailed scientific explanations.
relation to prior work: This work summarizes the problems and shortcomings of previous work. It proposes data generation models based on these problems to provide data and ideas for subsequent training of more robust models.
additional feedback: Non.
review_title: The authors developed a novel simulator for generating multimodal referring expressions and provided two large-scale datasets with the benchmarks. This work effectively solves the problem of the previous dataset always ignoring the perspective-awareness. This work provides excellent convenience and hints of research directions for subsequent investigators.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents CAESAR, a simulator capable of automatically generating multimodal referring expressions with verbal utterances and nonverbal cue (pointing gesture and gaze) to refer to an object. The major contributions of the paper includes the CAESAR simulator, as well as two large-scale datasets CAESAR-L and CAESAR-XL, with over a million samples in total. The most important advantage of these datasets compared to existing datasets is that these datasets provide ego (first-person), exo (third person) and top views of the scene where all previous datasets only provide exo views. The authors benchmarked various models on the datasets. 
strengths: 1. The paper presents two large scale datasets that have unique novelties compared to all existing datasets, such as these datasets provide ego (first-person), exo (third person) and top views of the scene where all previous datasets only provide exo views, and these two new datasets provide contrastive samples and ambiguous samples. These novel features can benefit research on interactive AI,
2. The authors not only provide the datasets, but also release a configurable simulator for generating the datasets, so if researchers are unsatisfied by the configurations used to generate the two provided datasets, they can use the simulator to generate their own data with their desired configurations.
3. The authors benchmarked their datasets on a few SOTA multimodal models such as CLIP, ViT+BERT, etc, and studied the impacts of modalities as well as the multiple perspectives, thus pointing out advantages of the datasets as well as providing potential future research directions for users of the datasets.
4. The authors carefully documented details of their dataset collection process and how to use their simulator tools in the supplemental materials. 

weaknesses: N/A
correctness: The dataset is constructed in a sound way.
clarity: The paper is clear and well written.
relation to prior work: The paper very clearly discussed how it differs from previous contributions (which is very clearly summarized in Table 1)
additional feedback: N/A
review_title: Large datasets and simulator that is novel and useful

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors of the paper propose a new embodied simulator developed in the Unity engine. The task of the simulator is to generate training data for multimodal referring expression task, i.e. containing verbal (language) and nonverbal (gesture, gaze) cues with respect to the target object. The authors point out the ambiguity of referring to objects in the real world - perspective, and mistakes in gestures, and address those while creating two proposed datasets with the use of their simulator. The authors provide various models and ablations explaining the shortcomings of current approaches to multimodal referring expressions.
strengths: I believe that the authors accurately identified the state of current datasets and benchmarks for referring expressions. Specifically, I think it is a good assessment to include a perspective as an important factor in the description of the given scene (which is the main differentiating factor with respect to YouRefIt [1]). I find that introducing a customisable tool that can generate any arbitrary number of samples for the task is valuable for the community. I also appreciate the inclusion of contrastive and ambiguous samples. 

[1] Yixin Chen, Qing Li, Deqian Kong, Yik Lun Kei, Song-Chun Zhu, Tao Gao, Yixin Zhu, and Siyuan Huang. Yourefit: Embodied reference understanding with language and gesture. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021
weaknesses: I find several aspects of the work that could be polished further, or provided with some more insight.

1. From my experience in Unity, I would recommend releasing the simulator as a standalone application. With new versions of Unity, packages sometimes become outdated w.r.t. the version of Unity and compatibility may be lost. I would suggest to the authors to consider that as a possibility to increase the sustainability of the simulator. Even though Unity offers to update packages when opened in a new version, it does not always resolve all updates correctly. 
2. I would expect a bit more thorough description of the dataset samples: what is the number of various scenes, how many verbal cues are generated for each scene, what is the number of images generated for each scene/expression, what is the range on the number of generated frames when including video.
3. Further, the creation of contrastive samples could be slightly adjusted. Specifically, with respect to situations when the verbal cue is pointing to a non-existing object (line 227). I am not certain if that is very helpful in training the model. My doubt is what would be the expected output of the model. If making an inaccurate verbal description, I would consider miscueing only one property of the object (e.g. given only one mug, yellow, on the table, referring to it as 'the red mug') such that there is still information to be learned in the expression. 
4. For the benchmark, if I understand correctly, a single verbal expression (from exo or ego perspective) is paired with all the views. Without providing any cue to which perspective verbal utterance corresponds, I think there may exist a lot of ambiguity in the samples, affecting the results.
5. On a smaller note, some comparison of using contrastive samples with the previous works (like [2]) could be useful. Also, some insights behind the choice of a specific engine from the visual quality perspective would be nice.

Finally, I find some things worth to be clarified (not necessarily weaknesses yet) - see Questions.

[2] Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances in Neural Information Processing Systems, 2020
correctness: The dataset is constructed in a sound way. The authors took care to balance the dataset with respect to the scenarios (templates) they propose, and also with respect to contrastive examples. 

clarity: Overall, the paper is written fairly clearly, however, there are some sections that would require some more insight or disambiguation (see Questions).
relation to prior work: Relation to prior work is well covered, especially with respect to referring expressions datasets. Some more clarification with respect to various dataset generation tools (starting line 123) would be useful. Specifically, what prevents the transferability to other environments (line 128).
additional feedback: Questions: 
- In line 54 you refer to the typical human mistake of pointing to another object. Do you consider any constraints on when mistakes are made in contrastive examples? Intuitively, one would make a mistake probably when not using the name of the object (pointing to 'the mug' vs pointing to 'that red thing'), and the proximity of other objects sharing some visual property with the target could also induce mistakes (pointing to the incorrect object in the group of e.g. yellow objects).
- Another big source of confusion occurs when ego and exo views are opposite to each other and the actor tries to adjust the instruction to the observer's perspective ('my left or your left' scenario). Do you think it would be useful to consider such examples, or based on your experiments it would introduce too much noise into the system?
- What is the source of the assets in the scenes? Were objects modelled by the authors, or obtained from Unity Asset Store? If later, or other, is the use of the objects licensed? If the models were not created, do they overlap with any existing dataset?
- Could you explain synthesising the pointing gestures more? How many various 'pointing gestures' were collected by OptiTrack? When synthesising the gestures, do you reconstruct the joints as close to the collected ones as possible, or is there some noise added to slightly change the gesture? Finally, can the target of the referring expression exist only in one of the places corresponding to the set collected with OptiTrack?
- Are the orientations of objects saved in annotations? It would be useful in order to allow for full scene reconstruction from annotation only.
- For templates in Table 2, is it always ensured that the target is unique (especially for templates 1, and 2)?
- For Table 3b, do you think that performance of *Ego* may have been affected by the view not being able to utilise gaze cue, especially in the presence of various contrastive examples?
review_title: Review XoLN

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work proposes 1) a new task for embodied referring expression, 2) a tool for easy-programming with a simulator for generating referring expression data, as well as 3) two generated datasets.

1) A new task for embodied referring expression.
- Two referring expressions are given in a verbal and a nonverbal (gazing/pointing/gazing+pointing) manner. The task is to decide whether or not the two referring expressions are indicating the same object.
- The nonverbal cues are provided to a model via rendered images of the scene from ego, exo, or top viewpoints. (**Please correct me if I'm wrong with regard to this point**)
- In order to perform this task, a model has to reason about spatial relationships described in the verbal cue, learn the links behind ego vs. exo vs. top viewpoints, and practice perspective-taking.

2) A tool for easy-programming with a simulator for data generation (CAESAR)
- CAESAR includes 80 objects and supports variations in color, size, position and rotation-angle.
- CAESAR is able to generate verbal referring expressions from 5 templates, which supports easy-to-difficult levels of compositional reasoning over object attributes and spatial relationships.
- CAESAR is able to synthesize the gesture and gaze of a humanoid avatar based on real human motions captured by OptiTrack

3) Two generated datasets
- A large (L) and an extra-large (XL) version
- The datasets can be used to test for understanding of nonverbal referring expressions as well as the perspective-taking ability of an embodied agent.
 
strengths: This paper correctly recognizes that, 
- the perspective-taking ability, and
- the ability to infer nonverbal cues from multimodal input,

are the crucial missing components for adapting vision models, which were trained on non-embodied, static datasets, to an embodied environment.

This paper incorporates the above intuitions into their task design:
- **Nonverbal cues from multimodal input**: A model is required to compare the referred objects by nonverbal cues vs. the objects referred by verbal expressions.
- **Perspective-taking**: A model is required to perform the referring expression task while being viewpoint-aware (ego/exo/top), because spatial cues (e.g. to the left/right of) are inherently viewpoint-dependent.

This paper develops a simulator for data generation, as well as two generated datasets to facilitate research for more capable embodied agents.
weaknesses: 1. CAESAR-L vs. CAESAR-XL, more explanation behind keeping two separate datasets would be great.
- What are the differences between -L and -XL except for the size-related factors (i.e. #samples, image resolution and image-to-sample ratio)?
- Do you expect these two datasets to support different evaluation scenarios or target different skillsets?
- Do you expect these two datasets to exhibit separate difficulty levels?
- Do you provide train/val/test splits? If yes, how do you make the split?
- If there is no significant difference, why not merging them into a single dataset?

2. Line272 "*force models to utilize nonverbal cues ... by recognizing which perspective did the given utterances come from*". I didn't get how the model is supposed to recognize that. When there are misalignments, one might consider two possibilities: 1) the current example is a "contrastive" example, where misalignment is anticipated, 2) it's necessary to fit oneself into another viewpoint. How do you expect a model to distinguish between these two possibilities? Please correct me if I'm misunderstanding. I'd appreciate the clarification.

3. (related to Q2) Line406 "*generates verbal expressions from multiple perspectives, actor and observer*". How could the model decide whether an input verbal expression comes from the actor or the observer? Do you expect the model to infer such information from the input images? And how?

4. Line302-307: I didn't get the essential difference between the Dual-encoder and the Late-fusion model besides using a different visual encoder (ViT vs. ResNet50). For example, does the Dual-encoder model perform any fusion earlier than the Late-fusion model?

5. Another clarification question: From where do you expect a model to extract gazing/pointing cues? Based on my current understanding, a model has to attend to the avatar's eyes and hands from the image input, correct? However, how could a gaze cue be extracted from a top or ego view? More explanation is greatly appreciated.

6. Line384 "*participants correctly validated the relations in 80.66% of the times*". It would be nice to provide some insight into when and where human participants failed at this task (accounting for ~20% of the examples). Could human failures be attributed to inherent ambiguity, unclear annotation instruction, the challenging nature of compositional reasoning, or any other reasons?

7. Table3: The best model performance is exceeding 80%, which is outperforming human participants. Where do you expect the future room for improvement to be?
correctness: The data construction, evaluation and experiment design are correct.
clarity: Yes
relation to prior work: Yes
additional feedback: ---
review_title: A dataset contribution where the task is to verify agreement between verbal referring expressions and nonverbal cues.

Meta-review: metareview: All the reviewers appreciated the effort and rigour that went into the design of the proposed idea. The presentation of the paper clearly shows the embedding of the work within the state of the art. It is particularly appreciated that this work is likely to lead to a break through in the analysis of nonverbal referring expressions given the possibility to generate multiple perspectives synthetically. The experiments compared with multiple state of the art models and modalities was also appreciated. 

The answers by the authors to reviewer comments were thorough and provided additional insights.

Not all reviewers were in agreement on some aspects of the work and the more negative comments questioned whether releasing the simulator will really have the impact that it could have on first appearances. (XoLN)

enpk's final response seems more positive than the rating so I think that they may have forgotten to update their score. 
The concerns of XoLN regarding the practical details of releasing a simulator are to be considered. However, it is not clear to me from the paper whether releasing the simulator is really considered a major contribution of the work rather than a minor contribution. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept