SlowGuess/ACL_2020_OCR huggingface dataset:
We conduct four analyses of the Refer360° dataset. First, we investigate if the random selection process of target locations can mitigate possible bias issues. Recent studies (Devlin et al., 2015; Agrawal et al., 2016; Jabri et al., 2016; Goyal et al., 2016) show that design decisions for collecting annotations may introduce bias into datasets. High-capacity machine learning models can exploit these issues which hinders the meaningful progress towards real language understanding (Zhou et al., 2015; Cirik et al., 2018b). Second, we study whether each instruction in an instruction sequence is critical in finding the target location, or whether some instruction sequences are overcomplete. It may be very well the case that, by just understanding the last instruction, one can easily locate the target location. Third, we perform a qualitative analysis of Refer360° to provide the types of linguistic reasoning required to find the target location accurately. Finally, we analyze the performance of the state-of-the-art on Refer360°.
5.1 Target Locations
The selection method for the target location plays a crucial role in the kind of language one needs to use to describe that location. Earlier studies on referring expression recognition datasets (Kazemzadeh et al., 2014; Hu et al., 2016; Mao et al., 2016; Strub et al., 2017) select the target location as object boxes annotated by humans. In Touchdown-SDR (Chen et al., 2018) instead, annotators decide the location of the target rather than choosing one of the pre-defined lists of object boxes<sup>8</sup>.
This could introduce a location bias to the dataset - i.e. if annotators get to select the target location, they may choose targets that are easy to describe, sometimes leading to trivial or uninteresting examples, and more broadly to artificially simple language overall. For instance, if there is only one pink object in the scene, annotators usually preferred describing that region rather than some other obscure location in the scene. Instead of letting annotators decide where to place targets in the scene, we randomly picked a target location in the scene and asked them to describe how to find that loca
tion. As a result, our instruction sequences are complex as we show next.
Comparison	Touchdown-SDR	Refer360°
The perplexity of the distribution of an object that the target is located on	9.53	17.86
The perplexity of the distribution of the closest objects	17.80	46.84
The average distance to the closest objects	8.64	23.88
To measure the differences in instructions for randomly or manually choosen targets, we compute three quantities. First, we compute the variety of objects that the target is located on using the perplexity of object frequencies. Similarly, we also compute the variety of objects closest to the target objects. Since we use objects near to the target location as anchor points, this is also another useful metric. The higher the perplexity of both metrics, the harder it is to predict the target location using just the object type or the closest object. Third, we measure the average distance between the target location and the nearest object. The closer the target location to another object, the easier it is to describe using the closest object as an anchor point.
Table 6: Statistics for target locations image in Touchdown-SDR and Refer  $360^{\circ}$ . Target is located on or near the wider variety of objects and further away from other objects.
Instructions	Average Distance	Accuracy
Last Sentence	73.01	0.37
Last 2 Sentences	42.32	0.63
All Sentences	11.35	0.88
Table 7: Results for instruction ablation human study. Annotators need all instructions to complete the task accurately.
Table 6 shows statistics for target locations in Touchdown-SDR and Refer360°. For both perplexity metrics, we observe that the target is located near or inside a wider variety of objects in Refer360°. Also, on average, the target location is further away from other objects for Refer360°. These statistics show that randomly choosing the target location helps us address possibly bias towards simple instructions and makes recognition more challenging.
5.2 Ablation of Instruction Sentences
While collecting instructions, we asked annotators to describe the target location using at least three and at most five sentences. It might be possible to
In our initial iterations for the data collection, we followed this procedure. However, we observed that in many cases, annotators chose the most salient, or unique object or region in the image. Figure 7 in the appendix compares the distribution of instruction sequence lengths for random and manual selection of targets.
Phenomenon	c	μ	Example from Refer360°
Coreference	96	1.6	on the very upper left corner of the blue part of that window
Comparison	15	0.1	the smaller building to the right of the spire
Sequencing	13	0.1	go right just a smidge and then go up above
Counting	30	0.3	shaped like a football and has 3 silver legs
Allocentric Spatial Mention	46	0.6	find the shelves with books nearest to you
Egocentric Spatial Mention	35	0.5	waldo is sitting on the right side of the window
Direction	92	1.6	look at the knife on the wall to the left
Temporal Condition	13	0.1	turn right until you see a mirror on the wall
3D understanding	22	0.2	counter with the two bar stools sitting in front of it
Inexact/Approximate Language	28	0.2	in front of the white strip at the bottom slightly off center
More than 2 Supporting Objects	47	0.5	now look on the floor in between the table and the chair
Table 8: Linguistic analysis of 100 randomly sampled examples from Refer360°. We annotate each example for the presence and count of each phenomenon.  $c$  is the total number of instructions out of the 100 containing at least one example of the phenomenon.  $\mu$  is the mean number of times each phenomenon appears per instruction sequence.
find the target location using only the last instruction, which may make the first sentences unnecessary. Such redundancy makes it harder to study the core challenges of grounding instructions to visual perception and actions. Thus, we conducted an ablation study with the same pool of annotators using 1K instructions from the dataset. Here we check whether Refer360° has strong dependencies between instructions.
We ran two ablation studies to examine the necessity of using all instruction sentences. For the first study, we ran a finding task with the same pool of annotators, where we provided only the final instruction. For the second study, similarly, we ran another finding task where we provided only the penultimate and the final instruction. We compare the average euclidean distance between the predicted locations and the target location, and the accuracy, i.e. for what percentage of the time the distance between the predicted location and the target location is less than 11 degrees.
Table 7 shows the result of our ablation analysis. Annotators' performance significantly dropped when they can only read the last instruction. They could find the target object only  $37\%$  of the time. Using the penultimate instruction helped them a lot, and they achieved  $63\%$  accuracy. The best performance is achieved when they observe the full instructions. These results show that each instruction is necessary for accurately finding the target location.
5.3 Linguistic Phenomena Observed
Before designing a system to address a language-related task, it is important the understand different kinds of linguistic phenomena observed in the task. We follow the procedure described in TouchdownSDR (Chen et al., 2018), and added a few novel phenomena including 3D understanding, inexact language, and the use of more than two supporting objects as linguistic phenomena. Table 8 shows the result of our analyses for 100 randomly sampled instances. Refer  $360^{\circ}$  requires reasoning for a rich set of linguistic phenomenon including the resolution of the coreference chains, counting objects, a rich set of spatial language phenomena such as multiple-supporting object mentions and 3D scene understanding.
5.4 Localization Experiments
Our analyses in the previous subsections suggest that Refer360° poses several challenges. In Section 5.1, we show that since the target locations are randomly chosen, it is harder to exploit possible location bias. In Section 5.2, we show that it is essential to model the sequential nature of the instructions. Section 5.3 shows that there are lots of interesting linguistic phenomena observed in Refer360°. We want to verify these claims by training the state-of-the-art model and measure its performance on our Refer360° dataset.
We use the same experimental setup in Touchdown-SDR using the scenes provided in the concurrent work (Mehta et al., 2020), where we slice 360scene into 8 FoVs covering the scene. We
7196
pass each of these FoVs to a pre-trained model (He et al., 2016), and extract features from fourth to the last layer before classification to get a feature map representation of the FoVs. We concatenate 8 FoV slices to a single tensor to represent the  $360^{\circ}$  scene.
We use the LingUNet model (Chen et al., 2018; Misra et al., 2018; Blukis et al., 2018), which performs the state-of-the-art results on TouchDownSDR dataset. LingUNet is an image-to-image encoder-decoder model where a language and image representations are fused to predict a probability over the input image. Instructions are fed to bi-directional Long Short-Term Memory (LSTM) recurrent neural network to induce a language representation. To induce fused image-text representations, the input image tensor is passed to a convolutional neural network conditioned on the test representations. The fused representation is then fed to deconvolution layers to predict the location of the target. We use the same accuracy and distance metrics described in Section 5.2.
Dataset	Accuracy (%)	Distance
Touchdown-SDR (reported)	26.1	708
Touchdown-SDR (replication)	23.5	715
Refer360°	13.0	1235
Table 9: Results for the LingUNet on two benchmark datasets. Since LinGUNet designed for observing the full instruction set and the holistic view of the scene, and it performs significantly worse on Refer360°.
As we can see in Table 9, LingUNet performs significantly worse on Refer360°  $^{9}$ . This might be due to the difference we highlighted in earlier sections. First and foremost, instructions must be completed sequentially. However, LingUNet does not model the sequential nature of the task for Refer360°, rather uses all instruction sequence and oracle-view of the  $360^{\circ}$  scene. Second, the scenes in Touchdown-SDR is from a single domain, but in Refer360°, we have a richer set of scenes for both indoor and outdoor.
We designed Refer360° to study 3D spatial language understanding for real scenes. We collected a fine-grained set of annotations that support study at many levels of language grounding. Refer360° is
a versatile dataset and enables investigation along three axes:
- Language: Refer  $360^{\circ}$  enables modeling tasks that study single instruction, multiple instructions, or interactive language where the next instruction is revealed only after reaching an intermediate milestone.
- Vision: Refer  $360^{\circ}$  enables modeling tasks that try to predict targets at different granularities: at the object level if trying to identify the closest object to the target, at the region level in a similar style to Touchdown-SDR, and finally, at the pixel level.
- Action: Refer  $360^{\circ}$  enables modeling tasks where the action space is static with the whole 360 image given upfront, where the action space consists of a sequence of discrete choices between fixed views, and when the action space is continuous, consisting of angles for rotation.
In our experiments, we presented one of these scenarios (single instruction, static, and pixel-level) since it was the closest to the pre-existing Touchdown-SDR system. However, one can also study a much larger number of scenarios and modeling tasks using Refer360°.
We are thankful to anonymous ACL conference reviewers for providing valuable feedback. We thank members of MultiComp Lab at CMU and Berg Lab at UCSD for useful discussions. We thank Howard Chen for helping us replicate their experiments. This material is partially supported by Siemens and the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Siemens or the National Science Foundation, and no official endorsement should be inferred.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960. Association for Computational Linguistics.
<sup>9</sup>We used publicly available code provided by authors to run the experiments. We could not replicate the exact numbers reported in the paper, yet, we use exactly the same setup for both Refer360° and Touchdown-SDR for a fair comparison.
7197
Hazan Anayurt, Sezai Artun Ozyegin, Ulfet Cetin, Utku Aktas, and Sinan Kalkan. 2019. Searching for ambiguous objects in videos using relational referring expressions. In Proceedings of the British Machine Vision Conference (BMVC).
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018a. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077-6086.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48.
Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. 2018. Mapping navigation instructions to continuous control actions with position-visitation prediction. arXiv preprint arXiv:1811.04179.
Joyce Y Chai, Pengyu Hong, and Michelle X Zhou. 2004. A probabilistic approach to reference resolution in multimodal user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces, pages 70-77. ACM.
Joyce Y Chai, Lanbo She, Rui Fang, Spencer Ottarson, Cody Littley, Changsong Liu, and Kenneth Hanson. 2014. Collaborative effort towards common ground in situated human-robot dialogue. In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, pages 33-40. ACM.
Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2018. Touchdown: Natural language navigation and spatial reasoning in visual street environments. arXiv preprint arXiv:1811.12354.
Shih-Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun, and Ming-Hsuan Yang. 2020a. Visual question answering on  $360^{\circ}$  images. In The IEEE Winter Conference on Applications of Computer Vision.
Shih-Han Chou, Yi-Chun Chen, Kuo-Hao Zeng, Hou- Ning Hu, Jianlong Fu, and Min Sun. 2018. Selfview grounding given a narrated 360 video. In Thirty-Second AAAI Conference on Artificial Intelligence.
Shih-Han Chou, Cheng Sun, Wen-Yen Chang, WanTing Hsu, Min Sun, and Jianlong Fu. 2020b. 360-indoor: Towards learning real-world objects in  $360^{\circ}$  indoor equirectangular images. In The IEEE Winter Conference on Applications of Computer Vision.
Volkan Cirik, Taylor Berg-Kirkpatrick, and Louis-Philippe Morency. 2018a. Using syntax to ground referring expressions in natural images. In AAAI, volume to appear.
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirkpatrick. 2018b. Visual referring expression recognition: What do systems actually learn? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 781-787. Association for Computational Linguistics.
Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick. 2015. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467.
Rui Fang, Changsong Liu, and Joyce Yue Chai. 2012. Integrating word acquisition and referential grounding towards physical world interaction. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 109-116. ACM.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. arXiv preprint arXiv:1612.00837.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017a. Learning to reason: End-to-end module networks for visual question answering. arXiv preprint arXiv:1704.05526.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. 2019. Are you looking? grounding to multiple modalities in vision-and-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6551-6557, Florence, Italy. Association for Computational Linguistics.
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2017b. Modeling relationships in referential expressions with compositional modular networks.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4555-4564.
7198
Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016. Revisiting visual question answering baselines. In European conference on computer vision, pages 727-739. Springer.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referit game: Referring to objects in photographs of natural scenes. In EMNLP.
Casey Kennington and David Schlangen. 2017. A simple generative model of incremental reference resolution for situated dialogue. Computer Speech & Language, 41:43-67.
Anna Khoreva, Anna Rohrbach, and Bernt Schiele. 2018. Video object segmentation with language referring expressions. In *Asian Conference on Computer Vision*, pages 123-141. Springer.
Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. 2014. What are you talking about? text-to-image coreference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3558-3565.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11-20.
Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. 2020. Retouchdown: Adding touchdown to streetlearn as a shareable resource for language grounding tasks in street view. arXiv preprint arXiv:2001.03671.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint arXiv:1809.00786.
Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis. 2016. Modeling context between objects for referring expression understanding. In ECCV.
Bryan A. Plummer, Kevin J. Shih, Yichen Li, Ke Xu, Svetlana Lazebnik, Stan Sclaroff, and Kate Saenko. 2018. Revisiting image-language networks for open-ended phrase detection. arXiv:1811.07212.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Yuanka Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton
van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Florian Strub, Harm de Vries, Jeremy Mary, Bilal Piot, Aaron C. Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In International Joint Conference on Artificial Intelligence (IJCAI).
Yu-Chuan Su, Dinesh Jayaraman, and Kristen Grauman. 2016. Pano2vid: Automatic cinematography for watching  $360^{\circ}$  videos. In Proceedings of the Asian Conference on Computer Vision (ACCV).
Erik Wijmans and Yasutaka Furukawa. 2017. Exploiting 2d floorplan for building-scale panorama rgbd alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 308-316.
Tom Williams, Saurav Acharya, Stephanie Schreitter, and Matthias Scheutz. 2016. Situated open world reference resolution for human-robot dialogue. In *The Eleventh ACM/IEEE International Conference on Human Robot Interaction*, pages 311–318. IEEE Press.
Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2012. Recognizing scene viewpoint using panoramic place representation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2695-2702. IEEE.
Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. 2018. Gaze prediction in dynamic 360 immersive videos. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5333-5342.
Hao Yang and Hui Zhang. 2016. Efficient 3d room shape recovery from a single panorama. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5422-5430.
Yang Yang, Shi Jin, Ruiyang Liu, Sing Bing Kang, and Jingyi Yu. 2018. Automatic 3d indoor scene modeling from single panorama. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3926-3934.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018a. Mattnet: Modular attention network for referring expression comprehension. In CVPR.
Youngjae Yu, Sangho Lee, Joonil Na, Jaeyun Kang, and Gunhee Kim. 2018b. A deep ranking model for spatio-temporal highlight detection from a  $360\textcircled{o}$  video. In Thirty-Second AAAI Conference on Artificial Intelligence.
Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2015. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167.
This section presents details omitted in the main document. It includes the details about the annotation task, screenshots for the MTurk interface for annotation tasks, the most detected objects in Refer360° and Touchdown-SDR, and text length for instructions collected with different methods.
A.1 Payment and Incentive Structure
One session of annotation consisted of finding task for 4 scenes and describing task for 4 scenes which took about 15 minutes to complete on average. The base pay for one session was  $2.25. For each instruction sequence that was accurately found by another annotator, we paid a bonus of$ 0.10 to both the annotator who found the location and the annotator who wrote the instruction sequence. Thus, for both the finding and describing task annotators have an interest in performing the task accurately. Next, we provide statistics of the Refer360° dataset.
Annotation Stage	# of Annotators	# of Collected Instructions	# of Verified Instructions
Stage I: Hiring	256	854	n\a
Stage II: Collection & Verification	86	20630	14062
Stage III: Verification	86	n\a	3073
Table 10: Statistics for data collection stages. Stage I is for hiring annotators. Stage II is for collecting and verifying the instructions. Last stage is further verifying hard instances that are not verified II.
List of Instruction Sentences