SlowGuess/ACL_2020_OCR huggingface dataset:
Volkan Cirik $^{1}$  Taylor Berg-Kirkpatrick $^{2}$  Louis-Philippe Morency $^{1}$
$^{1}$ Carnegie Mellon University  $^{2}$ University of California San Diego {vcirik,morency}@cs.cmu.edu {tberg}@eng.ucsd.edu
We propose a novel large-scale referring expression recognition dataset, Refer360°, consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in  $360^{\circ}$  scenes. Refer360° differs from existing related datasets in three ways. First, we propose a more realistic scenario where instructors and the followers have partial, yet dynamic, views of the scene - followers continuously modify their field-of-view (FoV) while interpreting instructions that specify a final target location. Second, instructions to find the target location consist of multiple steps for followers who will start at random FoVs. As a result, intermediate instructions are strongly grounded in object references and followers must identify intermediate FoVs to find the final target location correctly. Third, the target locations are neither restricted to predefined objects nor chosen by annotators; instead, they are distributed randomly across scenes. This "point anywhere" approach leads to more linguistically complex instructions, as shown in our analyses. Our examination of the dataset shows that Refer360° manifests linguistically rich phenomena in a language grounding task that poses novel challenges for computational modeling of language, vision, and navigation.
Imagine a scenario in which you are asked to retrieve medication from a bathroom. 'First, face the sink, then find the second drawer in the cabinet to your left. The pills should be inside that drawer behind the toothbrush.' Interpreting instruction sequences in order to locate targets in novel environments is challenging for AI systems (e.g. personal robots and self-driving cars). First, the system needs to ground the instructions into visual perception (Anderson et al., 2018b; Hu et al.,
Figure 1: An example from Refer  $360^{\circ}$ . Orange frames represent the field-of-view (FoV) of the follower after interpreting each instruction. Numbers in the frames represent the sequential order. Green lines show how FoVs change continuously. After each instruction, the follower changes the FoV to align with what the instruction describes. Please see Figure 2a to see the correct location of Waldo.
2019). This often requires identification of the mentioned object (Plummer et al., 2015) through physical relationships with surrounding objects (Hu et al., 2017b; Cirik et al., 2018a). Second, since human visual perception has limited field-of-view, instructions are often sequential: First, the correct FoV should be identified before searching for the final target. In many situations, the target location is not visually unique (e.g. in the middle of a plain wall), and several intermediate instructions are required.
To study these challenges, we introduce a novel dataset, named Refer360 $^{\circ 1}$ , for the task of localizing a target in  $360^{\circ}$  scenes given a sequence of instructions. Figure 1 presents an example scenario
Refer360° : A Referring Expression Recognition Dataset in 360° Images
1The annotations, learning simulator, and annotation setup are publicly available for further research https: //github.com/volkancirik/refer360.
7189
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7189-7202 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Examples are from (a) Refer360° (b) Touchdown-SDR, and (c) Google-Ref datasets. In Refer360°, the target location could be any random location on the image. In (b), annotators chose an existing object as the target location. In (c), boxes for objects were used as targets. Refer360° also seeks to increase the complexity of instruction following, making it more realistic by introducing a partial and dynamic FoV rather than providing a holistic oracle-like view of the image.
from Refer360°. For this scenario, finding the target location requires first finding the door leading outside, then looking at the coffee pot, and finally finding the trash can, which is the nearest object to the target. Here, instructions are given from the perspective of a partial field of view (FoV) of the scene, and these FoVs can dynamically be changed. Thus, the correct interpretation of the sequence of instructions will require reasoning about what is currently visible in the FoV (e.g., grounding of objects) but also what is not visible yet. These scenarios will often require adjusting the FoV based on intermediate instructions. An important feature of the Refer360° dataset is that the target location is not an object; instead, it can be any point in the scene, which makes the grounding task more challenging since it is harder to describe a location when we cannot readily refer to it with the name of an object.
Refer  $360^{\circ}$  consists of 17,137 instruction sequences with ground-truth actions to complete these instructions in  $360^{\circ}$  scenes. Refer  $360^{\circ}$  has some unique characteristics which differentiate it
from prior work. First, Refer360° allows the scene to be viewed through a partial FoV that can be dynamically changed as instructions are followed. This is in contrast with existing  $360^{\circ}$  scene-based datasets such as Touchdown-SDR (Chen et al., 2018) and 2D image-based referring expression datasets (Kazemzadeh et al., 2014; Hu et al., 2016; Mao et al., 2016), where the visual input is either fixed, corresponding to a holistic, oracle-like view, or consists of fixed, cardinal FoVs. The partial and dynamic FoV in Refer360° poses new challenges for language grounding (see Figure 2a, 2b, and 2c for an illustrative comparison). For instance, the mentioned objects may not be visible in the current FoV, and language may refer to the FoV itself. Further, since our annotators generate instructions while observing a partial and dynamic FoV, and do so for a follower whose first FoV will be initially located at random, the instruction following task is strongly sequential. To interpret the sequence of instructions to find the target correctly, a follower must reason about the sequence of FoVs referenced by the instructor.
7190
Dataset	Target Location Selection	Field of View (FoV)	Action Space	Intermediate Steps
Refer360°	Random Points	Dynamic with partial FoV	4 Directions	✓
Touchdown-SDR (Chen et al., 2018)	Human Selected Points	Oracle: Holistic & Static, 360°scenes	✕	✕
Google-Ref (Mao et al., 2016)	Annotated Objects	2D Images	✕	✕
Ref-UNC (Kazemzadeh et al., 2014)	Annotated Objects	2D Images	✕	✕
Table 1: Comparison of referring expression datasets, including our proposed Refer360° dataset. Refer360° poses a more challenging scenario where the system observes only a partial and dynamic FoV. Refer360° also has includes explicit alignments between intermediate instruction steps and human follower actions which can be used as an auxiliary evaluation metric or source of supervision.
Second, unlike other datasets, the target locations in Refer360° are randomly distributed and thus may occur anywhere – not just on predetermined objects. As a result, target locations are less prone to bias (Devlin et al., 2015; Agrawal et al., 2016; Jabri et al., 2016; Goyal et al., 2016; Cirik et al., 2018b). These random locations lead to more linguistically complex instructions, as shown in our analyses – when instead annotators choose the target location, they are likely to be biased towards locations that are more easily described (e.g. on top of a named object). Table 1 shows a comparison of similar datasets. In the following section, we motivate Refer360° dataset in more detail.
The vision behind Refer360° is to build systems that perform localization of any point in 3D space, bringing us closer to human-like reasoning. This is an important milestone towards better collaboration between AI systems (e.g. personal robots) and humans, allowing them to act within the same space. It might also pave the way for AI-agents interacting with virtual worlds. The Refer360° dataset was designed to address three technical challenges towards this vision.
First, learning environments we create need to reflect the characteristics of human's perception of 3D space. In such an environment, the agent only observes a partial FoV of the scene. This requires adjusting the FoV in accordance with instructions so that current view and instructions are aligned. The agent's FoV can be changed in a continuous manner, moving smoothly left, right, up, and down. This is analogous to a real-world robot performing motor actions to change its camera position, or a human changing their head's pitch and yaw. Further, real scenes are 3D, but the FoV is represented in 2D in our task. Thus, interpreting some instructions will require inferences about depth.
Second, the paradigm of  $360^{\circ}$  scenes with par
tial FoV will almost always necessitate instructions that consist of multiple intermediate steps. As the first intermediate step, the follower and instructor need to find a common referential FoV. Then, the instructor can continue giving guidance towards the target location, often by identifying objects that are physically related to the target location. This multi-step process can serve as a natural benchmark for measuring whether systems achieve localization through a human-like process of progressively getting closer to the target location by interpreting intermediate steps. In other words, this setup may helps researchers make sure that our systems are arriving at the referred location for the right reasons.
Third, since any point in the scene could be of interest, instructions will be more complex: many points in the scene will not correspond to easily named objects, and thus, when such points are allowed as targets, more sophisticated instructions will be required to unambiguously refer to them. The instructor may rely on description of physical relationships with the closest easily named locations in the scene (Nagaraja et al., 2016; Hu et al., 2017b; Cirik et al., 2018b). For instance, in Figure 2a, the target location is on the side of a trash bin, which is difficult to unique describe with a single word or a short phrase. In this case, the instructor may use the distance to the floor or to another object in the scene in order to describe the exact location of the target. This will additionally introduce description of degree (e.g. 'slightly above', 'a few inches away from') rather than more discrete spatial relationships (e.g. 'on top of the desk').
Referring expression recognition. Grounding a short phrase or a sentence into a visual modality such as video (Khoreva et al., 2018; Anayurt et al., 2019) or imagery (Kong et al., 2014; Plummer et al., 2015, 2018; Yu et al., 2018a) is a well
7191
studied problem in intelligent user interfaces (Chai et al., 2004), human-robot interaction (Fang et al., 2012; Chai et al., 2014; Williams et al., 2016), and situated dialogue (Kennington and Schlangen, 2017). Kazemzadeh et al. (2014), Hu et al. (2017a), and Mao et al. (2016) introduce two benchmark datasets for the real-world 2D images. Nagaraja et al. (2016) propose a model where the target and supporting objects (i.e. objects that are mentioned in order to disambiguate the target object) are identified and scored jointly. Hu et al. (2017b) introduce a compositional approach where they assume that the referring expression can be decomposed into a triplet consisting of the target object, the supporting object, and their spatial relationship. Similarly, Cirik et al. (2018a) propose a type of neural modular network (Andreas et al., 2016) where the grounding of referring expression depends on the parse tree of the input referring expression to learn to ground an unconstrained number of supporting objects.
360° Scenes. Although  $360^{\circ}$  scenes are well studied in the computer vision domain (Xiao et al., 2012; Su et al., 2016; Wijmans and Furukawa, 2017; Yang and Zhang, 2016; Xu et al., 2018; Yang et al., 2018; Yu et al., 2018b), few studies explore the challenges of  $360^{\circ}$  scenes in the context of language grounding. Chou et al. (2018) introduce a dataset where  $360^{\circ}$  videos are narrated. They address the task of predicting the field of view for the given narration. Anderson et al. (2018b) introduce the vision and language navigation task for simulated indoor environments where an agent is placed in a location in a house and follows the instructions to go to a target location. Here the agent observes a discretized view of the current location (i.e. the  $360^{\circ}$  scene is split into a fixed number of field of views). The most related work to Refer360° is Touchdown (Chen et al., 2018) which introduces two tasks: a vision and language navigation task and a spatial description resolution (SDR) task (i.e. a referring expression recognition task for a simulated outdoor environment). In contrast with Touchdown, in our setup instructors, followers, and learning systems observe a partial FoV of the scene, but they can change the FoV continuously to explore the scene. This approach yields instructions with a stronger sequential dependencies and with stronger reference to the FoV itself. We demonstrate some of these differences in analysis in Section 5. Concurrent work studies visual
question answering (Chou et al., 2020a) and object detection (Chou et al., 2020b) for  $360^{\circ}$ scenes. Another concurrent study (Qi et al., 2020) combines vision-and-language navigation and referring expression recognition into one task where the system is asked to localize the referred object after navigating to another point in a real images of rendered buildings.
4 Refer360° Dataset
In this section, we describe the details of the Refer360° dataset, a vision-and-language benchmark for localizing a target point in a panoramic image. Refer360° consists of 17,137 instruction sequences that describe randomly distributed target locations in 2,000 panoramic scenes from the SUN360 (Xiao et al., 2012) dataset. We first explain the annotation procedure for collecting and validating the instruction sequences. Later, we discuss the statistics of the Refer360° dataset.
4.1 Annotation Procedure
Annotation of the Refer360° dataset was carried out in three stages on Amazon Mechanical Turk with two tasks, namely a description task and a finding task. First we describe the two tasks in more detail.
Description Task. Our main goal is to collect instructions for finding any point in a  $360^{\circ}$  image. Annotators started this task looking at the ceiling of the  $360^{\circ}$  image with a random yaw $^{2}$ . We asked them to find the target location for which we use an icon of Waldo $^{3}$ . Target locations are chosen randomly – we discuss the details of this design choice in Section 4.2. The target can be at any longitude and can have a latitude within a range of 45 degrees from the top and bottom of the 360 image. This restriction in latitude is made for two reasons: (1) visual distortions happen at extreme points, and (2) during the finding task, the starting point is the “ceiling” of the 360 image. Annotators were asked to give instructions to find the target location using at least three instructions $^{4}$ .
Finding Task. We design this task to verify the quality of instruction sequences provided by anno
2We wanted to avoid introducing any bias by beginning the same position each time for each scene.
3https://en.wikipedia.org/wiki/Where%27s_Wally%3F
4Please see Figure 5 in Appendix to see a screenshot of the user interface we build for this task.
7192
tators in the description task. We asked annotators to complete the instruction sequences sentence by sentence. The initial field of view of annotators is always pointing at the ceiling of the  $360^{\circ}$  image with a random yaw. We asked annotators to change the FoV after each instruction so that the center of the FoV points to the location the intermediate instruction is describing. After moving the FoV to the correct position, annotators clicked a button to read the next instruction. We recorded the spherical coordinates of the center of the FoV after each instruction. As a result, our annotations include aligned intermediate steps that find the target location. After the final instruction, the annotators predicted the target location by changing the center of FoV or clicking on the FoV.
We collected and verified the quality of our data in three stages using description and finding tasks. In the first stage, we sought a pool of annotators providing high-quality annotations. For the second, aimed to collect a large number of annotations and verify their quality. In the third stage, we further verified instruction sequences that were not verified in the second stage.
Stage I. In this stage, we asked annotators to complete the finding task for four different scenes. We wrote the instruction sequences for this stage's finding task to give annotators an example of instruction sequences for describing the target location. Then, annotators completed the description task for 4 different scenes. A total of 256 annotators participated in this first stage. We manually inspected each instruction sequences provided by these annotators for their quality of descriptions of the target location and reduced the pool of annotators to 86.
Stage II. In this stage, for each annotation session, we asked annotators first to find the target location for four different scenes, and later, describe the target location four times for different scenes<sup>5</sup>. We used the finding task to verify the quality of the instruction sequences. If an annotator predicts the target location within a radius of 11 degrees in spherical coordinates, which is roughly equal to the size of the Waldo icon we used, we counted that instance as verified.
Stage III. After the second stage, we have some instructions where the annotators could not find
Scene Type	Scene Location	# of Images
Restaurant	Indoor	500
Shop	Indoor	250
Expo Showroom	Indoor	250
Living Room	Indoor	250
Bedroom	Indoor	250
Street	Outdoor	250
Plaza Courtyard	Outdoor	250
Table 2: Statistics for Panoramic Images used in Refer360° dataset.
the target accurately. This could mean either the instructions are not clear, or it is actually harder to find the target location with these instruction sequences. In the third stage, we did another round of the finding tasks to verify these harder instruction sequences.
After these three stages, we have a total of 17,137 instruction sequences in which at least one annotator was able to find the target location accurately. Statistics for data collection in these stages and the payment structure is in the Appendix.
We split our presentation of dataset statistics into two parts: namely, scene statistics and language statistics.
Scene Statistics: To investigate the challenges in localizing a target location for both indoor and outdoor scenes as well as for different kinds of indoor and outdoor scene categories, we use seven scene categories from the SUN360 (Xiao et al., 2012) dataset. We use total of 2,000 scenes. Table 2 shows the distribution of scene categories that comprise the Refer360° dataset.
We want to analyze the richness of the scenes in the Refer360° dataset and compare it with Touchdown-SDR. The domain of the scenes will affect the instruction one needs to use to describe a target location. To be more specific, when annotators give instructions, they use supporting objects as anchor points to help guide the attention of the follower. Thus, the availability of a rich set of objects is essential for describing the target location. Since the annotation of objects in  $360^{\circ}$  images is a laborious task itself, we use an off-the-shelf object detection method (Anderson et al., 2018a) to annotate scenes with objects. We split  $360^{\circ}$  images into 12 different 2D images covering the  $360^{\circ}$  view<sup>6</sup>. This provides us a proxy to analyze the
<sup>6</sup>We fixed the confidence threshold for detection of objects
5 Annotators never observed their own instruction sequences while doing the finding tasks.
7193
kind of objects usually observed in  $360^{\circ}$  images in Touchdown-SDR and Refer  $360^{\circ}$ .
Dataset	Avg # of Objects	Object Type PPL
Touchdown-SDR	93.81	15.93
Refer360°	62.44	42.93
Table 3 shows the average number of objects and the perplexity of the distribution of detected objects per  $360^{\circ}$  scene used in Refer360° and Touchdown-SDR datasets. As expected, the average number of detected objects in Touchdown-SDR scenes is higher than in Refer360° because all scenes depict outdoor settings from Google's StreetView API. However, this analysis shows that Refer360° has much larger diversity of object types and therefore will likely have greater lexical diversity in instructions.
Table 3: Statistics for detected objects per image in Touchdown-SDR and Refer  ${360}^{ \circ  }$  . On average,Refer  ${360}^{ \circ  }$  images contain fewer of objects. However, these objects are from a wider variety of object types.
Scenes	Dataset	Avg. Text Length	Vocab. Size	Size
360°	Refer360°	43.80	11220	17,137
360°	Touchdown-SDR	26.97	5705	9325
2D	Guess What?!	24.99	27713	160745
2D	Google-Ref	8.46	12108	142210
2D	Refer-UNC	3.51	21305	414138
Language Statistics: Refer  $360^{\circ}$  contains a total of 17,137 instruction sequences (8.57 per scene) describing target locations. Table 4 shows language statistics for Refer  $360^{\circ}$  and other referring expression recognition datasets. Refer  $360^{\circ}$  is bigger than Touchdown-SDR, yet, smaller than other datasets. This is because it is a more time-intensive and costly process to annotate and validate  $360^{\circ}$  images compared with 2D images.
Figure 3 shows the distribution of text length for the instructions. Compared to other referring expression recognition and image captioning datasets, Refer  $360^{\circ}$  contains the longest instructions on average. This is a result of two differences with previous tasks. First, previous datasets use the entire scene as a single field of view. Thus, there is reduced need to describe how to find the target loca
Figure 3: Distribution of the number of tokens for vision-and-language datasets similar to Refer360°
Table 4: Language statistics for Refer360° dataset and other referring expression recognition datasets.
Split	# of Instances
Train	13287
Validation Seen	900
Validation Unseen	1009
Test Seen	900
Test Unseen	1041
Table 5: Statistics for dataset splits in Refer360° dataset.
tion sequentially. In Touchdown-SDR, the recognition system or human annotator needs to find an FoV that includes the target location. In Refer360°, the finding task is carried out sequentially; thus, each instruction needs to be completed accurately to be able to find the target location. Second, in Refer360°, the target location is randomly distributed in scenes. As seen in Table 6, when the target location is randomly selected, the target location is on average further from other objects (we discuss this in more detail in Section 5.1).
Dataset Splits: We use a similar train, validation, and test split strategy as the Room-to-Room dataset (Anderson et al., 2018b). We reserve a subset of images from each scene category for validation and test splits for unseen scene evaluation i.e. these scenes are not observed in the train split to study generalization capabilities of models. The remaining scenes are pooled together for training, validation, and test splits for seen scenes evaluation. Table 5 shows statistics for the splits. Following the previous studies, the ground-truth annotations for test splits will not be released. Instead, we will provide an evaluation server where model predictions may be uploaded for scoring.
to 0.5 and maximum number of objects to 20. In the appendix, Figure 6 shows the most detected objects for Refer360° and Touchdown-SDR datasets.
7194
5 Analyses