SlowGuess/NAACL_2021_OCR huggingface dataset:
Roshanak Mirzaee\* Hossein Rajaby Faghihi\* Qiang Ning\* Parisa Kordjamshidi\*
\*Michigan State University \*Amazon
{mirzaeem,rajabyfa,kordjams}@msu.edu qning@amazon.com
This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.
Spatial reasoning is a cognitive process based on the construction of mental representations for spatial objects, relations, and transformations (Clements and Battista, 1992), which is necessary for many natural language understanding (NLU) tasks such as natural language navigation (Chen et al., 2019; Roman Roman et al., 2020; Kim et al., 2020), human-machine interaction (Landsiedel et al., 2017; Roman Roman et al., 2020), dialogue systems (Udagawa et al., 2020), and clinical analysis (Datta and Roberts, 2020).
Modern language models (LM), e.g., BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and XLNet (Yang et al., 2019) have seen great successes in natural language processing (NLP). However, there has been limited investigation into spatial reasoning capabilities of LMs. To the best of our knowledge, bAbI (Weston et al., 2015) (Fig 9) is the only dataset with direct textual spatial question answering (QA) (Task 17), but it is synthetic
and overly simplified: (1) The underlying scenes are spatially simple, with only three objects and relations only in four directions. (2) The stories for these scenes are two short, templated sentences, each describing a single relation between two objects. (3) The questions typically require up to two-steps reasoning due to the simplicity of those stories.
To address these issues, this paper proposes a new dataset, SPARTQA<sup>1</sup> (see Fig. 1). Specifically, (1) SPARTQA is built on NLVR's (Suhr et al., 2017) images containing more objects with richer spatial structures (Fig. 1b). (2) SPARTQA's stories are more natural, have more sentences, and richer in spatial relations in each sentence. (3) SPARTQA's questions require deeper reasoning and have four types: find relation (FR), find blocks (FB), choose object (CO), and yes/no (YN), which allows for more fine-grained analysis of models' capabilities.
We showed annotators random images from NLVR, and instructed them to describe objects and relationships not exhaustively at the cost of naturalness (Sec. 3). In total, we obtained 1.1k unique QA pair annotations on spatial reasoning, evenly distributed among the aforementioned types. Similar to bAbI, we keep this dataset in relatively small scale and suggest to use as little training data as possible. Experiments show that modern LMs (e.g., BERT) do not perform well in this low-resource setting.
This paper thus proposes a way to obtain distant supervision signals for spatial reasoning (Sec. 4). As spatial relationships are rarely mentioned in existing corpora, we take advantage of the fact that spatial language is grounded to the geometry of visual scenes. We are able to automatically generate stories for NLVR images (Suhr et al., 2017) via our newly designed context free grammars (CFG) and context-sensitive rules. In the process of story generation, we store the information about all ob
*Work was done while at the Allen Institute for AI.
1 SPAtial Reasoning on Textual Question Answering.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582-4598 June 6-11, 2021. ©2021 Association for Computational Linguistics
STORY:
We have three blocks, A, B and C. Block B is to the right of block C and it is below block A. Block A has two black medium squares. Medium black square number one is below medium black square number two and a medium blue square. It is touching the bottom edge of this block. The medium blue square is below medium black square number two. Block B contains one medium black square. Block C contains one medium blue square and one medium black square. The medium blue square is below the medium black square.
QUESTIONS:
FB: Which block(s) has a medium thing that is below a black square? A, B, C
FB: Which block(s) doesn't have any blue square that is to the left of a medium square? A, B
FR: What is the relation between the medium black square which is in block C and the medium square that is below a medium black square that is touching the bottom edge of a block? Left
CO: Which object is above a medium black square? the medium black square which is in block C or medium black square number two? medium black square number two
YN: Is there a square that is below medium square number two above all medium black squares that are touching the bottom edge of a block? Yes
(a) An example story and corresponding questions and answers.
Figure 1: Example from SPARTQA (specifically from SPARTQA-AUTO)
(b) An example NLVR image and the scene created in Fig. 1a, where the blocks in the NLVR image are rearranged.
jects and relationships, such that QA pairs can also be generated automatically. In contrast to bAbI, we use various spatial rules to infer new relationships in these QA pairs, which requires more complex reasoning capabilities. Hereafter, we call this automatically-generated dataset SPARTQA-AUTO, and the human-annotated one SPARTQA-HUMAN.
Experiments show that, by further pretraining on SPARTQA-AUTO, we improve LMs' performance on SPARTQA-HUMAN by a large margin. The spatially-improved LMs also show stronger performance on two external QA datasets, bAbI and boolQ (Clark et al., 2019): BERT further pretrained on SPARTQA-AUTO only requires half of the training data to achieve  $99\%$  accuracy on bAbI as compared to the original BERT; on boolQ's development set, this model shows better performance than BERT, with  $2.3\%$  relative error reduction.
Our contributions can be summarized as follows. First, we propose the first human-curated benchmark, SPARTQA-HUMAN, for spatial reasoning with richer spatial phenomena than the prior synthetic dataset bAbI (Task 17).
Second, we exploit the scene structure of images and design novel CFGs and spatial reasoning rules to automatically generate data (i.e., SPARTQA-AUTO) to obtain distant supervision signals for spatial reasoning over text.
Third, SPARTQA-AUTO proves to be a rich source of spatial knowledge that improved the performance of LMs on SPARTQA-HUMAN as well as on different data domains such as bAbI and boolQ.
Question answering is a useful format to evaluate machines' capability of reading comprehension (Gardner et al., 2019) and many recent works have been implementing this strategy to test machines' understanding of linguistic formalisms: He et al. (2015); Michael et al. (2018); Levy et al. (2017); Jia et al. (2018); Ning et al. (2020); Du
$^{2}$ Further pretraining LMs has become a common practice and baseline method for transferring knowledge between tasks (Phang et al., 2018; Zhou et al., 2020). We leave more advanced methods for future work.
3To the best of our knowledge, the test set or leaderboard of boolQ has not been released yet.
and Cardie (2020). An important advantage of QA is using natural language to annotate natural language, thus having the flexibility to get annotations on complex phenomena such as spatial reasoning. However, spatial reasoning phenomena have been covered minimally in the existing works.
To the best of our knowledge, Task 17 of the bAbI project (Weston et al., 2015) is the only QA dataset focused on textual spatial reasoning (examples in Appendix F). However, bAbI is synthetic and does not reflect the complexity of the spatial reasoning in natural language. Solving Task 17 of bAbI typically does not require sophisticated reasoning, which is an important capability emphasized by more recent works (e.g., Dua et al. (2019); Khashabi et al. (2018); Yang et al. (2018); Dasigi et al. (2019); Ning et al. (2020)).
Spatial reasoning is arguably more prominent in multi-modal QA benchmarks, e.g., NLVR (Suhr et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), CLEVR (Johnson et al., 2017). However, those spatial reasoning phenomena are mostly expressed naturally through images, while this paper focuses on studying spatial reasoning on natural language. Some other works on visual-spatial reasoning are based on geographical information inside maps and diagrams (Huang et al., 2019) and navigational instructions (Chen et al., 2019; Anderson et al., 2018).
As another approach to evaluate spatial reasoning capabilities of models, a dataset proposed in Ghanimifard and Dobnik (2017) generates a synthetic training set of spatial sentences and evaluates the models' ability to generate spatial facts and sentences containing composition and decomposition of relations on grounded objects.
3 SPARTQA-HUMAN
To mitigate the aforementioned problems of Task 17 of bAbI, i.e., simple scenes, stories, and questions, we describe the data annotation process of SPARTQA-HUMAN, and explain how those problems were addressed in this section.
First, we randomly selected a subset of NLVR images, each of which has three blocks containing multiple objects (see Fig 1b). The scenes shown by these images are more complicated than those described by bAbI because (1) there are more objects in NLVR images; (2) the spatial relationships in NLVR are not limited to just four relative directions as objects are placed arbitrarily within blocks.
Figure 2: For "A blue circle is above a big triangle. To the left of the big triangle, there is a square," if the question is: "Is the square to the left of the blue circle?", the answer is neither Yes nor No. Thus, the correct answer is "Do not Know" (DK) in our setting.
Second, two student volunteers produced textual description of those objects and their corresponding spatial relationships based on these images. Since the blocks are always horizontally aligned in each NLVR image, to allow for more flexibility, annotators could also rearrange these blocks (see Fig. 1a). Relationships between objects within the same block can take the forms of relative direction (e.g., left or above), qualitative distance (e.g., near or far), and topological relationship (e.g., touching or containing).
However, we instructed the annotators not to describe all objects and relationships, (1) to avoid unnecessarily verbose stories, and (2) to intentionally miss some information to enable more complex reasoning later. Therefore, annotators describe only a random subset of blocks, objects, and relationships.
To query more interesting phenomena, annotators were then encouraged to write questions requiring detecting relations and reasoning over them using multiple spatial rules. A spatial rule can be one of the transitivity  $(A\to B,B\to C\Rightarrow$ $A\to C)$  symmetry  $(A\rightarrow B\Rightarrow B\rightarrow A)$  , converse  $((A,R,B)\Rightarrow (B,\text{reverse}(R),A))$  , inclusion (obj1 in  $A$  ), and exclusion (obj1 not in  $B$  ) rules.
There are four types of questions (Q-TYPE). (1)  $FR$ : find relation between two objects. (2)  $FB$ : find the block that contains certain object(s). (3)  $CO$ : choose between two objects mentioned in the question that meets certain criteria. (4)  $YN$ : a yes/no question that tests if a claim on spatial relationship holds.
FB, FR, and CO questions are formulated as multiple-choice questions<sup>4</sup> and receive a list of candidate answers, and YN questions' answer is choosing from Yes, No, or "DK" (Do not Know). The "DK" option is due to the open-world assumption of the stories, where if something is not described
${}^{4}\mathrm{CO}$  can be considered as both single-choice and multiple-choices question.
Sets	FB	FR	YN	CO	Total
SPARTQA-HUMAN:
Test	104	105	194	107	510
Train	154	149	162	151	616
SPARTQA-AUTO:
Seen Test	3872	3712	3896	3594	15074
Unseen Test	3872	3721	3896	3598	15087
Dev	3842	3742	3860	3579	15023
Train	23654	23302	23968	22794	93673
Table 1: Number of questions per Q-TYPE
in the text, it is not considered as false (See Fig. 2).
Finally, annotators were able to create 1.1k QA pairs on spatial reasoning on the generated descriptions, distributed among the aforementioned types. We intentionally keep this data in a relatively small scale due to two reasons. First, there has been some consensus in our community that modern systems, given their sufficiently large model capacities, can easily find shortcuts and overfit a dataset if provided with a large training data (Gardner et al., 2020; Sen and Saffari, 2020). Second, collecting spatial reasoning QAs is very costly: The two annotators spent 45-60 mins on average to create a single story with 8-16 QA pairs. We estimate that SPARTQA-HUMAN costed about 100 human hours in total. The expert performance on 100 examples of SPARTQA-HUMAN's test set measured by their accuracy of answering the questions is  $92\%$  across four Q-TYPES on average, indicating its high quality.
4 Distant Supervision: SPARTQA-AUTO
Since human annotations are costly, it is important to investigate ways to generate distant supervision signals for spatial reasoning. However, unlike conventional distant supervision approaches (e.g., Mintz et al. (2009); Zeng et al. (2015); Zhou et al. (2020)) where distant supervision data can be selected from large corpora by implementing specialized filtering rules, spatial reasoning does not appear often in existing corpora. Therefore, similar to SPARTQA-HUMAN, we take advantage of the ground truth of NLVR images, design CFGs to generate stories, and use spatial reasoning rules to ask and answer spatial reasoning questions. This automatically generated data is called SPARTQA-AUTO, and below we describe its generation process in detail.
Story generation Since NLVR comes with structured descriptions of the ground truth locations of those objects, we were able to choose random
blocks and objects from each image programmatically. The benefit is two-fold. First, a random selection of blocks and objects allows us to create multiple stories for each image; second, this randomness also creates spatial reasoning opportunities with missing information.
Once we decide on a set of blocks and objects to be included, we determine their relationships: Those relationships between blocks are generated randomly; as for those between objects, we refer to the ground truth of these images to determine them.
Now we have a scene containing a set of blocks and objects and their associated relationships. To produce a story for this scene, we design CFGs to produce natural language sentences that describe those blocks/objects/relationships in various expressions (see Fig. 3 for two portions of our CFG describing relative and nested relations between objects).
The big black shape is above the medium triangle.
$$
S \rightarrow <   A r t i c l e > <   O b j e c t > i s <   R e l a t i o n > <   A r t i c l e > <   O b j e c t >.
$$
$$
\begin{array}{l} \text {A r t i c l e} \quad \rightarrow \text {t h e} | a \\ \text {R e l a t i o n} \rightarrow \text {a b o v e} | \text {l e f t} | \dots \\ O b j e c t \quad \longrightarrow <   S i z e > ^ {*} <   C o l o r > ^ {*} <   S h a p e | I n d _ {s h a p e} > \\ \text {S i z e} \quad \rightarrow \quad \text {s m a l l} \mid \text {m e d i u m} \mid \text {b i g} \\ \text {C o l o r} \quad \longrightarrow \text {y e l l o w} \mid \text {b l u e} \mid \text {b l a c k} \\ \begin{array}{c c} \text {S h a p e} & \longrightarrow \text {s q u a r e} | \text {t r i a n g l e} | \text {c i r c l e} \end{array} \\ I n d \_ s h a p e \longrightarrow s h a p e | o b j e c t | t h i n g \\ \end{array}
$$
(a) Part of the grammar describing relations between objects
The big black shape is above the object that is to the right of the medium triangle
$$
\begin{array}{l} S \to <   A r t i c l e > <   O b j e c t > i s <   R e l a t i o n > <   A r t i c l e > \\ <   O b j e c t >. \end{array}
$$
$$
O b j e c t \rightarrow <   S i z e > ^ {*} <   C o l o r > ^ {*} <   S h a p e | I n d _ {s h a p e} > |
$$
$$
<   I n d \_ s h a p e > t h a t i s <   R e l a t i o n > <   O b j e c t >
$$
(b) Part of the grammar describing nested relationships.
Figure 3: Two parts of our designed CFG
Being grounded to visual scenes guarantees spatial coherency in a story, and using CFGs helps to have correct sentences (grammatically) and various expressions. We also design context-sensitive rules to limited options for each CFG's variable based on the chosen entities (e.g. black circle), or what is described in the previous sentences (e.g. Block A has  $a$  circle. The circle is below  $a$  triangle.)
Question generation To generate questions based on a passage, there are rule-based sys
Figure 4: Find the implicit relation between obj1 and obj4 by Transitivity rule. (1) Find a set of objects that have a relation with obj1. Continue the same process on the new set until obj4 is found. (2) Get the union of the intermediate relations between these two objects and it is the final answer.
tems (Heilman and Smith, 2009; Labutov et al., 2015), neural networks (Du et al., 2017), and their combinations (Dhole and Manning, 2020). However, in our approach, during generating each story, the program stores the information about the entities and their relationships. Thus, without processing the raw text, which is error-prone, we generate questions by only looking at the stored data. The question generation operates based on four primary functionalities, Choose-objects, Describe-objects, Find-all-relations, and Find-similar-objects. These modules are responsible to control the logical consistency, correctness, and the number of steps required for reasoning in each question.
Choose-objects randomly chooses up to three objects from the set of possible objects in a story under a set of constraints such as preventing selection of similar objects, or excluding objects with relations that are directly mentioned in the text.
Describe-Objects generates a mention phrase for an object using parts of its full name (presented in the story). The generated phrase is either pointing to a unique object or a group of objects such as "the big circle," or "big circles." To describe a unique object, it chooses an attribute or a group of attributes that apply to a unique object among others in the story. To increase the steps of reasoning, the description may include the relationship of the object to other objects instead of using a direct unique description. For example, "the circle which is above the black triangle."
Find-all-relations completes the relationship graph between objects by applying a set of spatial rules such as transitivity, symmetry, converse, inclusion, and exclusion on top of the direct relations described in the story. As shown in Fig. 4, it does an exhaustive search over all combinations of the relations that link two objects to each other.
Find-similar-objects finds all the mentions matching a description from the question to objects
in the story. For instance, for the question "is there any blue circle above the big blue triangle?", this module finds all the mentions in the story matching the description "a blue circle".
Similar to the SPARTQA-HUMAN, we provide four Q-TYPES FR, FB, CO, and YN. To generate FR questions, we choose two objects using Choose-objects module and question their relationships. The YN Q-TYPE is similar to FR, but the question specifies one relationship of interest chosen from all relation extracted by Find-all-relations module to be questioned about the objects. Since most of the time, Yes/No questions are simpler problems, we make this question type more complex by adding quantifiers (adding "all" and "any"). These quantifiers help to evaluate the models' capability to aggregate relations between more than two objects in the story and do the reasoning over all find relations to find the final answer. In FB Q-TYPE, we mention an object by its indirect relation to another object using the nested relation in Describe-objects module and ask to find the blocks containing or not containing this object. Finally, the CO question selects an anchor object (Choose-objects) and specifies a relationship (using Find-all-relations) in the question. Two other objects are chosen as candidates to check whether the specified relationship holds between them and the anchor object. We tend to force the algorithm to choose objects as candidates that at least have one relationship to the anchor object. To see more details about different question' templates see Table 7 in the Appendix.
Answer generation We compute all direct and indirect relationships between objects using Find-all-relations function and based on the Q-TYPES generate the final answer.
For instance, in YN Q-TYPE if the asked relation exists in the found relations, the answer is "Yes", if the inverse relation exists it must be "No", and otherwise, it is "DK".
4.1 Corpus Statistics
We generate the train, dev, and test set splits based on the same splits of the images in the NLVR dataset. On average, each story contains 9 sentences (Min:3, Max: 22) and 118 tokens (Min: 66,
5The SPARTQA-AUTO generation code and the file of dataset are available at https://github.com/HLR/SpartQA_generation
Max: 274). Also, the average tokens of each question (on all Q-TYPE) is 23 (Min:6, Max: 57).
Table 1 shows the total number of each question type in SPARTQA-AUTO (Check Appendix to see more statistic information about the labels in Tab 8.)
5 Models for Spatial Reasoning over Language
This section describes the model architectures on different Q-TYPES: FR, YN, FB, and CO. All Q-TYPES can be cast into a sequence classification task, and the three transformer-based LMs tested in this paper, BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and XLNet (Yang et al., 2019), can all handle this type of tasks by classifying the representation of [CLS], a special token pretended to each target sequence (see Appendix E). Depending on the Q-TYPE, the input sequence and how we do inference may be different.
FR and YN both have a predefined label set as candidate answers, and their input sequences are both the concatenation of a story and a question. While the answer to a YN question is a single label chosen from Yes, No, and  $DK$ , FR questions can have multiple correct answers. Therefore, we treat each candidate answer to FR as an independent binary classification problem, and take the union as the final answer. As for YN, we choose the label with the highest confidence (Fig 8b).
As the candidate answers to FB and CO are not fixed and depend on each story and its question the input sequences to these Q-TYPES are concatenated with each candidate answer. Since the defined YN and FR model has moderately less accurate results on FB and CO Q-TYPES, we add a LSTM (Hochreiter and Schmidhuber, 1997) layer to improve it. Hence, to find the final answer, we run the model with each candidate answer and then apply an LSTM layer on top of all token representations. Then, we use the last vector of the LSTM outputs for classification (Fig 8a). The final answers are selected based on Eq. (1).
$$
\begin{array}{l} x _ {i} = [ s, c _ {i}, q ] \\ \vec {T _ {i}} = [ \vec {t _ {1} ^ {i}}, \dots , \vec {t _ {m ^ {i}} ^ {i}} ] = L M (x _ {i}) \\ [ \vec {h} _ {1} ^ {i}, \dots , \vec {h} _ {m ^ {i}} ^ {i} ] = \operatorname {L S T M} (\vec {T} _ {i}) \tag {1} \\ \vec {y} _ {i} = \left[ y _ {i} ^ {0}, y _ {i} ^ {1} \right] = \operatorname {S o f t m a x} \left(\vec {h} _ {m i} ^ {i T} W\right)) \\ \text {A n s w e r} = \left\{c _ {i} \mid \underset {j} {\arg \max } \left(y _ {i} ^ {j}\right) = 1 \right\} \\ \end{array}
$$