SlowGuess/NAACL_2021_OCR huggingface dataset:
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011.
Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158-1172.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Taher Rahgooy, Umar Manzoor, and Parisa Kordjamshidi. 2018. Visually guided spatial relation extraction from text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 788-794.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020. RMM: A recursive mental model for dialogue navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1732-1745, Online. Association for Computational Linguistics.
Priyanka Sen and Amir Saffari. 2020. What do models learn from question answering datasets? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2429-2438.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217-223, Vancouver, Canada. Association for Computational Linguistics.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from
context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316.
Takuma Udagawa, Takato Yamazaki, and Akiko Aizawa. 2020. A linguistic analysis of visually grounded dialogues based on spatial expressions. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 750–765, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. pages 2369-2380.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1753-1762.
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7579-7589.
A Question Templates and statistics Information
Table 7 shows the templates used to create questions in SPARTQA-AUTO. The "<object>" is a variable replaced by objects from the story (using Choose-objects and Describe-objects modules), and the "<relation>" variable can be replaced by the chosen relations between objects (using Find-all-relations module).
The articles and the indefinite pronouns in each template play an essential role in understanding the question's objective. For example, "Are all blue circles near to a triangle?" is different from "Are there any blue circles near to a triangle?", and "Are there any blue circles near to all triangles?". Therefore, we check the uniqueness of the object definition, using "a" or "the" in proper places and randomly place the terms "any" or "all" in the YN questions to generate different questions.
Table 8 shows the percentage of correct labels in train and test sets. In multi-choice Q-TYPES, more than one label can be true.
B Sentences of the Dataset
Table 10 shows some generated sentences in SPARTQA-AUTO with some specific features that challenge models to understand different forms of relation description in spatial language.
C Additional Evaluation Sets
Here we describe three extra evaluation sets provided with this dataset in more detail, including unseen test, consistency, and contrast sets.
C.1 Unseen Evaluation Set
We propose an unseen test set alongside the seen test of SPARTQA-AUTO to check whether a model is using shortcuts in the language surface by describing objects and relations with new vocabularies in the samples. This set has minor modifications that should not affect the performance of a consistent and reliable model. The modifications are randomly applied on a number of generated stories and questions and include changing names of shapes, colors, sizes, and relationships' names (describing relationships using different language expressions). The modification choices are described in Table 9.
C.2 Contrast and Consistency Evaluation
For probing the consistency and semantic sensitivity of models, we provide two extra evaluation test
sets, Consistency and Contrast<sup>9</sup>.
Consistency set is made by changing parts of the question in a way that it still asks about the same information (Hudson and Manning, 2019; Suhr et al., 2019). For instance, for the question, "What is the relation between the blue circle and the big shape? Left," we create a similar question in the form of "What is the relation between the big shape and the blue circle? Right". Answering these questions around a pivot question is possible for human without the need for extra reasoning over the story and based on the main questions' answer. Hence, the evaluation on this set shows that models understand the real underlying semantics rather than overfit on the structure of questions.
Contrast set: This set is made by minor changes in a question that changes the answer (Gardner et al., 2020). As an instance, in the question "Is the blue circle below the black triangle? Yes," we create a contrast question "Is the blue circle below all triangles? No" by changing "the black triangle" to "all triangles". The evaluation on this set shows the robustness of the model and its sensitivity to the semantic changes when there are minor changes in the language surface  $^{10}$ .
D Extra Annotations
Alongside the main SPARTQA-AUTO's stories and questions we provided some extra annotation to help the models to understand the spatial language better.
D.1 Detailed Annotation and Scene-Graphs
Providing in-depth human annotations is quite expensive and time-consuming. In SPARTQA-AUTO, we generated fine-grained scene-graph based on the story. This scene-graph contains blocks' description, their relations, and the objects' attributes alongside their direct relations with each other. The scene-graphs can be used for the models to understand all spatial relations directly mentioned in the textual context. Figure 7 shows an example of this scene-graph. The scene-graph can provide strong supervision for question answering challenges and
${}^{9}$  for some questions,it is not possible to generate a complementary set
Based on the original contrast set paper, consistency and contrast set should be generated manually to control the semantic change. In our case that we are probing the spatial language understanding of models, we must change parts that affect spatial understanding, which can be implemented by some static rules.
Q-Type	Q-Templates	Candidate answer
FR	what is the relation between <object>and <object>?	Left, Right, Below, Above, Touching, Far from, Near to
CO	What is <relation>the <object>?
an <object1>or an <object2>?
Which object is <relation>an <object>?
the <object1>or the <object2>?	Object1, object2, Both, None
YN	Is (the | a) <object1><relation>(the | a) <object2>?
Is there any <object1>s <relation>All <object2>s?	Yes, No, Don’t Know
FB	Which block has an <object>?
Which block doesn’t have an <object>?	Name of blocks, None
Table 7: Questions and answers templates.
Figure 7: Scene-graph
can be used to evaluate models based on their steps of reasoning and decisions.
D.2 SpRL Annotation
We also provided spatial annotations for each sentence and question, based on Spatial Role Labeling (SpRL) annotation scheme (Kordjamshidi et al., 2010)(Fig. 11). This annotation is generated by hand-crafted rules during the main data generation. SpRL is used for recognizing spatial expressions and arguments in a sentence. This annotation is useful for applications that need to detect and reason about spatial expressions and arguments.
E QA Language Models for Spatial Reasoning over Text
Figures 8a and 8b depict the architecture used for further fine-tuning language models on SPARTQA described in section 5.
F bAbI and boolQ Datasets
Figure 9 shows an example of the bAbI dataset (Weston et al., 2015) task 17.
To solve task 17 of bAbI, we implement two SpRL+rule-based and neural network models. The
(a)  $\mathrm{LM}_{QA}$  Architecture for CO and FB Q-TYPES Correct Answer  $y\in$  {candidate answers}
(b)  $\mathrm{LM}_{QA}$  Architecture for FR and YN Q-TYPES
Figure 8:  $\mathrm{LM}_{QA}$  for Spatial Reasoning over Text
Figure 9: An example of bAbI dataset, task 17.
"The pink rectangle is below the red square. The red square is below the blue square."
1. Is the red square below the pink rectangle? No
2. Is the pink rectangle below the blue square? Yes
SpRL+rule-based model first, finds different spa
Q-TYPE	Candidate Answers	train	test
FR
(Multiple Choices)	Left	20.7	17.9
Right	21.4	16.7
Above	26.9	25.4
Below	37.2	42.9
Near to	5.8	2.9
Far from	1.3	0.56
Touching	0.57	0.27
DK	0.52	0.32
FB
(multiple Choices)	A	49.8	49.4
B	50.1	50
C	35.1	62
[]	7.1	90.5
CO
(Single choice)	Object1	25.4	26
Object2	25.3	24.9
Both	44.3	43.9
None	4.9	5.0
YN
(Single choice)	Yes	53.3	50.5
No	18.7	23.6
DK	27.8	25.9
Table 8: The percentage of each correct label in all samples. *The candidate answers for the FB Q-TYPE can be varied, based on its story. **CO can be considered as a multiple choice or single choice question. E.g., in "which object is above the triangle? the blue circle or the black circle?" you can consider two labels with boolean classification on each "blue circle" and "black circle" or consider it as a four labels classification: "blue circle," "black circle," "both of them," and "None of them." *** DK, None, [], all mean none of the actual labels are correct.
tial relation triplets (Landmark, Spatial-indicator, trajectory) for each fact in a story the applies spatial rules over these extracted triplets and report all possible relations between two asked objects. Finally, it checks whether the asked relation existed in the find relation. This model solves task 17 of the bAbI with  $100\%$  accuracy.
To implement the neural network approach, we use huggingface implementation of pre-trained BERT (Devlin et al., 2019). We apply a boolean classifier on the output of “[CLS)” token from the last layer of BERT model for each “Yes” and “No” answers (the same as model used on YN question types.) We use Adamw (Loshchilov and Hutter, 2017) optimizer and  $2e - 6$  learning rate with negative log-likelihood loss objective and train the model on the 10k, 5k, 2k, 1k, 500, and 100 portion of bAbI's training questions. The model yields  $100\%$  accuracy on 10k, and 5k and  $99\%$  accuracy
Type	Original Set	Unseen Set
Shapes	Square, Circle, Triangle	Rectangle, Oval, Diamond
Relations	Left, Right, Above, Below	Left side, Right side, Top, Under
Colors	Yellow, Black, Below	Green, Red, White
Size	Small, Medium, Big	Little, Midsize, Large
Table 9: Modifications on the unseen set
on 2k and 1k training samples.
Figure 10 shows an example of boolQ dataset. To Answering the questions of this dataset, we use the same setting as neural network model on bAbI to further fine-tune BERT on boolQ.
Q: Has the UK been hit by a hurricane?
P: The Great Storm of 1987 was a violent extratropical cyclone which caused casualties in England, France and the Channel Islands ...
A: Yes. [An example event is given.]
Q: Does France have a Prime Minister and a President?
P: ... The extent to which those decisions lie with the Prime Minister or President depends upon ...
A: Yes. [Both are mentioned, so it can be inferred both exist.]
Figure 10: An example of boolQ dataset.
Figure 11: SpRL annotation for an example sentence from SPARTQA.
Examples	Features
Block A is above Block C and B.	Using conjunction to describe relation between more than two blocks.
The small circle is above the yellow square and the big black shape.	Using conjunction to describe relationships be-tween more than two objects.
The yellow square number one is to the right of and above the blue circle.	Using conjunction for more than one relation.
Block B has two medium yellow squares and two blue circles.	Describing a group of objects with the same properties. In the next sentences, they are men-tioned by an assigned number. For example, the blue circle number two.
The blue circle is below the object which is to the right of the big square.	Using nested relations between objects in their description.
A small blue circle is near to the big circle. It is to the left of the medium yellow square.	Using coreferences for an entity described in the previous sentences.
There is a block named A. One small yellow square is touching the bottom edge of this block.	The verb matches the number of the subject.
What is the relation between black object and a big circle?	Using shape, object, and thing, which are a gen-eral description of an object. It could be the “black triangle” or the “black circle” mentioned in the story.
Table 10: Particular features of the dataset