SlowGuess/NAACL_2021_OCR huggingface dataset:
where  $s$  is the story,  $c_{i}$  is the candidate answer,  $q$  is the question, [ ] indicates the concatenation of the listed vectors, and  $m_{i}$  is tokens' number in  $x_{i}$ . The parameter vector,  $W$ , is shared for all candidates.
5.1 Training and Inference
We train the models based on the summation of the cross-entropy losses of all binary classifiers in the architecture. For FR and YN Q-TYPES, there are multiple classifiers, while there is only one classifier used for CO and FB Q-TYPES.
We remove inconsistent answers in post-processing for FR and YN Q-TYPES during inference phase. For instance on FR, left and right relations between two objects cannot be valid at the same time. For YN, as there is only one valid answer amongst the three candidates, we select the candidate with the maximal predicted probability of being the true answer.
As fine-tuning LMs has become a common baseline approach to knowledge transfer from a source dataset to a target task, including but not limited to Phang et al. (2018); Zhou et al. (2020); He et al. (2020b), we study the capability of spatial reasoning of modern LMs, specifically BERT, ALBERT, and XLNet, after fine-tuning them on SPARTQA-AUTO. This fine-tuning process is also known as further pretraining, to distinguish with the fine-tuning process on one's target task. It is an open problem to find out better transfer learning techniques than simple further pretraining, as suggested in He et al. (2020a); Khashabi et al. (2020), which is beyond the scope of this work. All experiments use the models proposed in Sec. 5. We use AdamW (Loshchilov and Hutter, 2017) with  $2 \times 10^{-6}$  learning rate and Focal Loss (Lin et al., 2017) with  $\gamma = 2$  for training all the models. $^{6}$
6.1 Further pretraining on SPARTQA-AUTO improves spatial reasoning
Table 2 shows performance on SPARTQA-HUMAN in a low-resource setting, where  $0.6\mathrm{k}$  QA pairs from SPARTQA-HUMAN are used for fine-tuning these LMs and  $0.5\mathrm{k}$  for testing (see Table 1 for information on this split). During our annotation, we found that the description of "near to" and "far
$^{6}$ All codes are available at https://github.com/HLR/SpartQA-baselines
Note this low-resource setting can also be viewed as a spatial reasoning probe to these LMs (Tenney et al., 2019).
#	Model	FB	FR	CO	YN	Avg
1	Majority	28.84	24.52	40.18	53.60	36.64
2	BERT	16.34	20	26.16	45.36	30.17
3	BERT(Stories only; MLM)	21.15	16.19	27.1	51.54	32.90
4	BERT(SPARTQA-AUTO; MLM)	19.23	29.54	32.71	47.42	34.88
5	BERT(SPARTQA-AUTO)	62.5	46.66	32.71	47.42	47.25
6	Human	91.66	95.23	91.66	90.69	92.31
from” varies largely between annotators. Therefore, we ignore these two relations from FR Q-TYPE in our evaluations.
In Table 2, System 5, BERT (SPARTQA-AUTO), is the proposed method of further pretraining BERT on SPARTQA-AUTO. We can see that System 2, the original BERT, performs consistently lower than System 5, indicating that having SPARTQA-AUTO as a further pretraining task improves BERT's spatial understanding.
Table 2: Further pretraining BERT on SPARTQA-AUTO improves accuracies on SPARTQA-HUMAN. All systems are fine-tuned on the training data of SPARTQA-HUMAN, but Systems 3-5 are also further pretrained in different ways. System 3: further pretrained on the stories from SPARTQA-AUTO as a masked language model (MLM) task. System 4: further pretrained on both stories and QA annotations as MLM. System 5: the proposed model that is further pretrained on SPARTQA-AUTO as a QA task. Avg: The micro-average on all four Q-TYPES.
Model	F1
Majority	35
BERT	50
BERT (Stories only; MLM)	53
BERT (SPARTQA-AUTO; MLM)	48
BERT (SPARTQA-AUTO)	48
Table 3: Switching from accuracy in Table 2 to  $F_{1}$  shows that the models are all performing better than the majority baseline on YN Q-TYPE.
In addition, we implement another two baselines. System 3, BERT (Stories only; MLM): further pretraining BERT only on the stories of SPARTQA-AUTO as a masked language model (MLM) task; System 4, BERT (SPARTQA-AUTO; MLM): we convert the QA pairs in SPARTQA-AUTO into textual statements and further pretrain BERT on the text as an MLM (see Fig. 5 for an example conversion).
To convert each question and its answer into a sentence, we utilize static templates for each question type which removes the question words and rearranges other parts into a sentence.
We can see that System 3 slightly improves over System 2, an observation consistent with many prior works that seeing more text generally helps an LM (e.g., Gururangan et al. (2020)). The signif
A big circle is above a triangle. A blue square is below the triangle.
What is the relation between the circle and the blue object?
Answer: Above
A big circle is above a triangle. A blue square is below the triangle. The circle is [MASK] the blue object.
Figure 5: Convert a triplet of (paragraph, question, answer) into a single piece of text for the MLM task.
icant gap between System 3 and the proposed System 5 indicates that supervision signals come more from our annotations in SPARTQA-AUTO rather than from seeing more unannotated text. System 4 is another way to make use of the annotations in SPARTQA-AUTO, but it is shown to be not as effective as further pretraining BERT on SPARTQA-AUTO as a QA task.
While the proposed System 5 overall performs better than the other three baseline systems, one exception is its accuracy on YN, which is lower than that of System 3. Since all systems' YN accuracies are also lower than the majority baseline $^{8}$ , we hypothesize that this is due to imbalanced data. To verify it, we compute the  $F_{1}$  score for YN Q-TYPE in Table 3, where we see all systems effectively achieve better scores than the majority baseline. However, further pretraining BERT on SPARTQA-AUTO still does not beat other baseline systems, which implies that straightforward pretraining is not necessarily helpful in capturing the complex reasoning phenomena required by YN questions.
The human performance is evaluated on 100 ran
which predicts the label that is most common in each set of SPARTQA
#	Models	FB	FR	CO	YN
Seen	Unseen	Human*	Seen	Unseen	Human*	Seen	Unseen	Human*	Seen	Unseen	Human*
1	Majority	48.70	48.70	28.84	40.81	40.81	24.52	20.59	20.38	40.18	49.94	49.91	53.60
2	BERT	87.13	69.38	62.5	85.68	73.71	46.66	71.44	61.09	32.71	78.29	76.81	47.42
3	ALBERT	97.66	83.53	56.73	91.61	83.70	44.76	95.20	84.55	49.53	79.38	75.05	41.75
4	XLNet	98.00	84.85	73.07	94.60	91.63	57.14	97.11	90.88	50.46	79.91	78.54	39.69
5	Human	85	91.66	90	95.23	94.44	91.66	90	90.69
Table 4: Spatial reasoning is challenging. We further pretrain three transformer-based LMs, BERT, ALBERT, and XLNet, on SPARTQA-AUTO, and test their accuracy in three ways: Seen and Unseen are both from SPARTQA-AUTO, where Unseen has applied minor modifications to its vocabulary; to get those Human columns, all models are fine-tuned on SPARTQA-HUMAN's training data. Human performance on Seen and Unseen is the same since the changes applied to Unseen does not affect human reasoning.
dom questions from each SPARTQA-AUTO and SPARTQA-HUMAN test set. The respondents are graduate students that were trained by some examples of the dataset before answering the final questions. We can see from Table 2 that all systems' performances fall behind human performance by a large margin. We expand on the difficulty of SPARTQA in the next subsection.
6.2 SPARTQA is challenging
In addition to BERT, we continue to test another two LMs, ALBERT and XLNet (Table 5). We further pretrain these LMs on SPARTQA-AUTO, and test them on SPARTQA-HUMAN (the numbers of BERT are copied from Table 2) and two held-out test sets of SPARTQA-AUTO, Seen and Unseen. Note that when a system is tested against SPARTQA-HUMAN, it is fine-tuned on SPARTQA-HUMAN's training data following its further pretraining on SPARTQA-AUTO. We use the unseen set to test to what extent the baseline models use shortcuts in the language surface. This set applies minor modifications randomly on a number of stories and questions to change the names of shapes, colors, sizes, and relationships in the vocabulary of the stories, which do not influence the reasoning steps (more details in Appendix C.1).
All models perform worst in YN across all Q-TYPES, which suggests that YN presents a more complex phenomena, probably due to additional quantifiers in the questions. XLNet performs the best on all Q-TYPES except its accuracy on SPARTQA-HUMAN's YN section. However, the drops in Unseen and human suggest overfitting on the training vocabulary. The low accuracies on human test set from all models show that solving this benchmark is still a challenging problem and requires more sophisticated methods like considering spatial roles and relations extraction (Kordjamshidi
et al., 2010; Dan et al., 2020; Rahgooy et al., 2018) to understand stories and questions better.
To evaluate the reliability of the models, we also provide two extra consistency and contrast test sets. Consistency set is made by changing a part of the question in a way that seeks for the same information (Hudson and Manning, 2019; Suhr et al., 2019). Given a pivot question and answer of a specific consistency set, answering other questions in the set does not need extra reasoning over the story.
Contrast set is made by minimal modification in a question to change its answer (Gardner et al., 2020). For contrast sets, there is a need to go back to the story to find the new answer for the question's minor variations (see Appendix C.2 for examples.) The consistency and contrast sets are evaluated only on the correctly predicted questions to check if the actual understanding and reasoning occurs. This ensures the reliability of the models.
Table 5 shows the result of this evaluation on four Q-TYPES of SPARTQA-AUTO, where we can see, for another time, that the high scores on the Seen test set are likely due to overfitting on training data rather than correct detection of spatial terms and reasoning over them.
6.3 Extrinsic evaluation
In this subsection, we take BERT as an example to show, once pretrained on SPARTQA-AUTO, BERT can achieve better performance on two extrinsic evaluation datasets, namely bAbI and boolQ.
We draw the learning curve on bAbI, using the original BERT as a baseline and BERT further pretrained on SPARTQA-AUTO (Fig. 6). Although both systems achieve perfect accuracy given large enough training data (i.e., 5k and 10k), BERT (SPARTQA-AUTO) is showing better scores given less training data. Specifically, to achieve an accuracy of  $99\%$ , BERT (SPARTQA-AUTO) requires
Models	FB	FR	CO	YN
Consistency	Consistency	Contrast	Consistency	Contrast	Consistency	Contrast	Consistency
BERT	69.44	76.13	42.47	16.99	15.58	48.07	71.41
AlBERT	84.77	82.42	41.69	58.42	62.51	48.78	69.19
XLNet	85.2	88.56	50	71.10	72.31	51.08	69.18
Figure 6: Learning curve of BERT and BERT further pretrained on SPARTQA-AUTO on bAbI.
Table 5: Evaluation of consistency and semantic sensitivity of models in Table 4. All the results are on the correctly predicted questions of Seen test set of SPARTQA-AUTO.
Model	Accuracy
Majority baseline	62.2
Recurrent model (ReM)	62.2
ReM fine-tuned on SQuAD	69.8
ReM fine-tuned on QNLI	71.4
ReM fine-tuned on NQ	72.8
BERT (our setup)	71.9
BERT (SPARTQA-AUTO)	74.2
Table 6: System performances on the dev set of boolQ (since the test set is not available to us). Top: numbers reported in Clark et al., 2019). Bottom: numbers from our experiments. BERT (SPARTQA-AUTO): further pretraining BERT on SPARTQA-AUTO as a QA task.
1k training examples, while BERT requires twice as much. We also notice that BERT (SPARTQA-AUTO) converges faster in our experiments.
As another evaluation dataset, we chose boolQ for two reasons. First, we needed a QA dataset with Yes/No questions. To our knowledge boolQ is the only available one used in the recent work. Second, indeed, SPARTQA and boolQ are from different domains, however, boolQ needs multi-step reasoning in which we wanted to see if SPARTQA helps.
Table 6 shows that further pretraining BERT on SPARTQA-AUTO yields a better result than the original BERT and those reported numbers in Clark et al. (2019), which also tested on various distant supervision signals such as SQuAD (Rajpurkar et al., 2016), Google's Natural Question dataset NQ (Kwiatkowski et al., 2019), and QNLI from
GLUE (Wang et al., 2018).
We observe that many of the boolQ examples answered correctly by the BERT further pretrained on SPARTQA-AUTO require multi-step reasoning. Our hypothesis is that since solving SPARTQA-AUTO questions needs multi-step reasoning, finetuning BERT on SPARTQA-AUTO generally improves this capability of the base model.
Spatial reasoning is an important problem in natural language understanding. We propose the first human-created QA benchmark on spatial reasoning, and experiments show that state-of-the-art pretrained language models (LM) do not have the capability to solve this task given limited training data, while humans can solve those spatial reasoning questions reliably. To improve LMs' capability on this task, we propose to use hand-crafted grammar and spatial reasoning rules to automatically generate a large corpus of spatial descriptions and corresponding question-answer annotations; further pretraining LMs on this distant supervision dataset significantly enhances their spatial language understanding and reasoning. We also show that a spatially-improved LM can have better results on two extrinsic datasets (bAbI and boolQ).
This project is supported by National Science Foundation (NSF) CAREER award #2028626 and (partially) supported by the Office of Naval Research grant #N00014-20-1-2005. We thank the reviewers for their helpful comments to improve this paper and Timothy Moran for his help in the human data generation.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-
and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. TOUCHDOWN: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12538-12547.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936.
Douglas H Clements and Michael T Battista. 1992. Geometry and spatial reasoning. Handbook of research on mathematics teaching and learning, pages 420-464.
Soham Dan, Parisa Kordjamshidi, Julia Bonn, Archna Bhatia, Zheng Cai, Martha Palmer, and Dan Roth. 2020. From spatial relations to spatial configurations. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5855-5864, Marseille, France. European Language Resources Association.
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925-5932.
Surabhi Datta and Kirk Roberts. 2020. A hybrid deep learning approach for spatial trigger extraction from radiology reports. In Proceedings of the Third International Workshop on Spatial Language Understanding, pages 50-55, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Kaustubh Dhole and Christopher D. Manning. 2020. Syn-QG: Syntactic and shallow semantic rules for question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 752-765.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342-1352.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020.
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019. Question Answering is a Format; when is it useful? ArXiv, abs/1909.11291.
Mehdi Ghanimifard and Simon Dobnik. 2017. Learning to compose spatial relations with grounded neural language models. In IWCS 2017-12th International Conference on Computational Semantics-Long papers.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Hangfeng He, Qiang Ning, and Dan Roth. 2020a. QuASE: Question-answer driven sentence encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8743-8758, Online. Association for Computational Linguistics.
Hangfeng He, Mingyuan Zhang, Qiang Ning, and Dan Roth. 2020b. Foreshadowing the benefits of incidental supervision. arXiv preprint arXiv:2006.05500.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-Answer Driven Semantic Role Labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643-653.
Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA LANGUAGE TECHNOLOGIES INST.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Zixian Huang, Yulin Shen, Xiao Li, Yu'ang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, and Yuzhong Qu. 2019. GeoSQA: A benchmark for scenario-based question answering in the geography domain at high school level. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5866-5871.
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. TempQuestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, pages 1057-1062.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896-1907.
Hyounghun Kim, Abhaysinh Zala, Graham Burri, Hao Tan, and Mohit Bansal. 2020. *ArraMon: A joint navigation-assembly instruction interpretation task in dynamic environments*. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3910–3927, Online. Association for Computational Linguistics.
Parisa Kordjamshidi, Marie-Francine Moens, and Martijn van Otterlo. 2010. Spatial Role Labeling: Task definition and annotation scheme. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), pages 413-420. European Language Resources Association (ELRA).
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural Questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 889-898.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Christian Landsiedel, Verena Rieser, Matthew Walter, and Dirk Wollherr. 2017. A review of spatial reasoning and interaction for real-world robotics. Advanced Robotics, 31(5):222-242.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-Shot relation extraction via reading comprehension. In CONLL, pages 333-342.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 560-568.