SlowGuess/NAACL_2022_OCR huggingface dataset:
Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5412-5422, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018a. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. ArXiv preprint, abs/1810.12885.
Wanjun Zhong, Junjie Huang, Qian Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. 2022. Reasoning over hybrid chain for table-and-text open domain qa. ArXiv preprint, abs/2201.05880.
Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2019. Improving question answering by commonsense-based pre-training. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 16-28. Springer.
Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020a. A heterogeneous graph with factual, temporal and logical knowledge for question answering over dynamic contexts. *ArXiv* preprint, abs/2004.12057.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020b. Reasoning over semantic-level graph for fact checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180, Online. Association for Computational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China. Association for Computational Linguistics.
A Pseudo-code of Legitimate Assignments Deduction
B Function Definition
In this part, we present the detailed description and trigger words for each logical constraint functions in Table 8.
C Question Type
In this part, we list common question types in the AR-LSAT datasets and their ratio in Table 6 and
give examples in Table 7. We further introduce how we calculate a score for dominant question type with a group of legitimate assignments.
1) Must be true/false: this question type needs to select answer that must be true in all the assignments. We match all the assignments with the option. If one option accords/conflicts with one assignment, the single matching score will be 1/-1, otherwise the score will be 0. We then calculate the sum of all the matching scores as the final score.
2) Could be true/false: this question type needs to select answer that could be true in one of the legitimate assignments. We match all the assignments with the option. If one option accords/conflicts with one assignment, the single matching score will be  $1/-1$ , otherwise the score will be 0. We then calculate the maximum matching scores as the final score. The Acceptable solution question type also use this method to calculate score.
3) Maximum number of participants in a position: this question type needs to calculate the maximum possible number of participants in a specified position (group). We calculate the maximum number of participants in all the legetimate assignments and calculate the absolute difference with the number in the option as the final score.
4) Find the earliest position of a participant: this question type needs to calculate the earliest possible position of a specific participant. We calculate the index of the earliest position of the participant in all the legitimate assignments and calculate the absolute difference with the number in the option as the final score.
5) Count the number of possible positions that a participant can be assigned in: for this question type, we count all the non-repetitive assignments of the specific participant and calculate the absolute difference with the number in the option as the final score.
D.1 Descriptions
- LSTM (Gers et al., 1999) is a classical RNN-based model. We apply Bi-LSTM with GloVE (Pennington et al., 2014) embedding.
Question Type	Description
Acceptable solution (15.6%)	identify a feasible solution that can satisfy all the rules
Complete list (3.5%)	identify a complete and accurate list of participants under given condition
Could be true/false (26.8%)	select answer that could be true/false under given condition
Must be true/false (26.4%)	select answer that must be true/false under given condition
Negation (14.7%)	questions that contain negation
Substitution (4.3%)	find a new rule that can substitute one of the old rules for the desiring result
Condition for determined solution (3.5%)	identify a new rule so that the feasible solution is determined
Calculation (3%)	calculate possible participants in a group
Earliest/lastest position (1.3%)	identify the earliest/lastest position that a participant can be assigned to
Maximum/minimum members (1.3%)	identify the maximum/minimum number of participants in a specific group
Table 6: The ratio and description of each question type in the test set of the AR-LSAT dataset.
Question Type	Example
Acceptable solution	Which one of the following could be the schedule of the students’ reports?
Complete list	Which one of the following could be a complete and accurate list of the books placed on the bottom shelf?
Could be true/false with condition	If Himalayans are not featured on day 7, which one of the following could be true?
Must be true/false with condition	If Theresa tests G on the second day, then which one of the following must be true?
Negation	P CANNOT be performed at?
Substitution	Which one of the following, if substituted for the condition that Waite’s audition must take place earlier than the two recorded auditions, would have the same effect in determining the order of the auditions?
Condition for unique solution	The assignment of parking spaces to each of the new employees is fully and uniquely determined if which one of the following is true?
Calculation	How many of the students are there who could be the one assigned to 1921?
Earliest/latest position	If Zircon performs in an earlier slot than Yardsign, which one of the following is the earliest slot in which Wellspring could perform?
Maximum/minimum members	What is the minimum number of solos in which Wayne performs a traditional piece?
Table 7: The examples of question types in the AR-LSAT dataset.
- BERT (Devlin et al., 2019) is a transformer-based model pre-trained on BooksCorpus and Wikipedia with two unsupervised learning task: Masked LM and Nest Sentence Prediction.
- XLNet (Yang et al., 2019) is also a transformer-based model, pre-trained on BooksCorpus, Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl with Permutation Language Modeling.
- RoBERTa (Liu et al., 2019) is a transformer-based model with the same model structure as BERT but trained on a larger corpus and on a different training setting.
- ALBERT (Lan et al., 2020) is a most recent transformer-based pre-trained model. ALBERT uses parameter-reduction techniques that support large-scale configurations.
For all the baselines, we employ cross-entropy loss as the loss function and select AdamW as the optimizer for model training/ fine-tuning. These baselines add a simple classification layer on the top of
them and take the last hidden state as the input. For all the Transformer-based models, we employ base model as the backbone.
Type	Function	Arguments	Description	Trigger Words
Relational Functions	Before	participant 1 participant 2	whether participant 1 is in the position before participant 2	before, above, precede, earlier
After	whether participant 1 is in the position after participant 2	after, larger, higher bigger, older
Last	whether participant 1 is in the last position of participant 2	immediately before, last
Next	whether participant 1 is next to participant 2	immediately after, next
Adjacent	whether participant 1 is neighboring to participant 2	neighboring, adjacent
Different	whether participant 1 in the different position with participant 2	different
Same	whether the first participant in the same position with the second participant	same, also
BeforeEqual	whether participant 1 before or equals to the position of participant 2	no later
AfterEqual	whether participant 1 after or equals to the position of participant 2	no earlier
To	participant position	Whether the participant is assigned to the position	to, on, give, in
Compos. Functions	IfThen	function set 1 function set 2	If rules in rule set 1 satisfied, then rules in rule set 2 satisfied	If... then, If ..., ...
IFF	Rules in rule set 1 satisfied if and only if rules in rule set 2 satisfied	if and only if
And	Rules in rule set 1 satisfied and rules in the rule set 2 satisfied	and
Or	Rules in rule set 1 satisfied or rules in rule set 2 satisfied	or
Unless	Rules in rule set 1 satisfied unless rules in rule set 2 satisfied	unless
Neither	Neither rules in rule set 1 satisfied nor rules in rule set 2 satisfied	Neither ... nor ...
Counting Functions	FirstPos	participant number	Whether the participant is in the last (number) positions	one of the last (number)
LastPos	Whether the participant is in the first (number) positions	one of the first (number)
Table 8: Detailed function descriptions and corresponding trigger words