SlowGuess/NAACL_2021_OCR huggingface dataset:
ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, Stanford University.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3410-3416, Hong Kong, China. Association for Computational Linguistics.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208-4219, Brussels, Belgium. Association for Computational Linguistics.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, k—oray kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, volume 29, pages 3630-3638. Curran Associates, Inc.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019b. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020-4026, Florence, Italy. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
A Identifying Superficial Cues
We identify tokens predictive of the correct answer using productivity, as defined by Niven and Kao (2019). Let  $\mathbb{T}_j^{(i)}$  be the set of tokens in the alternatives for data point  $i$  with label  $j$ . The applicability  $\alpha_{k}$  of a token  $k$  counts how often this token occurs in an alternative with one label, but not the other:
$$
\alpha_ {k} = \sum_ {i = 1} ^ {n} \mathbb {1} \left[ \exists j, k \in \mathbb {T} _ {j} ^ {(i)} \wedge k \notin \mathbb {T} _ {\neg j} ^ {(i)} \right]
$$
The productivity  $\pi_{k}$  of a token is the proportion of applicable instances for which it predicts the correct answer:
$$
\pi_ {k} = \frac {\sum_ {i = 1} ^ {n} \mathbb {1} \left[ \exists j , k \in \mathbb {T} _ {j} ^ {(i)} \wedge k \notin \mathbb {T} _ {\neg j} ^ {(i)} \wedge y _ {i} = j \right]}{\alpha_ {k}}
$$
The most productive tokens in Cs-Exshown in Table 2.
We use English datasets, namely COPA  $^{3}$  (Roemmle et al., 2011), Balanced COPA  $^{4}$  (Kavumba et al., 2019), and Commonsense Explanation  $^{5}$  (Wang et al., 2019b) for all our experiments.
B.1 Balanced COPA
The Balanced Choice of Plausible Alternatives (Kavumba et al., 2019, Balanced COPA) counters superficial cues in the answer choices by extending the training set of Choice of Plausible Alternatives (Roemmele et al., 2011, COPA) with twin questions for each of the original COPA instances. Examples of twin questions are shown below:
Original instance
Premise: My body cast a shadow over the grass. What was the CAUSE of this?
a) The sun was rising. (Correct)
b) The grass was cut.
Mirrored instance:
<sup>3</sup>https://people.ict.usc.edu/~gordon/downloads/COPA-resources.tgz
<sup>4</sup>https://balanced-copa.github.io/
5https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation
Premise: The garden looked well-groomed.. What was the CAUSE of this?
a) The sun was rising.
b) The grass was cut. (Correct)
Premise: The woman tolerated her friend's difficult behavior. What was the CAUSE of this?
a) The woman knew her friend was going through a hard time. (Correct)
b) The woman felt that her friend took advantage of her kindness.
Premise: The woman did not tolerate her friend's difficult behavior anymore.. What was the CAUSE of this?
a) The woman knew her friend was going through a hard time.
b) The woman felt that her friend took advantage of her kindness. (Correct)
B.2 Commonsense Explanation
Commonsense Explanation (Cs-Ex) (Wang et al., 2019b) is a multiple-choice benchmark that consists of three subtasks. Here we focus on a commonsense explanation task. Given a false statement, Cs-Ex requires a model to pick the reason why a false statement does not make sense. For example:
FalseStatement: He drinks apple.
a) Apple juice are very tasty and milk too
b) Apple can not be drunk (correct)
c) Apple cannot eat a human
In our experiments, we use a state-of-the-art recent pre-trained language model, namely RoBERTa (Liu et al., 2019), an optimized variant of BERT (Devlin et al., 2019). We use RoBERTa-base and RoBERTa-large with 110M and 355M parameters respectively from the publicly available Huggingface source code (Wolf et al., 2019). We run all our experiments on a single NVIDIA Tesla V100 GPU with 16GB memory.
We use Adam (Kingma and Ba, 2015) with a warm-up proportion of 0.06 and a weight decay of 0.01. We randomly split the training data into
Dataset	Word	Train	Dev
Prod.	Cov.	Prod.	Cov.
Cs-Ex	not	72	47	59	40
to	35	37	33	43
Table 2: Productivity (Prod.) and Coverage (Cov.) of the top 2 most productive tokens in each dataset. We have highlighted the most productive results. In Commonsense-Explanation, if one always picks an option with 'not' then one can achieve  $59\%$  with a coverage of  $40\%$  data points on the dev set.
training data and validation data with a ratio of 9:1. We train the models for a maximum of 10 epochs with early stopping based on the validation loss.
C.1 COPA Baselines
We use grid search for hyperparameter from learning rates  $\{1\mathrm{e} - 5,8\mathrm{e} - 6,6\mathrm{e} - 6,4\mathrm{e} - 6,2\mathrm{e} - 6,1\mathrm{e} - 6\}$  , batch sizes  $\{4,8,16,32,64\}$  , gradient accumulation  $\{1,$ $2,4,8\}$  , Adam  $\beta_{2}$ $\{0.98,0.99\}$  , and with gradient norm clipping of 1 and no gradient norm clipping. We pick the best performing hyperparameters on the validation set.
C.2 Commonsense Explanation (Cs-Ex) Baselines
We test learning rates 1e-5, 8e-6, 6e-6, 4e-6, 2e-6 and 1e-6, Adam  $\beta_{2}$  0.99, and with gradient norm clipping of 1. For RoBERTa-base, we use batch sizes of 64 with gradient accumulation 1, and for RoBERTa-large, we use a batch size of 32 with gradient accumulation 2. We pick the best performing hyperparameters on the validation set.
C.3 Adversarial Trained Baseline
We follow the setup defined by Belinkov et al. (2019). Specifically we optimize the objective function:
$$
L = L _ {\text {s c o r e r}} + \lambda_ {\text {L o s s}} L _ {\text {A d v}}
$$
$$
L _ {\text {A d v}} = L \left(c _ {\text {c h o i c e}} \left(\lambda_ {\text {E n c}} \operatorname {G R L} _ {\lambda} \left(g _ {C} (C)\right), y\right)\right)
$$
Where  $L_{\mathrm{scorer}}$  is the loss of the multiple-choice scorer (or head),  $GRL_{\lambda}$  is the gradient reversal layer (Ganin and Lempitsky, 2015),  $\lambda_{\mathrm{Loss}}$  is the importance of the adversarial loss  $(L_{\mathrm{Adv}})$ ,  $\lambda_{\mathrm{Enc}}$  is the scaling factor that multiplies the gradients after reversing them,  $c_{\mathrm{choice}}$  maps the answer choice representation  $C$  to an output  $y$ . The goal is to obtain a representation  $g_{C}(C)$  so that it is maximally informative for multiple-choice answering while simultaneously minimizes the ability of  $c_{\mathrm{choice}}$
to accurately predict the correct choice (refer to Belinkov et al. (2019) for further details). We use grid search to tune hyperparameters  $\lambda_{\mathrm{Enc}}$  {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} and  $\lambda_{\mathrm{Loss}}$  {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} and report the best performing results on the development set.
C.4 Meta-Training
In the inner-loop, we pick the maximum batch size that fits in GPU memory. For all the experiments, we use vanilla Stochastic Gradient Descent for the inner-loop with learning rate  $\alpha 0.01$  (it worked well in the first run therefore we do not modify it for the rest of the experiments), and Adam for the outer-loop with learning rate  $\beta$  1e-5 (based on the best learning rate for the RoBERTa baseline).