SlowGuess/NAACL_2021_OCR huggingface dataset:
MNLI. Multi-genre Natural Language Inference (Williams et al., 2018). Given a pair of sentences, the task is to predict whether whether the second sentence is an entailment, contradiction, or neutral with respect to the first one.
QQP. Quora Question Pairs (Iyer et al., 2017). The task is to determine whether a pair of questions asked on Quora are semantically equivalent.
QNLI. Question Natural Language Inference. It is a binary classification task constructed from SQuAD (Rajpurkar et al., 2016b). The task is to predict whether a context sentence contains the answer to a question sentence.
SST. Stanford Sentiment Treebank (Socher et al., 2013). This task is binary task to determine if a sentence is positive or negative in sentiment.
STS. Semantic Textual Similarity (Cer et al., 2017). The tasks is to predict how similar two sentences are on a 1-5 scale in terms of semantic meaning.
CoLA. Corpus of Linguistic Acceptability (Warstadt et al., 2018). The task is to determine whether a given sentence is linguistically "acceptable".
MRPC. Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005). The task is to predict whether two sentences are semantically equivalent.
RTE. Recognizing Textual Entailment (Giampaicolo et al., 2007). Given a premise and a hypothesis, the task is to predict whether the premise entails the hypothesis.
Hyperparameter	Small	Base
Number of layers	12	12
Hidden Size	256	768
FFN inner hidden size	1024	3072
Attention heads	4	12
Attention head size	64	64
Embedding Size	128	768
Mask percent	15	15
Learning Rate Decay	Linear	Linear
Warmup steps	10000	10000
Learning Rate	5e-4	2e-4
Adam ε	1e-6	1e-6
Adam β1	0.9	0.9
Adam β2	0.999	0.999
Attention Dropout	0.1	0.1
Dropout	0.1	0.1
Weight Decay	0.01	0.01
Batch Size	128	256
Table 4: Pre-train hyperparameters.
Hyperparameter	Value
Learning Rate	3e-4 for Small, 1e-4 for Base
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.999
Layerwise LR decay	0.8
Learning rate decay	Linear
Warmup fraction	0.1
Attention Dropout	0.1
Dropout	0.1
Weight Decay	0
Batch Size	32
Train Epochs	10 for RTE and STS, 2 for SQuAD, 3 for other tasks
Table 5: Fine-tune hyperparameters.