SlowGuess/ACL_2020_OCR huggingface dataset:
Context-Only Training. During the first stage of our training process, we train BERTRAM with a maximum sequence length of 96 and a batch size of 48 contexts for  $\mathrm{BERT}_{\mathrm{base}}$  and 24 contexts for RoBERTalarge. These parameters are chosen such that a batch fits on a single Nvidia GeForce GTX 1080Ti. Each context in a batch is mapped to a word  $w$  from the set of training words, and each batch contains at least 4 and at most 32 contexts per word. For  $\mathrm{BERT}_{\mathrm{base}}$  and RoBERTalarge, we pretrain the context part for 5 and 3 epochs, respectively. We use a maximum learning rate of  $5 \cdot 10^{-5}$  and perform linear warmup for the first  $10\%$  of training examples, after which the learning rate is linearly decayed.
Form-Only Training. In the second stage of our training process, we use the same parameters as Schick and Schütze (2020), as our form-only model is the very same as theirs. That is, we use a learning rate of 0.01, a batch size of 64 words and we apply  $n$ -gram dropout with a probability of  $10\%$ . We pretrain the form-only part for 20 epochs.
Combined Training. For the final stage, we use the same training configuration as for context-only training, but we keep  $n$ -gram dropout from the form-only stage. We perform combined training for 3 epochs. For ADD, when using RoBERTa as an underlying language model, we do not just prepad the input with the surface-form embedding followed by a colon, but additionally wrap the surface-form embedding in double quotes. That is, we prepad the input with  $e^{\nu}$ ,  $v_{(w,C)}^{\mathrm{form}}$ ,  $e^{\nu}$ ,  $e$ . We found this to perform slightly better in preliminary experiments with some toy examples.
WNLaMPro In order to ensure comparability with results of Schick and Schütze (2020), we use only WWC to obtain contexts for WNLaMPro keywords.
Rarified Datasets To obtain rarified instances of MNLI, AG's News and DBPedia, we train  $\mathrm{BERT}_{\mathrm{base}}$  and  $\mathrm{RoBERTa}_{\mathrm{large}}$  on each task's training set for 3 epochs. We use a batch size of 32, a maximum sequence length of 128 and a weight decay factor of 0.01. For BERT, we perform linear warmup for the first  $10\%$  of training examples and use a maximum learning rate of  $5 \cdot 10^{-5}$ . After reaching its peak value, the learning rate is linearly decayed. For RoBERTa, we found training to be unstable with these parameters, so we chose a lower learning rate of  $1 \cdot 10^{-5}$  and performed linear warmup for the first 10,000 training steps.
To obtain results for our baselines on the rarified datasets, we use the original Mimick implementation of Pinter et al. (2017), the A La Carte implementation of Khodak et al. (2018) and the Attentive Mimicking implementation of Schick and Schütze (2019a) with their default hyperparameter settings. As A La Carte can only be used for words with at least one context, we keep the original BERT embeddings whenever no such context is available.
While using BERTRAM allows us to completely remove the original BERT embeddings for all rare words and still obtain improvements in accuracy on all three rarified downstream tasks, the same is not true for RoBERTa, where removing the original sequence of subword token embeddings for a given rare word (i.e., not using the SLASH variant) hurts performance with accuracy dropping by 5.6, 7.4 and 2.1 points for MNLI, AG's News and DBPedia, respectively. We believe this to be due to the vast amount of additional contexts for rare words in RoBERTa's training set that are not available to BERTRAM.