SlowGuess/ACL_2020_OCR huggingface dataset:
ods in Natural Language Processing, pages 1526- 1534. Association for Computational Linguistics.
Mihai Surdeanu, David McClosky, Mason R. Smith, Andrey Gusev, and Christopher D. Manning. 2011. Customizing an information extraction system to a new domain. In Proceedings of the ACL 2011 Workshop on Relational Models of Semantics, RELMS '11, pages 2-10, Stroudsburg, PA, USA. Association for Computational Linguistics.
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018. RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1257-1266, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation Classification via Multi-Level Attention CNNs. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1298-1307, Berlin, Germany. Association for Computational Linguistics.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344. Dublin City University and Association for Computational Linguistics.
Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. Computing Research Repository (CoRR), abs/1508.01006.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205-2215. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45. Association for Computational Linguistics.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 427-434. Association for Computational Linguistics.
A Further Training Details
A.1 Probing Task Training
The probing task results reported in the main part are obtained by fitting a Logistic Regression classifier to the binary and multi-class classification task. We tune the  $l_{2}$  penalty of the classifier with grid-search on the validation set.
A.2 Relation Extraction Training
For vanilla models we use 300-dimensional pretrained GloVe embeddings (Pennington et al., 2014) as input. Variants with ELMo use the contextualized word representations in combination with GloVe embeddings and models with BERT only use the computed representations. For models trained on TACRED we use 30-dimensional positional offset embeddings for head and tail (50-dimensional embeddings for SemEval). Similar for the batch-size we use 50 on TACRED and 30 on SemEval. If not mentioned otherwise, we use the same hyperparameters for models with and without entity masking.
A.2.1 Hyperparameters
CNN For training on TACRED we use the hyperparameters of Zhang et al. (2017). We employ Adagrad as an optimizer, with an initial learning rate of 0.1 and run training for 50 epochs. Starting from the 15th epoch, we gradually decrease the learning rate by a factor of 0.9. For the CNN we use 500 filters of sizes [2, 3, 4, 5] and apply  $l_{2}$  regularization with a coefficient of  $10^{-3}$  to all filter weights. We use tanh as activation and apply dropout on the encoder output with a probability of 0.5. We use the same hyperparameters for variants with ELMo. For variants with BERT, we use an initial learning rate of 0.01 and decrease the learning rate by a factor of 0.9 every time the validation F1 score is plateauing. Also we use 200 filters of sizes [2, 3, 4, 5].
On SemEval, we use the hyperparameters of Nguyen and Grishman (2015). We employ Adadelta with initial learning rate of 1 and run it for 50 epochs. We apply  $l_{2}$  regularization with a coefficient of  $10^{-5}$  to all filter weights. We use embedding and encoder dropout of 0.5, word dropout of 0.04 and 150 filters of sizes [2, 3, 4, 5]. For variants using BERT, we decrease the learning rate by a factor of 0.9 every time the validation F1 score is plateauing.
BiLSTM For training on TACRED we use the hyperparameters of Zhang et al. (2017). We employ Adagrad with an initial learning rate of 0.01, train for 30 epochs and gradually decrease the learning rate by a factor of 0.9, starting from the 15th epoch. We use word dropout of 0.04 and recurrent dropout of 0.5. The BiLSTM consists of two layers of hidden dimension 500 for each direction. For training with ELMo and BERT we decrease the learning rate by a factor of 0.9 every time the validation F1 score is plateauing.
On SemEval we instead use two BiLSTM layers with hidden dimension 300 for each direction, and also use embedding and encoder dropout of 0.5.
GCN On TACRED and SemEval we reuse the hyperparameters of Zhang et al. (2018). We employ SGD as optimizer with an initial learning rate of 0.3, which is reduced by a factor of 0.9 every time the validation F1 score plateaus. We use dropout of 0.5 between all but the last GCN layer, word dropout of 0.04, and embedding and encoder dropout of 0.5. Similar to the authors we use path-centric pruning with  $\mathrm{K} = 1$ . On TACRED we use two 200-dimensional GCN layers and similar two 200-dimensional feedforward layers with ReLU activation, whereas on SemEval we instead use a single 200-dimensional GCN layer.
Self-Attention After hyperparameter tuning we found 8 layers of multi-headed self-attention to perform best. Each layer uses 8 attention heads with attention dropout of 0.1, keys and values are projected to 256 dimensions before computing the similarity and aggregated in a feedforward layer with 512 dimensions. For training we use Adam optimizer with an initial learning rate of  $10^{-4}$ , which is reduced by a factor of 0.9 every time the validation F1 score plateaus. In addition we use word dropout of 0.04, embedding dropout of 0.5, and encoder dropout of 0.5.
B Relation Extraction Results
Table 4 and Table 5 show the relation extraction performances we obtained after training our model variants on the SemEval and TACRED dataset, respectively.
P	R	F1
BoE	53.7	60.8	55.7
CNN	81.8	78.9	80.2
+ ELMo	87.5	81.6	84.4
+ BERT ↓	89.5	83.4	86.3
+ BERT ↑	88.9	83.3	86.0
Bi-LSTM	82.7	77.9	80.1
+ ELMo	87.3	80.6	83.7
+ BERT ↓	88.3	83.2	85.6
+ BERT ↑	87.5	83.0	85.1
GCN	81.9	77.5	79.6
+ ELMo	86.1	82.6	84.2
+ BERT ↓	89.2	82.6	85.7
+ BERT ↑	87.6	81.4	84.3
S-Att.	83.3	77.7	80.2
+ ELMo	87.7	79.9	83.6
+ BERT ↓	89.7	81.9	85.6
+ BERT ↑	88.9	81.5	84.9
Table 4: Relation extraction test set performance on SemEval.  $\uparrow$  and  $\downarrow$  indicate the cased and uncased version of BERT. Due to the small dataset size, we report the mean across 5 randomly initialized runs.
P	R	F1
BoE	50.0	32.6	39.4
CNN	72.3	45.5	55.9
+ ELMo	73.8	48.9	58.8
+ BERT ↓	71.9	51.1	59.7
+ BERT ↑	69.8	54.3	61.0
CNN ×	67.2	53.4	59.5
+ ELMo	72.3	53.8	61.7
+ BERT ↓	69.0	62.0	65.3
+ BERT ↑	71.9	61.1	66.1
Bi-LSTM	53.3	57.4	55.3
+ ELMo	65.1	58.8	61.8
+ BERT ↓	65.3	59.9	62.5
+ BERT ↑	65.2	61.2	63.1
Bi-LSTM ×	62.5	63.4	62.9
+ ELMo	63.3	64.9	64.1
+ BERT ↓	64.9	66.0	65.4
+ BERT ↑	68.3	64.0	66.1
GCN	65.4	51.1	57.4
+ ELMo	66.2	58.5	62.1
+ BERT ↓	66.1	59.9	62.9
+ BERT ↑	66.2	57.4	61.5
GCN ×	68.1	59.8	63.7
+ ELMo	68.5	62.6	65.4
+ BERT ↓	68.1	64.5	66.3
+ BERT ↑	66.6	65.3	65.9
S-Att.	56.9	58.3	57.6
+ ELMo	64.4	65.0	64.7
+ BERT ↓	60.6	67.6	63.9
+ BERT ↑	63.5	64.1	63.8
S-Att. ×	65.0	66.8	65.9
+ ELMo	64.0	69.4	66.6
+ BERT ↓	64.0	69.7	66.7
+ BERT ↑	69.2	64.7	66.9
Table 5: Relation extraction test set performance on TA-CRED.  $\uparrow$  and  $\downarrow$  indicate the cased and uncased version of BERT,  $\otimes$  models with entity masking.