SlowGuess/ACL_2020_OCR huggingface dataset:
Model	Original	Revised
P	R	F1	P	R	F1
BoE	50.0	32.6	39.4	51.8	35.9	42.4
CNN	72.3	45.5	55.9	79.8	53.5	64.1
CNN, masked	67.2	53.5	59.5	72.5	61.4	66.5
CNN w/ POS/NER	72.2	54.7	62.2	79.7	64.3	71.2
CNN + ELMo	73.8	48.8	58.8	82.1	57.9	67.9
CNN + ELMo, masked	72.3	53.8	61.7	79.8	63.2	70.5
CNN + ELMo, masked w/ POS/NER	69.2	59.0	63.7	76.0	69.1	72.4
CNN + BERT uncased	71.9	51.1	59.7	79.5	60.2	68.5
CNN + BERT uncased, masked	69.0	62.0	65.3	74.9	71.7	73.2
CNN + BERT cased	69.7	54.3	61.0	77.6	64.3	70.4
CNN + BERT cased, masked	71.8	61.1	66.1	78.1	70.8	74.3
LSTM	59.3	47.5	52.7	65.9	56.2	60.6
LSTM, masked	63.4	51.7	57.0	68.7	59.7	63.9
LSTM, masked w/ POS/NER	65.4	56.8	60.8	71.2	66.0	68.5
LSTM + ELMo	61.5	61.3	61.4	68.1	72.2	70.1
LSTM + ELMo, masked	63.9	64.9	64.4	69.3	75.0	72.1
LSTM + ELMo, masked w/ POS/NER	61.7	67.8	64.6	66.1	77.3	71.2
LSTM + BERT uncased	64.7	60.2	62.4	71.6	71.0	71.3
LSTM + BERT uncased, masked	65.3	64.8	65.1	70.4	74.3	72.3
LSTM + BERT cased	66.2	59.8	62.8	73.5	70.8	72.1
LSTM + BERT cased, masked	68.9	61.9	65.2	75.0	71.8	73.4
Bi-LSTM	53.3	57.4	55.3	58.6	67.2	62.6
Bi-LSTM, masked	62.5	63.4	62.9	67.7	73.1	70.3
Bi-LSTM + ELMo	65.0	58.7	61.7	72.6	69.8	71.1
Bi-LSTM + ELMo, masked	63.3	64.8	64.1	68.9	75.2	71.9
Bi-LSTM + ELMo w/ POS/NER	64.8	57.9	61.2	72.1	68.6	70.3
Bi-LSTM + ELMo, masked w/ POS/NER	63.0	65.9	64.4	67.5	75.2	71.2
Bi-LSTM + BERT uncased	65.3	59.9	62.5	71.8	70.2	71.0
Bi-LSTM + BERT uncased, masked	64.9	66.0	65.4	69.6	75.3	72.4
Bi-LSTM + BERT cased	65.2	61.2	63.1	72.1	72.1	72.1
Bi-LSTM + BERT cased, masked	68.3	64.0	66.1	74.1	73.9	74.0
GCN	65.6	50.5	57.1	72.4	59.3	65.2
GCN, masked	68.2	58.0	62.7	74.3	67.4	70.7
GCN, masked w/ POS/NER	68.6	60.2	64.2	74.2	69.3	71.7
GCN + ELMo	66.5	57.6	61.7	73.4	67.7	70.4
GCN + ELMo, masked	68.5	61.3	64.7	74.5	71.0	72.7
GCN + ELMo, masked w/ POS/NER	67.9	64.8	66.3	73.3	74.4	73.9
GCN + BERT uncased	66.3	58.8	62.4	73.1	69.1	71.0
GCN + BERT uncased, masked	68.7	64.0	66.3	74.8	74.1	74.5
GCN + BERT cased	66.5	56.4	61.0	74.4	67.1	70.5
GCN + BERT cased, masked	67.2	64.6	65.9	72.9	74.7	73.8
S-Att.	56.9	58.3	57.6	62.2	67.8	64.9
S-Att., masked	65.0	66.8	65.9	69.3	75.8	72.4
S-Att. + ELMo	64.4	65.0	64.7	71.5	76.8	74.1
S-Att. + ELMo, masked	64.0	69.4	66.6	68.9	79.6	73.8
S-Att. + BERT uncased	60.6	67.6	63.9	66.3	78.7	72.0
S-Att. + BERT uncased, masked	64.0	69.7	66.7	68.9	80.0	74.0
S-Att. + BERT cased	63.5	64.1	63.8	70.4	75.7	73.0
S-Att. + BERT cased, masked	69.2	64.7	66.9	75.1	74.8	75.0
Average	65.6	59.5	62.1	71.8	69.2	70.1
Table 6: Test set performance on TACRED and the revised version for all 49 models we used to select the most challenging instances. We use the same entity masking strategy as Zhang et al. (2017), replacing each entity in the original sentence with a special  $<\mathrm{NER}> - \{\mathrm{SUBJ}, \mathrm{OBJ}\}$  token where  $<\mathrm{NER}>$  is the corresponding NER tag. For models w/ POS/NER we concatenate part-of-speech and named entity tag embeddings to each input token embedding.