SlowGuess/ACL_2020_OCR huggingface dataset:
We perform inference of the CopyAttention with BERT model on CaRB test set with beam sizes of 1, 3, 5, 7, and 11. We observe in Figure 5 that AUC increases with increasing beam size. A system can surge its AUC by adding several low confidence tuples to its predicted set of tuples. This adds low precision - high recall points to the Precision-Recall curve of the system leading to higher AUC.
On the other hand, Last F1 experiences a drop at very high beam sizes, thereby capturing the decline in performance. Optimal F1 saturates at high beam sizes since its calculation ignores the extractions
Figure 5: Measuring performance of CopyAttention with BERT model upon changing the beam size
below the optimal confidence threshold.
This analysis also shows the importance of using Last F1 as a metric for measuring the performance of OpenIE systems.
C Evaluation on other datasets
We use sentences from other benchmarks with the CaRB evaluation policy and we find similar improvements, as shown in Table 9. IMOJIE consistently outperforms our strongest baseline, CopyAttention with BERT, over different test sets. This confirms that IMOJIE is domain agnostic.
D Visualizing Attention
Attention has been used in a wide variety of settings to help the model learn to focus on important things (Bahdanau et al., 2015; Xu et al., 2015; Lu et al., 2019). However, the IMOJIE model is able to use attention to understand which words have already been generated, to focus on remaining words. In order to understand how the model achieves this, we visualize the learnt attention weights. There are two attention weights of importance, the learnt attention inside the BERT encoder and the attention between the decoder and encoder. We use BertViz (Vig, 2019) to visualize the attention inside BERT.
We consider the following sentence as the running example - "he served as the first prime minister of australia and became a founding justice of the high court of australia". We visualize the attention after producing the first extraction - "he; served; as the first prime minister of australia". Intuitively, we understand that the model must focus on the words "founding" and "justice" in order to generate the next extraction - "he; became; a founding justice of the high court of australia". In Figure 8 and Figure 9 (where the left-hand column contains the words
5882
Model	Dataset
Wire57	Penn	Web
CopyAttention + BERT	45.60, 27.70, 39.70	18.20, 7.9, 12.40	30.10, 18.00, 14.60
IMoJIE	46.20, 26.60, 46.20	20.20, 8.70, 15.50	30.40, 15.50, 26.40
Table 9: Evaluation on other datasets with the CaRB evaluation strategy
which are used to attend while right-hand column contains the words which are attended over), we see that the words "prime" and "minister" of the original sentence have high attention over the same words in the first extraction. But the attention for "founding" and "justice" are limited to the original sentence.
Based on these patterns, the decoder is able to give a high attention to the words "founding" and "justice" (as shown in Figure 10), in-order to successfully generate the second extraction "he; became; a founding justice of the high court of australia".
Figure 6: BERT attention for the word 'founding'
Figure 7: BERT attention for the word 'justice'
Figure 8: BERT attention for the word 'prime'
5884
Figure 9: BERT attention for the word 'minister'
5885
Figure 10: Attention weights for the decoder
5886