SlowGuess/ACL_2020_OCR huggingface dataset:
Complete results Tables 5 and 6 present the full set of results of our experiments on document and sentence-level multimodal QE on our main test set, the WMT'18 test set. These are a super-set of the results presented in the main paper but include all combinations of multimodality integration and fusion strategies for sentence-level prediction, as well as different numbers of principal components kept for document-level QuEst prediction models.
Additional test set Tables 7 and 8 present the full set of results of our experiments on the WMT'19 Task 2 test set on document and sentence-level multimodal QE, respectively. This was the follow-up edition of the WMT'18 Task 4, where the same training set is used, but a new test set is released.
For document-level, we observe nuanced results with more modest benefits in using visual features, regardless of the integration method or fusion strategy.
For sentence-level, we observe on the one hand quite significant improvements with a gain of almost 8 points in Pearson's  $r$  over BiRNN, our monomodal baseline without pre-trained word embedding. It is interesting to note that almost all
Pearson	MAE	RMSE
BiRNN	0.495	0.531	0.788
+Vis-last conc	0.476	0.550	0.802
+Vis-last-mult	0.481	0.543	0.812
+Vis-annot-mult	0.494	0.531	0.793
+Vis-embed-conc	0.501	0.536	0.780
+Vis-embed-mult	0.481	0.567	0.819
+Vis-embed-mult2	0.491	0.575	0.831
QuEst	0.503	0.547	0.802
+Vis-1	0.497	0.545	0.801
+Vis-2	0.536	0.534	0.790
+Vis-3	0.528	0.538	0.793
+Vis-5	0.520	0.539	0.797
+Vis-10	0.520	0.536	0.796
+Vis-15	0.515	0.540	0.801
Table 5: Document-level results for BiRNN and QuEst++ on the WMT'18 dataset, with and without visual features.
Pearson	MAE	RMSE
BiRNN	0.504	0.539	0.754
+Vis-last-conc	0.483	0.531	0.746
+Vis-last-mult	0.462	0.511	0.733
+Vis-annot-mult	0.460	0.521	0.741
+Vis-embed-conc	0.467	0.541	0.765
+Vis-embed-mult	0.473	0.534	0.753
+Vis-embed-mult2	0.535	0.569	0.792
BERT-BiRNN	0.590	0.455	0.659
+Vis-last-conc	0.360	0.993	1.252
+Vis-last-mult	0.529	0.520	0.744
+Vis-annot-mult	0.602	0.454	0.654
+Vis-embed-conc	0.576	0.474	0.694
+Vis-embed-mult	0.598	0.486	0.686
+Vis-embed-mult2	0.570	0.573	0.770
Table 6: Sentence-level results for BiRNN and BERT-BiRNN on the WMT'18 Task 4 dataset, with and without visual features.
multimodal variants achieve better performance compared to the monomodal BiRNN baseline, with a peak when the visual features are fused with the word embedding representations by elementwise multiplication. On the other hand, we do not observe any gain in using visual features on the WMT'19 test set compared to our monomodal baseline with pre-trained word-embedding (BERT-BiRNN). Here that the BERT-BiRNN baseline model already performs very well. According to the task organisers, the mean MQM value on the WMT'19 test set is higher than on the WMT'18 test set, but actually closer to the training data (Fonseca
et al., 2019). We therefore hypothesise here that the highly dimensional and contextualised word-level representations from BERT are already enough and do not benefit from the extra information provided by the visual features.
Pearson	MAE	RMSE
BiRNN	0.367	0.335	0.413
+Vis-last-conc	0.332	0.416	0.503
+Vis-last-mult	0.261	0.329	0.421
+Vis-annot-mult	0.332	0.276	0.353
+Vis-embed-conc	0.370	0.364	0.439
+Vis-embed-mult	0.335	0.313	0.398
+Vis-embed-mult2	0.344	0.285	0.361
Table 7: Document-level results for BiRNN on the WMT'19 Task 2 test set, with and without visual features.
Metrics	Pearson	MAE	RMSE
BiRNN	0.485	0.616	0.922
+Vis-last-conc	0.492	0.602	0.908
+Vis-last-mult	0.520	0.584	0.895
+Vis-annot-mult	0.508	0.591	0.901
+Vis-embed-conc	0.470	0.614	0.927
+Vis-embed-mult	0.474	0.613	0.927
+Vis-embed-mult2	0.563	0.609	0.944
BERT-BiRNN	0.652	0.556	0.842
+Vis-last-mult	0.605	0.568	0.854
+Vis-annot-mult	0.596	0.565	0.845
+Vis-embed-conc	0.594	0.571	0.853
+Vis-embed-mult	0.596	0.560	0.827
+Vis-embed-mult2	0.590	0.581	0.853
Table 8: Sentence-level results for BiRNN and BERT-BiRNN on the WMT'19 Task 2 test dataset, with and without visual features.