Question: Does the model proposed by the authors achieve significant performance gains in the test sets over the baseline for both Italian and Finnish query translation?
Evidence: Table TABREF14 shows the effectiveness of our model (multi-task transformer) over the baseline transformer BIBREF8 . Our model achieves significant performance gains in the test sets over the baseline for both Italian and Finnish query translation. The overall low MAP for NMT can possibly be improved with larger TC. Moreover, our model validation approach requires access to RC index, and it slows down overall training process. Hence, we could not train our model for a large number of epochs - it may be another cause of the low performance.
We want to show that translation terms generated by our multi-task transformer are roughly equally likely to be seen in the Europarl corpus (TC) or the CLEF corpus (RC). Given a translation term INLINEFORM0 , we compute the ratio of the probability of seeing INLINEFORM1 in TC and RC, INLINEFORM2 . Here, INLINEFORM3 and INLINEFORM4 is calculated similarly. Given a query INLINEFORM5 and its translation INLINEFORM6 provided by model INLINEFORM7 , we calculate the balance of INLINEFORM8 , INLINEFORM9 . If INLINEFORM10 is close to 1, the translation terms are as likely in TC as in RC. Figure FIGREF15 shows the balance values for transformer and our model for a random sample of 20 queries from the validation set of Italian queries, respectively. Figure FIGREF16 shows the balance values for transformer and our model for a random sample of 20 queries from the test set of Italian queries, respectively. It is evident that our model achieves better balance compared to baseline transformer, except for a very few cases.
Given a query INLINEFORM0 , consider INLINEFORM1 as the set of terms from human translation of INLINEFORM2 and INLINEFORM3 as the set of translation terms generated by model INLINEFORM4 . We define INLINEFORM5 and INLINEFORM6 as precision and recall of INLINEFORM7 for model INLINEFORM8 . In Table TABREF19 , we report average precision and recall for both transformer and our model across our train and validation query set over two language pairs. Our model generates precise translation, i.e. it avoids terms that might be useless or even harmful for retrieval. Generally, from our observation, avoided terms are highly likely terms from TC and they are generated because of translation model overfitting. Our model achieves a regularization effect through an auxiliary task. This confirms results from existing multi-tasking literature BIBREF15 .
To explore translation quality, consider pair of sample translations provided by two models. For example, against an Italian query, medaglia oro super vinse medaglia oro super olimpiadi invernali lillehammer, translated term set from our model is {gold, coin, super, free, harmonising, won, winter, olympics}, while transformer output is {olympic, gold, one, coin, super, years, won, parliament, also, two, winter}. Term set from human translation is: {super, gold, medal, won, lillehammer, olypmic, winter, games}. Transformer comes up with terms like parliament, also, two and years that never appears in human translation. We found that these terms are very likely in Europarl and rare in CLEF. Our model also generates terms such as harmonising, free, olympics that not generated by transformer. However, we found that these terms are equally likely in Europarl and CLEF.
Our model achieves significant performance gains in the test sets over the baseline for both Italian and Finnish query translation.
Answer: Yes, it does.