Question: Which baseline performs best according to the results?
Evidence: Secondly, we fine-tune multilingual Bert (M-Bert) on the task BIBREF5 which has been pretrained jointly in 104 languages and has established itself as a state of the art for various multilingual tasks BIBREF18, BIBREF19. Within the field of stance detection, Bert can outperform both feature-based and other neural approaches in a monolingual English setting BIBREF10.
In the context of Bert we interpret the x-stance task as sequence pair classification inspired by natural language inference tasks BIBREF20. We follow the procedure outlined by BIBREF5 for such tasks. We designate the question as segment A and the comment as segment B. The two segments are separated with the special token [SEP], and the special token [CLS] is prepended to the sequence. The final hidden state corresponding to [CLS] is then classified by a linear layer.
We fine-tune the full model with a cross-entropy loss, using the AllenNLP library BIBREF21 as a basis for our implementation.
We upsampled the `favor' class so that the two classes are balanced when summing over all questions and topics. A maximum sequence length of 512 subwords and a batch size of 16 was chosen for all training runs. We then performed a grid search within the following range of hyperparameters based on the validation accuracy:
Learning rate: 5e-5, 3e-5, 2e-5
Number of epochs: 3, 4
The grid search was repeated independently for every variant that we tested. Furthermore, the standard recommendations for fine-tuning Bert were used: Adam with ${\beta }_1=0.9$ and ${\beta }_2=0.999$; an L2 weight decay of $0.01$; a learning rate warmup over the first 10% of the steps; and a linear decay of the learning rate. A dropout probability of 0.1 was set on all layers.
Table TABREF36 shows the results for the cross-lingual setting. M-Bert performs consistently better than the majority class baselines. Even the zero-short performance in Italian, while significantly lower than the supervised scores, is much better than the target-wise majority class baseline.
Results for the cross-target setting are given in Table TABREF37. Similar to the cross-lingual setting, M-Bert performs worse in a cross-target setting but easily surpasses the majority class baselines. Furthermore, the cross-question score of M-Bert is slightly lower than the cross-topic score.
Secondly, we fine-tune multilingual Bert (M-Bert) on the task BIBREF5 which has been pretrained jointly in 104 languages and has established itself as a state of the art for various multilingual tasks BIBREF18, BIBREF19. Within the field of stance detection, Bert can outperform both feature-based and other neural approaches in a monolingual English setting BIBREF10.
In the context of Bert we interpret the x-stance task as sequence pair classification inspired by natural language inference tasks BIBREF20. We follow the procedure outlined by BIBREF5 for such tasks. We designate the question as segment A and the comment as segment B. The two segments are separated with the special token [SEP], and the special token [CLS] is prepended to the sequence. The final hidden state corresponding to [CLS] is then classified by a linear layer.
We fine-tune the full model with a cross-entropy loss, using the AllenNLP library BIBREF21 as a basis for our implementation.
We upsampled the `favor' class so that the two classes are balanced when summing over all questions and topics. A maximum sequence length of 512 subwords and a batch size of 16 was chosen for all training runs. We then performed a grid search within the following range of hyperparameters based on the validation accuracy:
Learning rate: 5e-5, 3e-5, 2e-5
Number of epochs: 3, 4
The grid search was repeated independently for every variant that we tested. Furthermore, the standard recommendations for fine-tuning Bert were used: Adam with ${\beta }_1=0.9$ and ${\beta }_2=0.999$; an L2 weight decay of $0.01$; a learning rate warmup over the first 10% of the steps; and a linear decay of the learning rate. A dropout probability of 0.1 was set on all layers.
Table TABREF36 shows the results for the cross-lingual setting. M-Bert performs consistently better than the majority class baselines. Even the zero-short performance in Italian, while significantly lower than the supervised scores, is much better than the target-wise majority class baseline.
Results for the cross-target setting are given in Table TABREF37. Similar to the cross-lingual setting, M-Bert performs worse in a cross-target setting but easily surpasses the majority class baselines. Furthermore, the cross-question score of M-Bert is slightly lower than the cross-topic score.
Table 3 shows the results for the crosslingual setting. M-BERT performs consistently better than the previous baselines.****Results for the cross-target setting are given in Table 4. Similar to the cross-lingual setting, model performance drops in the cross-target setting, but M-BERT remains the strongest baseline and easily surpasses the majority class baselines.
Answer: M-BERT.