RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 9: Halting plot for a 12-layer SUT encoder and decoder. Number represents the top expert being used.
For WMT’14 English-German, we find that these translation models halt much later, suggesting that the translation task requires more computational steps than the 6-layer SUT we used.
However, further increasing the number of layers to 12 layers does not bring about much benefit, as evidenced by the halting in Figure 4 , which is an example of the halting mechanism using nearly all layers.
Looking at the top-scoring expert for the plot, it appears the experts selected are largely word dependent, and tend to repeat.; Halting plot for a 12-layer SUT encoder and decoder. Number represents the top expert being used.
For WMT’14 English-German, we find that these translation models halt much later, suggesting that the translation task requires more computational steps than the 6-layer SUT we used.
However, further increasing the number of layers to 12 layers does not bring about much benefit, as evidenced by the halting in Figure 4 , which is an example of the halting mechanism using nearly all layers.
Looking at the top-scoring expert for the plot, it appears the experts selected are largely word dependent, and tend to repeat.
BLEU score on WMT14 En-De translation datasets.
MACs (Multiply–Accumulate Operations) 1 1 1 The open-source tool ptflops ( https://github.com/sovrasov/flops-counter.pytorch ) is used to calculate the MACs. measures the computational complexity of each model. a Vaswani et al. ( 2017 ) , b Liu et al. ( 2020 ) , c Peng et al. ( 2020 ) , d Zhang et al. ( 2022 ) , e Dehghani et al. ( 2018 ) , f Myle et al. ( 2018 ) , g Wu et al. ( 2018 )
Ablation Study. “– MIM loss” means replacing the MIM loss with the load balancing loss used in (Fedus et al., 2021 ) .
“– MoMHA” means replacing MoMHA with the MoA introduced in (Zhang et al., 2022 ) .
FFD Expert-Word co-occurrences.
CFQ Results. Results on UT are an average of 5 runs on different seeds.
Test accuracy of the models, trained on operation lengths of \leq 6 , with their out-of-distribution results shown here (lengths 7-12).
LSTM baseline from Bowman et al. ( 2015 ) , and Transformer baseline from Shen et al. ( 2019 )
Hyperparameters for different models.
Training Hyperparameters for different models.
The BSZ represent the maximum number of tokens in each batch.
Results on Long-range Arena ListOps benchmark. We include the Transformer and top 2 results from Tay et al. ( 2020 ) for brevity, and the best performing external result on the leaderboard at https://github.com/google-research/long-range-arena
Experimental results for the Vanilla Transformers (VT) and a Universal Transformer (SUT with 1 expert). results are full-string match accuracy. 1 Furrer et al. ( 2020 ) 2 Hupkes et al. ( 2020 )