RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 3 : Heatmap of Kullback-Leibler Divergence between the output distribution of top-2 routing strategy ( p_{\text{top-2}}(x_{t}|x_{<t}) ) and different rank- k routing strategies ( p_{\text{rank-k}}(x_{t}|x_{<t}) ).
The k in rank- k routing ranges from 1 to 8.
The values in the heatmap are scaled by 10^{5} .
This example is taken from the GSM8K dataset. An additional quantitative study of the KLD is provided in Appendix A .; Heatmap of Kullback-Leibler Divergence between the output distribution of top-2 routing strategy ( p_{\text{top-2}}(x_{t}|x_{<t}) ) and different rank- k routing strategies ( p_{\text{rank-k}}(x_{t}|x_{<t}) ).
The k in rank- k routing ranges from 1 to 8.
The values in the heatmap are scaled by 10^{5} .
This example is taken from the GSM8K dataset. An additional quantitative study of the KLD is provided in Appendix A .
Figure 4 : Experimental results of different weak activations. We set the strong activation with top-2 routing in SCMoE. The detailed results with their hyperparameters are report in Appendix Table 8 .; Experimental results of different weak activations. We set the strong activation with top-2 routing in SCMoE. The detailed results with their hyperparameters are report in Appendix Table 8 .
Figure 5 : Experimental results on combining SCMoE with self-consistency on GSM8K using Mixtral 8x7B.; Experimental results on combining SCMoE with self-consistency on GSM8K using Mixtral 8x7B.
Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with Mixtral 8x7B. We report the best results for each method here. The performance of each method with different hyperparameters can be found in the Appendix Table 7 .
Experimental results of different strong activations.
We set the weak activation with rank- 2 routing.
For each benchmark,
we select the top- k routing yielding the best performance in Figure 1 as the ideal strong activation.
The specific hyperparameter settings can be found in Table 9 .
Averaged decoding latency for each method. CS is short for contrastive search and CD is short for contrastive decoding. We set k = 3 for ensemble routing, while for dynamic routing we set threshold = 0.5. The speeds are tested on 4 A100 40G with batch size = 1.
Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with DeepSeekMoE-16B. We report the best results for each method here. The performance of each method with different hyperparameters can be found in the Appendix Table 10 .
Average KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and different distribution across three token sets using the GSM8K dataset.
Specifically, we compare p_{\text{top-2}}(x_{t}|x_{<t}) with p(x_{t}|x_{<t}) generated by Mixtral 8x7B with rank- k routing, Mixtral 8x7B with random-1 routing and Mistral-7B, respectively.
“ \uparrow ” and “ \downarrow ”: the percentage increase and decrease relative to the “All” token set.
The values in the table are scaled by 10^{5} .
The proportion of experts that are activated by rank- k routing during weak activation but not activated by top-2 routing in strong activation on GSM8K with Mixtral 8x7B. Unchosen experts refer to the experts not selected using default top-2 routing.
Details for Table 1 . Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with Mixtral 8x7B. The performance of each method with different hyperparameters.
Details for Figure 4 . Experimental results of different weak activations with Mixtral 8x7B. We set the strong activation with top-2 routing in SCMoE.
Details for Table 2 . Experimental results of different strong activations on GSM8K, StrategyQA, MBPP and HumanEval with Mixtral 8x7B. We set the weak activation with rank-2 routing.
Details for Table 4 . Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with DeepSeekMoE-16B. The performance of each method with different
hyperparameters. In SCMoE, "A/B" refers to top- k and \beta , respectively.
Failed examples of DoLa.