## 4 Experiments
### Ablation Study  
To analyze the influence of the number of blocks and decay factors in a detailed manner, we conduct several comprehensive ablation experiments. Table 6 shows the accuracy of our models (w/BERT_base) with different blocks on three datasets. The 1-block model indicates that we only use the top-layer representation of the BERT_base model without learning rate decaying, which is same as the general BERT_base baseline. As we can see, the performance does not increase monotonously with the increase of the blocks. The model performs best when the number of blocks is 2 or 3, and as the number continues to increase, the accuracy falls. Especially when the number of blocks increases to 12 (use outputs of all layers), the performance reaches the worst. It is even worse than the 1-block model on the RACE dataset. This is because the representations of adjacent layers are similar. As shown in Figure 4, for any two layers, with the decrease of their distance, their representations tend to be more similar. In other words, for two layers with close distance, the samples they can correctly solve highly overlap. Simply increasing the number of blocks not only fails to address more samples, but also introduces additional noises. In addition, using too many blocks may cause the model to be overfitting. Therefore, it is very important to properly select the number of blocks. According to Table 6 and Figure 4, we think that setting the number of blocks to three is universally effective. Next, we study the effectiveness of different decay factors in Figure 5. We can see that the model without learning rate decaying (\(\alpha=1\)) performs worst, where its third-block (top-layer) accuracy and final accuracy are both lower than the baseline which only uses the top-layer representation. This shows that only adding decision modules at different blocks not only fails in improving the final performance, but also loses the performance of the top layer. In contrast, all our models (\(\alpha>1\)) perform well and the top-layer performance even grows slightly. This illustrates that our learning rate decaying method is very effective for maintaining the top-layer performance and improving final accuracy. Moreover, the performance of the first block (bottom block) falls sharply with the increase of the decay factor \(\alpha\); the most likely reason is that the learning rate of the first block is smallest, when the final performance in the development dataset reaches the peak, the first block is still in the state of under fitting. As \(\alpha\) increases, the learning rate of the first block falls gradually and the degree of under fitting rises, so that the corresponding accuracy goes down. The performance drop does affect the final accuracy, but this does not mean that the first block is completely useless. We try to remove the first-block decision module during testing and find that the accuracy decreases by 0.2%. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Datasets/Models**} & \multicolumn{3}{c}{**RACE**} & \multirow{2}{*}{**High**} & \multirow{2}{*}{**ReCO**} & \multirow{2}{*}{**Ag.news**} \\  & **All** & **Middle** & & & \\ \hline
1-block & 65.2 & 71.8 & 62.4 & 78.5 & 94.6 \\
2-block & 66.4 & 72.1 & 64.0 & 78.9 & 94.9 \\
3-block & 66.7 & 71.9 & 64.5 & 80.2 & 95.0 \\
4-block & 66.0 & 71.1 & 63.9 & 79.4 & 94.7 \\
6-block & 65.4 & 70.1 & 63.5 & 79.7 & 94.9 \\
12-block & 64.8 & 70.5 & 62.4 & 79.2 & 94.7 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparing the performance of different numbers of blocks. **Convergence analysis of each layer. ** Due to the decaying learning rate in our model, the convergence speed of each decision module is different. Figure 6 shows the convergence in accuracy of each block. As the figure shows, the accuracy of the third block first reaches the peak, then is the second block, and the first block is converged at lastly.