## 4 Experiments
### Information Type Analysis  
In this section, we explicitly analyze which types of reading comprehension task that can be addressed by each layer. Generally speaking, answering different questions requires different information. We collect the results predicted by a 3-block model on the ReCO development dataset and split them into three groups according to the source of \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Datasets/Models**} & \multicolumn{2}{c}{**Book Review**} & \multicolumn{2}{c}{**Ag.news**} \\  & **Acc** & **FLOPs** & **Acc** & **FLOPs** \\ \hline BERT\_base & 86.9 & 21,785 M & 94.5 & 21,785 M \\ \hline Ours & & & & \\ \(v=0.0\) & 87.9 & 21,785 M & 95.0 & 21,785 M \\ \(v=0.1\) & 87.8 & 16,856 M & 95.0 & 10,584 M \\ \(v=0.3\) & 87.8 & 13,942 M & 94.6 & 8705 M \\ \(v=0.5\) & 87.7 & 12,045 M & 93.6 & 7738 M \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison with the BERT_base model on classification datasets. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Datasets/Models**} & \multirow{2}{*}{**Ag.news**} & \multirow{2}{*}{**Book Review**} \\  & & & \\ \hline BERT base & 94.5 & 86.9 \\ RoBERTa\_base & 95.0 * & 87.1 * \\ RoBERTa\_large & 95.4 * & 88.3 * \\ \hline Ours & & & \\ BERT\_base & 95.0 \(\pm\) 0.02 & 87.8 \(\pm\) 0.04 \\ RoBERTa\_base & 95.2 \(\pm\) 0.01 & 88.3 \(\pm\) 0.03 \\ RoBERTa\_large & 95.5 \(\pm\) 0.04 & 88.7 \(\pm\) 0.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison with the baselines on two two classification datasets. * indicates our re-implementation. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Datasets/Models**} & \multicolumn{2}{c}{**RACE**} & \multicolumn{2}{c}{**ReCO**} & \multicolumn{2}{c}{**Dream**} \\  & **Acc** & **FLOPs** & **Acc** & **FLOPs** & **Acc** & **FLOPs** \\ \hline BERT\_base & 65.0 & 173,975 M & 77.1 & 43,486 M & 63.2 & 130,464 M \\ \hline Ours & & & & & \\ \(v=0.0\) & 65.6 & 173,953 M & **79.9** & 43,487 M & 64.1 & 130,465 M \\ \(v=0.1\) & 65.6 & 171,344 M & 79.9 & 39,306 M & 64.1 & 130,465 M  
the final answer (selected from which layer). Then we randomly select 100 samples from each group, respectively, and ask volunteers to label the information type required by each sample. The information types are mainly taken from ReCO [3] and the findings of [PERSON] et al. [8]. ReCO presents seven information types, such as \"Lexical Knowledge\", \"Syntactic Knowledge\", \"Specific Knowledge\". [PERSON] et al present four information types, such as \"Syntactic Features\" and \"Semantic Features\". Considering that too fine-grained categories are difficult for volunteers to distinguish, we defined the four information types (\"Lexical Knowledge\", \"Syntactic Knowledge\", \"Semantic Knowledge\" and \"Specific Knowledge\") in our paper. In order to ensure that the labeled categories are accurate and not affected by other information, we randomly shuffle all samples and only provide the volunteers with passages, questions and options. Table 5 shows the frequencies of the required information type in each layer. The information types are mainly taken from the popular MRC work [3]. As we can see, more than half of the samples in the first block require lexical knowledge such as synonymy matching. In the second block, the model prefers answering the questions that require syntactic knowledge such as sentence structure information. The third block is devoted to addressing the reasoning tasks such as logical reasoning, casual inference, and so on, which generally need deep semantics knowledge. These results show that the low layer and middle layer are good at solving MRC problems, needing shallow information and the top layer is skilled in addressing deep reasoning tasks. Moreover, there are also some questions that require specific external knowledge, and the third block performs best on them. One possible reason is that the third block contains more background knowledge due to the deeper network.