## 1 Introduction  
Machine reading comprehension (MRC) [1, 2, 3, 4] has attracted increasing interest in recent years. It is presented to evaluate how well a machine understands human language by asking the machine to answer questions according to a given passage. In particular, multi-choice machine reading comprehension [2] is one of the most difficult tasks in MRC since it usually requires different natural language processing skills such as word matching, syntactic structure matching, logical reasoning and summarization, compared to span-selection tasks [1]. In other words, addressing multi-choice reading comprehension tasks requires various information items due to the abundant diversity of the questions, options and passages. Therefore, researchers propose a variety of methods to provide richer information for the different reading comprehension samples. Among them, pre-trained language models such as BERT [5] and RoBERTa [6] have become an important trend. A pre-trained language model usually uses several self-supervised pre-training tasks to train a deep transformer-based network [7] and learn rich information from large-scale corpora. It brings great success to MRC and achieves SOTA performance in multiple tasks simply by fine-tuning. [PERSON] et al. [8] conducted a series of experiments on BERT and found that its intermediate layers learn a rich hierarchy of linguistic information. Specifically, BERT learns phrase-level information and surface information in the low layer, syntactic features in the middle layer and semantic features at the top layer. In other words, BERT naturally contains different information at different layers of the network. However, existing methods generally use the top-layer representation of pre-trained models to address the MRC tasks [3, 9, 10]. Therefore, while the pre-trained models encode different information at different layers, the current methods cannot directly exploit the diverse information for the various questions and passages. How to adaptively select the required information in pre-trained model for MRC task is still a problem to be solved. An intuitive idea to address the above problem is building multiple decision modules by making use of the outputs at different layers of pre-trained models respectively, and then synthesizing the multiple decisions to reach the final solution. In this way, the representations at intermediate layers could be exploited directly, which helps the different samples to utilize the required information from different layers. However, another problem arises. Since each decision module has the similar supervisory signal, the output representations at those layers tend to be similar at fine-tuning. This damages the information diversity in different layers of the original pre-trained models. An effective way should make use of the multi-layer representations without damaging the information diversity. This paper therefore proposes a simple but effective multi-decision based transformer model with learning rate decaying to address the above problem. For a pre-trained model with stacked \(L\) transformer layers, we divide them evenly into \(N\) blocks, each block consists of \(L/N\) Transformer layers. We then use the output of each block for decisions. For example, if \(L=12\) and \(N=3\), the representation of the 4 th layer (the output of 1 st block), the 8 th layer (the output of 2 nd block) and the 12 th layer (the output of 3 rd block) are used. Each block answers the input question separately. The final answer is then chosen by two manners: \"performance first\" and \"speed first\". When updating the parameters, in order to avoid the information diversity being damaged, we weaken the influences of supervisory signals on lower layer parameters. This is achieved by a learning rate decaying method which gradually reduces the learning rate from top to bottom of the transformer stack.