Early Exit with Disentangled Representation and Equiangular Tight Frame
Year: 2023

Abstract:
Dynamic early exit has demonstrated great potential in coping with the sharply increasing number of pre-trained language model parameters, which can achieve a good trade-off between performance and efficiency. The existing early exit paradigm relies on training parametrical internal classifiers at each intermediate layer to complete specific tasks. Based on the predictions of these internal classifiers, different methods are designed to decide when to exit. Under this circumstance, each intermediate layer takes on both generic language representation learning and task-specific feature extraction, which makes each intermediate layer struggle to balance two types of backward loss signals during training. To break this dilemma, we propose an adapter method to decouple the two distinct types of representation and further introduce a non-parametric simplex equiangular tight frame classifier (ETF) for improvement. Extensive experiments on monolingual and multilingual tasks demonstrate that our method gains significant improvements over strong PLM backbones and early exit methods.

heading: 1 Introduction
text: In recent years, fundamental models that rely on the scaling effect have penetrated different NLP scenarios (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019; Clark et al., 2020; Lewis et al., 2020; Raffel et al., 2020; Brown et al., 2020; He et al., 2021a; OpenAI, 2022). However, with the increasing number of the pre-trained model parameters, the expensive inference cost hinders their usage in practical applications. Besides, Overthinking problem (Kaya et al., 2019) also restricts the ability of PLMs. Specifically, since PLMs are overparameterized, they can give correct answers according to the shallow representations at earlier layers, while the high-level representation
∗Equal contribution. †Corresponding author.
may instead contain too much over-complicated or irrelevant information to make the accurate prediction (Xin et al., 2020b; Liu et al., 2020).
To achieve a good trade-off between inference cost and performance, the early exit mechanism (Xin et al., 2020a; Zhou et al., 2020; Li et al., 2021b; He et al., 2021c; Xin et al., 2021; Banino et al., 2021; Balagansky and Gavrilov, 2022; Sun et al., 2022), a kind of adaptive inference strategy, has been proposed. These methods insert an internal classifier after each layer of the PLMs to predict the label of a given instance. In the inference stage, if the prediction is confident enough earlier, the sample will end the inference without going through the entire PLM. Nevertheless, how to train a competitive early exit PLM is not a trivial problem. Since each classifier tries to be optimized, different optimization procedures from different classifiers may conflict and interfere with each other (Phuong and Lampert, 2019). Exist-
14128
ing methods can be divided into two categories: (1) jointly training all classifiers and using heuristics or learnable methods to weight loss functions (Zhou et al., 2020; Li et al., 2021b; Zhu, 2021; Liao et al., 2021); (2) training classifiers in different stages and freezing the Transformer layer when training internal classifiers.
However, in the early exit mechanism, each Transformer layer plays two roles: providing classifiable representation for the corresponding internal classifier and semantic features for subsequent Transformer layers. The former class of methods is obsessed with improving the classification ability of the middle layer of PLMs while ignoring its ability to capture other linguistic features. The latter class focuses on maintaining the ability to extract semantic representations while constraining the performance of internal classifiers. Although some work (Xin et al., 2021) has attempted to balance the two roles of the Transformer layer by alternate training, they have not explicitly decoupled the two distinct types of representation.
In this work, we propose a novel early exit method that enables the PLM’s intermediate representation to consider both the classification ability and the ability to capture semantic information. Specifically, our approach hands off the taskspecific representation of the intermediate transformer layer to an adapter module (Houlsby et al., 2019; He et al., 2022) with a small number of parameters, thus disentangling the task-specific and universal representation (see Figure 1). In addition, due to the limited expressive power of the adapter, we use a simplex equiangular tight frame classifier (ETF) (Yang et al., 2022) to enhance the classification ability of the internal classifiers. Experimental results demonstrate that our proposed early exit method significantly improves the performance of monolingual and multilingual tasks over existing strong early exit methods.
Overall, our contributions are shown below:
• We empirically study the ability to extract generic linguistic representation in the internal layers of early exit models. The study reveals that the internal layers have difficulty taking on both generic language representation learning and task-specific feature extraction. • We propose an early exit method which use adapter modules to disentangle the two conflicting representations and utilize equiangular tight frame classifiers to improve representations for
classification. • Experimental results and analysis on monolin-
gual and multilingual tasks demonstrate that our proposed early exit method performs better than previous methods. Our code is available at https://github.com/Jikai0Wang/DREE.

heading: 2 Preliminary Study
text: It is controversial whether the last classifier and the internal classifiers should be trained jointly in early exit. Some studies (Xin et al., 2020b, 2021; Liu et al., 2020) divide the training into two stages to train the last classifier and the internal classifiers, respectively, to preserve the best model ability for the final layer. While others (Zhou et al., 2020; Balagansky and Gavrilov, 2022) train the whole model simultaneously. In this section, we study the question:“Can the same representation serve both the classifier and subsequent layers?”
Following Durrani et al. (2021), we train layerwise probes to check how much linguistic knowledge is preserved in each layer after finetuning. We evaluate the model on two linguistic tasks: POS tagging using the Universal Dependencies v2.5 English dataset from XTREME (Hu et al., 2020) and syntactic chunking using CoNLL 2000 shared task dataset (Tjong Kim Sang and Buchholz, 2000).
We conduct experiments on PABEE (Zhou et al.,
2020), which jointly trains the last and internal classifiers. As shown in Figure 2, the red lines refer to the performance of BERT without any finetuning. The orange and blue lines represent the performance of BERT finetuned on SST-2 with and without PABEE. The model preserves the information of the downstream tasks in the higher layers with a corresponding loss of the linguistic knowledge learned in the pre-training after finetuning. Moreover, the gaps between the blue and orange lines are apparent, indicating that the training with PABEE further disturbs linguistic knowledge. On the other hand, a representation trained only to serve the subsequent layers often has a poor performance for classification, especially in the lower layers. As a result, to maintain the internal representations’ capability and improve the classification performance, it is essential to train disentangled representations in the intermediate layers.

heading: 3 Method
text: In this section, we introduce our method for early exit, which first utilizes adapter modules to disentangle generic and task-specific representation. To further improve the classification representation, we replace learnable classifiers with equiangular tight frame (ETF) classifiers. Throughout this section, we consider the case of multi-class classification with samples {(xi, yi)}ni=1, where xi is a token sequence and yi is its label.

heading: 3.1 Disentangled Internal Representations
text: Existing early exit methods utilize the intermediate layers to learn both task-specific representation for task prediction and extract generic linguistic representation for the subsequent layers, making the model struggle to balance the two learning objectives. To address such a dilemma, we investigate how to improve task-specific representations of internal Transformer layers without hindering the learning of the generic representation. Inspired by the recent emergence of efficient tuning that freezing most of the parameters of PLMs and training with a small number of parameters can achieve comparable performance to full-parameter finetuning, we propose to fix the internal Transformer layers to consistently extract linguistic representation and utilize additional adapter modules to learn task-specific representation. For a given sample pair (x, y), each Transformer layer Li outputs a
hidden state hi as the input to Li+1:
si = LN(FFN(SA(hi−1)) + hi−1) s′i = FFN(FFN(si)) + si hi = LN(s ′ i)
(1)
where SA is a self-attention sub-layer, FFN is a feed-forward sub-layer, and LN means Layernorm. At the same time, the adapter module outputs another hidden state h′ for classification:
h′i = LN(Adapter(si) + s ′ i) (2)
Following Houlsby et al. (2019), the adapter module includes a stack of down- and up-scale fully connected neural network:
Adapter(si) = fup(ReLU(fdown(si))) (3)
where fdown ∈ Rd×m, fup ∈ Rm×d are the down and up projection layers. d is the dimension of the PLM, and m is the hidden size of the adapter. We pass h into the next transformer layer and use h′ for classification. Unlike the standard adapter, to reduce the inference time overhead caused by the adapter module, we use the parallel adapter (He et al., 2022) and only add an adapter module to the feed-forward sub-layer, as shown in Figure 3. We have experimented with the sequential adapter, but its performance is inferior to the parallel adapter.

heading: 3.2 Improving Classifiable Representation
text: A recent study (Zhu et al., 2021; Ji et al., 2021; Tirer and Bruna, 2022; Yang et al., 2022) has shown a phenomenon called neural collapse that the within-class means of features and the classifier vectors converge to the vertices of a simplex equiangular tight frame (ETF) at the terminal phase of training (Figure 4 shows a simple example of the three-class classification). Galanti et al. (2021) demonstrate that this phenomenon shows better generalization. Now that the distribution of optimal classification representation is known theoretically, Yang et al. (2022) propose the ETF classifier, which is randomly initialized as an equiangular tight frame and free from training.
Definition 1 (ETF classifier). A simplex equiangular tight frame is a collection of vectors wi ∈ Rd, i = 1, 2, ...,K, d ≥ K − 1, it satisfies:
W ETF =
√ K K − 1U ( IK − 1 K 1K1 T K ) , (4)
where W ETF = [w1,w2, ...,wK ] ∈ Rd×K , IK ∈ RK×K is the identity matrix, 1K ∈ RK is a vector of all ones and U ∈ RK×K is orthonormal.
Due to the limited generalization of the adapter compared to full-parameter fine-tuning, we utilize the ETF classifier to induce better classifiable representation. For the hidden state h′i from the Transformer layer Li, the ETF classifier output the probability pi(y|x):
pi(y|x) = Softmax(W ETFh′i) (5)
During the training phase, ETF classifiers are frozen, and we fine-tune the adapter parameters to align classification features with ETF classifiers, aiming to achieve neural collapse phenomenon with better generalization.

heading: 3.3 Training and Inference
text: Training. Following Xin et al. (2020b), we divide the training into two stages. The first stage trains the last classifier, i.e., without launching early exit, and the second stage trains internal classifiers. In the first stage, we fine-tune the whole parameters of the PLM and the last classifier with labeled data from downstream tasks without training the adapter modules and the internal classifiers. We use the cross-entropy loss for classification:
Lstage1 = CE(logits, y). (6)
We use the mean squared error for regression tasks:
Lstage1 = (y − ŷ)2. (7)
In the second stage, we freeze all parameters finetuned in the first stage and then train adapter modules and internal classifiers to enable the PLM early exit. The loss of an internal classifier is also calculated with cross-entropy for classification:
Li = CE(logitsi, y). (8)
For regression, we also use the mean squared error:
Li = (y − ŷi)2. (9)
The total loss Lstage2 uses a weighted average:
Lstage2 = ∑n
j=1(n− j) · Lj∑n j=1(n− j) , (10)
where n represents the number of hidden layers. Note that we give the early internal classifiers a bigger weight since they need more transformation to fit representations for classification. Inference. Following Zhou et al. (2020), we adopt a patience-based strategy to decide which layer to exit. Specifically, we set the patience to t. Given a sample x, the early-exit model predicts from bottom to top. If the predictions of t consecutive intermediate classifiers remain “unchanged”, the model exits at that layer and outputs the corresponding prediction. While for regression, we consider the prediction as “unchanged” by:
unchanged = { True if |yi − yi−1| < δ False if |yi − yi−1| ≥ δ
(11) where yi represents the prediction of the current layer and δ is a pre-defined threshold.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our proposed approach on monolingual tasks and multilingual tasks. For monolingual tasks, our experiments are conducted on the GLUE benchmark (Wang et al., 2018), including MRPC, QQP, SST-2, MNLI(matched/mismatched), QNLI, RTE, CoLA, and STS-B. Following Zhou et al. (2020), if a dataset has more than one metric, we report the arithmetic mean of the metrics. For multilingual tasks, we conduct experiments on paraphrase identification task (PAWS-X; Yang et al., 2019) and natural language inference (XNLI; Conneau et al., 2018). We show details of all the datasets in the Appendix.

heading: 4.2 Baselines
text: We compare our methods with the following competitive models: Backbone models. We use BERT-base and ALBERT-base for monolingual tasks, which are widely used in studies of early exit. For multilingual tasks, we use multilingual BERT. All these pre-trained language models are released by HuggingFace1. PABEE. PABEE is a patience-based early exit method proposed by Zhou et al. (2020). PonderNet. PonderNet is proposed by Banino et al. (2021), which treats the exit layer’s index as a latent variable. PBERT/PALBERT. Balagansky and Gavrilov (2022) proposed a deterministic Q-exit criterion to improve the performance PonderNet. We apply the Q-exit criterion to BERT and ALBERT, and we denote them as PBERT and PALBERT.

heading: 4.3 Experimental Setup
text: Training. For all experiments, we set the batch size for training to 32. We search for the best learning rate in {1e-5, 2e-5, 3e-5, 5e-5} for all baselines and the first training stage of our method. The range of learning of the second training stage of our method is in {1e-3, 2e-3, 3e-3, 5e-3, 8e-3, 9e3}. The downsample sizes of adapters are searched in {32, 64, 128, 256}. Since the effect of using the last layer for classification remains the same during the second training stage, we choose the best checkpoint on the development set with patience set to 6. To avoid the propensity of the model to use
1https://huggingface.co/
later layers for classification, which results in poor acceleration, we filter checkpoints whose average Flops (M) on the development set are greater than τ on the verification set. τ is chosen in {8000, 9000, 10000}. We conduct the experiments on one NVIDIA GTX3090 GPU. Inference. We set the batch size for inference to 1. Patience is set to 6 for patience-based methods following Zhou et al. (2020), accelerating the inference while maintaining a satisfactory effect. For PonderNet, PBERT, and PALBERT, the threshold for early exit is set to 0.5 following Balagansky and Gavrilov (2022). For STS-B, we set the threshold δ to 0.5. We use Flops to calculate the speedup ratio.

heading: 4.4 Main Results
text: We report our experimental results with BERT and ALBERT backbone for monolingual tasks on GLUE in Table 1. Our method outperforms all compared approaches from the perspective of the macro score while maintaining the average speedup ratio between 1.25 and 1.27 on both the development and test sets. Our approach works well on small datasets such as CoLA, RTE, MRPC, and STS-B, on which the previous approaches of early exit often have poor performance.
We also conduct experiments on multilingual tasks to examine the generality of our approach. Table 2 shows a comparison of our approach with baseline and PABEE. In addition to accelerating inference, our approach outperforms mBERT and PABEE on both PAWSX and XNLI.
Since our approach introduces an adapter in each intermediate layer, a small amount of computational overhead is added. In order to make a fairer and more comprehensive evaluation of our approach, we adjust the patience to obtain the effect at different speed-up ratios. We compare our approach with PABEE, as we both adopt a patiencebased early exit mechanism, making it convenient to adjust the speed-up ratio. As shown in Figure 5, we experiment on 3 GLUE datasets with ALBERT backbone. The ability to capture semantic information is trained in the first training stage and maintained by freezing transformer layer parameters during the second training stage, improving performance when using the whole model for inference. Moreover, during training, the disentangled representations avoid conflicts between two different loss signals in the transformer layer. This division of functions makes the model represen-
tation cleaner and more meaningful. As a result, the performance penalty caused by accelerated inference reduces significantly at a speed-up ratio between 1 and 1.5. It is worth noting that there is an upward trend in the curves of our approach at a low speed-up ratio, indicating that our approach helps to solve the overthinking problem.

heading: 5 Analysis and Discussion

heading: 5.1 Ablation Study
text: We conduct an ablation study with BERT backbone on MNLI, SST-2, and MRPC. Three experiments
are performed based on our method:
• w/o ETF: We remove ETF classifiers and replace them with original trainable classifiers. • w/o adapters: We remove all adapter modules and directly use the output of the layernorm after two feed-forward layers as the classifier’s input for each intermediate layer. • w/o two-stage: We train the transformer layers and the adapter modules simultaneously instead of adopting the two-stage training strategy.
The results of the ablation study are shown in Table 3. As bridges between hidden states and classifiers, the adapters undertake to learn task-specific representations and convert intermediate layer representations into representations for classification. Note that the additional computation caused by the adapter modules is about 2% on BERT and 3% on ALBERT, which is acceptable considering the benefits it brings. Acceleration and performance are both degraded without using adapter modules. Moreover, the performance and stability of the model drop significantly and without using the two-stage training strategy, which indicates that the gradients returned by the higher layers mixed with the gradients returned by the classifiers confuse the model’s representation and mislead the optimization. ETF classifiers generally outperform learnable classifiers in performance and efficiency especially for low-resource datasets. They further improve the model’s performance by enhancing the intermediate layers’ classification ability while reducing the number of learnable parameters.

heading: 5.2 Impact of Adapter Hidden Size
text: The hidden size of adapters m affect many aspects of our approach. Experiments are conducted to study the impact of m. We choose ALBERT as the backbone because it has a bigger hidden size. Figure 6 shows that the hidden size of adapters has little effect on the accuracy while it makes a difference in the speed-up ratio. The accuracy rate reaches the top with a good speed-up ratio when m is set to 32. And a large size leads to a decrease
both in performance and acceleration. As shown in Figure 7, a bigger hidden size of adapters encourages the samples to exit at a lower layer. Based on this observation, we assume that a small size restricts the ability of adapters to express so that the intermediate classifiers have difficulties in giving a unified prediction. While a large size makes the lower layers too confident to wait for a high-level representation from the higher layers, resulting in a wrong decision about when to exit in some cases. This finding demonstrates that a proper hidden size of adapters gives the success to stop inference at a proper layer.

heading: 5.3 Error Analysis
text: In order to further explore the performanceboosting aspects of our method, we conduct an error analysis between PABEE and our approach. Suppose a model has a stable performance improvement compared to the baseline. In that case, the model should predict as correctly as possible for the samples that the baseline predicts correctly and that the model can give correct predictions for the instances that the baseline fails. Thus, we divide the samples in the development sets into two
classes PABEE (Correct) and PABEE (Wrong), according to the prediction results of PABEE. Then we test our approach on the two classes to observe the change in the distribution of correctly predicted samples. As shown in Figure 8, our method correctly predicts the vast majority of samples in PABEE (Correct) and about half in PABEE (Wrong). There can be two reasons for the prediction error: (1) The model stops inference too early; (2) Biased representations fail to make a correct understanding of the samples. So the experimental results indicate that our approach improves the representation accuracy and gives better answers to when to exit.

heading: 6 Related Work
text: Early exit Early exit focuses on enabling inputadaptive inference to reduce the computational cost, which has been proven effective on various NLP tasks (Elbayad et al., 2019; Xin et al., 2020a; Li et al., 2021b; He et al., 2021c; Xin et al., 2021; Sun et al., 2022; Schuster et al., 2022). Because the model needs to decide whether stop inference at a specific layer when using early exit, so there are two main problems: ‘How to decide whether stop inference at each layer?’ and ‘How to induce a better internal classifier for each layer?’.
To the former problem, Xin et al. (2020b); Schwartz et al. (2020); Xie et al. (2021) use confidence-based criterion; Zhou et al. (2020) propose a novel and effective patience mechanism;
Sun et al. (2021) propose a voting-based strategy. In addition, Banino et al. (2021); Balagansky and Gavrilov (2022) utilize a latent variable to predict the exit layer’s index. For the latter one, Liu et al. (2020); Geng et al. (2021) use self-distillation to induce better internal classifiers; Zhu (2021) extent self-distillation to mutual distillation and use learnable weights to balance off different internal classifiers’ objectives; Liao et al. (2021) use global past and future information with imitation learning to train internal classifiers; Sun et al. (2021) maximize the mutual information of internal classifiers to enhance the diversity of classifiers, making it suitable for voting-based strategy. Besides, Li et al. (2021a) dynamically cascade proper-sized and complete models, enabling shallow layers with high-level semantic information. Liu et al. (2022) propose a novel pre-training method that encourages the intermediate layers of the pre-trained model to learn high-level semantics, making the pre-trained model more suitable for the early exiting mechanism.
Adapter With the increase in the number of parameters of PLMs, full-parameter fine-tuning faces difficulties in computing resources and is prone to the over-fitting problem (Phang et al., 2018; Dodge et al., 2020; Zhang et al., 2021). Therefore, research on efficient fine-tuning (Houlsby et al., 2019; Zaken et al., 2022; Hu et al., 2021; Li and Liang, 2021) is developing rapidly. Adapter is a promising efficient fine-tuning method. Adapter-based methods inject small-scale adapters to the Transformer layers and only tune these adapters. Many studies (Pfeiffer et al., 2020a,b; Guo et al., 2020; Rücklé et al., 2021; He et al., 2021b; Han et al., 2021) have shown that adapter-based methods can achieve comparable performance to the full-parameter finetuning. To further reduce the parameter amount of the adapter module, Karimi Mahabadi et al. (2021) propose Compacter, a more lightweight adapter that utilizes a combination of hypercomplex multiplication and parameter sharing. Due to the modularity of the adapter, adapter-based tuning is suitable for multi-task learning (Stickland and Murray, 2019; Mahabadi et al., 2021) and can provide different task-specific representations for various tasks. Adapterfusion Pfeiffer et al. (2021) proposes a fusion method named AdapterFusion that fuses adapter representations of different tasks and makes full use of cross-task knowledge.

heading: 7 Conclusion
text: In this paper, we investigate the internal representation of early exit models and observe that the internal layers have difficulty providing good generic linguistic representations for subsequent layers and good task-specific representations for internal classifiers. We propose an adapter-based method to disentangle the two conflicting representations and utilize equiangular tight frame classifiers to improve representations for classification. Experiments on the GLUE benchmark and two cross-lingual transfer tasks demonstrate that our proposed method performs better than existing methods. For future work, we would like to explore: (1) strengthening the information interaction between layers to make full use of the previous layers’ predictions, thus optimizing the representation for the internal classifiers; (2) optimizing the early exiting mechanism to further improve performance and accelerate inference; (3) applying our method to more advanced pre-trained models such as DeBERTa (He et al., 2020, 2021a), ElasticBERT (Liu et al., 2022), etc.
Limitations
Even though our work improves early exit performance effectively, some limitations are still listed below:
• Our approach focuses on making the intermediate representations of early exit models capable of general linguistic representation learning and task-specific representation extraction. Therefore, we did not fully use the model’s high-level representation and fuse representations of previous layers, which may restrict the performance of our method. For future work, we would like to strengthen the information interaction between layers to make full use of the previous layers’ predictions, thus optimizing the representation for the internal classifiers. • Although our early exit method has achieved better performance, we have lost some inference speed due to the introduction of additional adapter modules. In the future, we will try more efficient adapter-based tuning. • In recent years, the parameter size of generative pre-trained models has been continuously increasing, leading to remarkable performance on various NLP tasks. There is an urgent need to develop inference acceleration methods for generative pre-trained models. Unfortunately, our
method is limited to discriminative pre-trained models. Our future work will investigate early exit strategies for generative pre-trained models.

heading: Acknowledgements
text: This work is supported by the National Science Foundation of China (No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and JSSCBS20210661. This work is partially supported by the joint research project of Alibaba and Soochow University.

heading: A Details of Datasets
text: We show details of datasets for monolingual tasks in Table 4 and multilingual tasks in Table 5.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: the last section

heading: 3 A2. Did you discuss any potential risks of your work?
text: the last section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section 4.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4.1
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, The datasets we use are widely recognized public datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? section 4.3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4.4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we would like to explore: (1) strengthening the information interaction between layers to make full use of the previous layers’ predictions, thus optimizing the representation for the internal classifiers; (2) optimizing the early exiting mechanism to further improve performance and accelerate inference; (3) applying our method to more advanced pre-trained models such as DeBERTa (He et al. , 2020, 2021a), ElasticBERT (Liu et al. , 2022), etc. Limitations
Even though our work improves early exit performance effectively, some limitations are still listed below:
• Our approach focuses on making the intermediate representations of early exit models capable of general linguistic representation learning and task-specific representation extraction. Therefore, we did not fully use the model’s high-level representation and fuse representations of previous layers, which may restrict the performance of our method. For future work, we would like to strengthen the information interaction between layers to make full use of the previous layers’ predictions, thus optimizing the representation for the internal classifiers. • Although our early exit method has achieved better performance, we have lost some inference speed due to the introduction of additional adapter modules. In the future, we will try more efficient adapter-based tuning. • In recent years, the parameter size of generative pre-trained models has been continuously increasing, leading to remarkable performance on various NLP tasks. There is an urgent need to develop inference acceleration methods for generative pre-trained models. Unfortunately, our
method is limited to discriminative pre-trained models. Our future work will investigate early exit strategies for generative pre-trained models