Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification
Year: 2023

Abstract:
Out-of-domain (OOD) intent classification is an active field of natural language understanding, which is of great practical significance for intelligent devices such as the Task-Oriented Dialogue System. It mainly contains two challenges: it requires the model to know what it knows and what it does not know. This paper investigates “overthinking” in the openworld scenario and its impact on OOD intent classification. Inspired by this, we propose a two-birds-one-stone method, which allows the model to decide whether to make a decision on OOD classification early during inference and can ensure accuracy and accelerate inference. At the same time, to adapt to the behavior of dynamic inference, we also propose a training method based on ensemble methods. In addition to bringing certain theoretical insights, we also conduct detailed experiments on three real-world intent datasets. Compared with the previous baselines, our method can not only improve inference speed, but also achieve significant performance improvements. Code is publicly available.1

heading: 1 Introduction
text: With the increasing popularity of intelligent devices such as the Task-Oriented Dialogue System and the increasingly open environment they face, it becomes more and more challenging to accurately understand the intents behind the utterance of the users, on one hand, it needs to ensure the accuracy of In-domain (IND) intents, and on the other hand, it needs to be able to effectively identify OOD intent. These days, it has became a de facto standard to take the semantic representation of the last layer of finetuned Pre-train Models (PTMs) for intent understanding (Zhang et al., 2021b; Zhou et al., 2022). However, in the closed-world scenario (i.e.,
∗Corresponding author. 1https://github.com/zyh190507/Dynamic-Ensemble-for-
OOD
training and test set come from the same distribution), previous studies have pointed out that PTMs would be “overthinking” the semantic features of the sample (Kaya et al., 2019; Zhou et al., 2020), i.e., the representation of the sample may become too complex after reaching the last layer through multiple stacked transformer layers, thus affecting the final decision of the model.
Naturally, in the open-world scenario (i.e., the condition of the same distribution is not maintained), will the PTMs be “overthinking” the difference between IND and OOD intents? We want to explore the evolution of the model’s ability to distinguish IND and OOD through the forward process of two widely used models BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2020).
To this end, we first attach extra Internal Classifiers (ICs), which also are referred to as off-ramps, between different internal transformer layers of the model as shown in Figure 2 (a) and train these internal classifiers and all transformer layers in two stages as suggested in Xin et al. (2020). The outputs of these classifiers are used to measure the ability of different layers to distinguish between IND and OOD, specifically, by comparing the confidence (calculated by maximum softmax probabil-
10659
ity) or uncertainty (calculated by entropy) outputted by each layer on In- and Out-of-domain samples. Ideally, IND should be given higher confidence (or lower uncertainty) than OOD, and a larger difference or gap suggests a stronger ability to distinguish between them.
The results are shown in Figure 1. There are two types of curves in Figure 1, where the Pred Entropy (red) refers to the entropy (average value) of the sample output by the internal layers of the model, which is used to measure the certainty in its output. The curve of “Diff” shows the trend of the variation in the difference between the average values of the confidence (blue, and the orange refers to entropy) of all IND samples and OOD samples as the number of layers increases. We observe that with the increase of the number of layers, the model becomes more “confident” (lower entropy in output), but the ability to distinguish IND and OOD decreases (the difference is getting smaller) after reaching a certain peak.
The above experiments show that in the openworld scenario, PTMs also would be “overthinking” for the difference between IND and OOD. The “overthinking” behavior of the model in different scenarios also inspires us it is unnecessary to forward the last layer to distinguish IND and OOD or identify a specific class of IND during inference. To this end, we design an inference strategy to let the passed ICs dynamic ensemble and let these ICs vote to determine whether the sample is IND or OOD and whether they have enough confidence to exit early (Section 3.2). Furthermore, we introduce a training method that can adapt to the accurate and accelerated behavior in the inference stage. On the one hand, according to the theory of ensemble methods, we can improve the accuracy of IND recognition of the ensembled ICs by reducing the redundancy between ICs to increase the diversity among ICs during the training process. On the other hand, we can reduce the open space risk by regulating the recognition behavior of the adjacent space of each training sample during the training process to improve the OOD detection capability.
In addition to the theoretical explanation of the reason for introducing the training method, we also carry out detailed verification on three challenging intent datasets. The experiment shows that our method can not only substantially improve the accuracy, but also effectively accelerate inference. We summarize our key insights and contributions
as follows: Firstly, we explore the “overthinking” phenomenon of the model in the open-world scenario and its impact on the discrimination between IND and OOD. We hope that these explorations can inspire follow-up studies. Secondly, we design a simple but effective inference strategy to improve the model recognition ability and speed up the model inference. At the same time, we also introduce a novel training method to adapt the inference strategy. Thirdly, detailed experiments show that compared with the existing methods, our proposed method can substantially improve the accuracy but also effectively accelerate inference.

heading: 2 Related Work
text: Our work is related to two types of research: OOD Intent Classification and Early Exiting. OOD Intent Classification is also known as OOD detection. Such research can be roughly divided into two categories according to whether additional OOD samples are introduced (or synthesized) in the training process: supervised (The training set contains additional OOD samples) and unsupervised (there are no additional OOD samples in the training set). Since supervised methods require a large number of time-consuming OOD samples and the constructed OOD samples cannot cover all OOD classes, this work focuses on unsupervised methods.
Scheirer et al. (2012) articulate open space risk as the optimization objective of the task through establishing a constrained optimization. OpenMax (Bendale and Boult, 2016) models the output of the penultimate layer of the network as a specific distribution (Weibull Distribution) for OOD detection. MSP (Hendrycks and Gimpel, 2017) uses the confidence (maximum softmax probability) of the network to distinguish IND and OOD. DOC (Shu et al., 2017) connects N binary classifiers at the end of the network for OOD classification. LCML (Lin and Xu, 2019) introduces Margin loss to learn more discriminative semantic features to facilitate OOD detection. ADB (Zhang et al., 2021b) reduces open space risk by introducing adaptive boundaries for each IND class. Lin et al. (2021) extract image features by virtue of projecting the complexity of the images onto the network layers to construct an OOD detector in the domain of computer vision. Xu et al. (2021) use all the intermediate representations of pre-trained transformers to build an OOD detector. SCL (Zeng et al., 2021) learns
to adapt to downstream detection features by contrastive learning paradigms. KNN-CL (Zhou et al., 2020) further improves the learned semantic features by constraining positive and negative samples in contrastive learning. Different from the existing work, our method can dynamically decide whether to exit early without going through the whole model during inference, which speeds up the speed of inference and effectively guarantees the accuracy of recognition.
Early Exiting It is considered an effective way to accelerate the speed of inference of large pretrained language models (Xin et al., 2020; Zhou et al., 2020; Liu et al., 2020; Sun et al., 2021). The mechanism allows samples to exit from some internal layers of the model in advance to accelerate the inference process of the model without losing accuracy as much as possible. However, the existing research on Early Exiting is based on the closed-world scenario. In this work, based on further elaborating the manifestations of the “overthinking” phenomenon in the open-world scenario, we introduce the Early Exiting mechanism into the open-world scenario to help improve the model’s ability to distinguish IND and OOD while accelerating the inference of the model.

heading: 3 Proposed Method

heading: 3.1 Training Objective
text: We attach an internal classifier to each layer of BERT to enable early exiting. Denote Zl as the output distribution generated by the Internal Classifier (IC) in the l-th layer (lowercase zl to represent the feature of a specific input sample at layer l), Z1:l as the joint output of internal classifier from the first layer to the l-th layer, φ as Inference (ensemble) strategy. Training ICs as an Ensemble Before that, we need to solve a concern. Since the training set only has data with IND samples, will be trained on this training set to lead to the model overfit to the training data and damage the detection of OOD? Fortunately, Vaze et al. (2021) show that it will not damage or even improve the recognition of IND, and even help detect OOD.
Therefore, to accelerate inference without losing accuracy, we only need to ensure the decision made by the ensemble constructed by the passed l ICs can be accurate enough in IND samples, i.e., our objective is to optimize: min p(φ(Z1:l) ̸= Y ), Y is the ground truth label. Based on previous stud-
ies (Fano, 1961; Hellman and Raviv, 1970), the decision error has the following upper bound:
p(φ(Z1:l) ̸= Y ) ≤ Ent(Y )− I(Z1:l;Y )
2 , (1)
where Ent(Y ) is the entropy of random variable Y. I(Z1:l;Y ) indicates mutual information between Z1:l and Y . According to the upper bound (1), we can optimize decision error by maximizing I . However, due to the exponential combinatorial property of possible value Z and Y , it is extremely difficult to calculate I(Z1:l;Y ) directly. We decompose I(Z1:l;Y ) to obtain a computable lower bound as suggested in Zhou and Li (2010); Sun et al. (2021):
I(Z1:l;Y ) ≥ l∑ i=1 I(Zi;Y )
︸ ︷︷ ︸ Lrel
− l∑
i=2
I(Zi;Z1:i−1)
︸ ︷︷ ︸ Lred
, (2)
where ∑l
i=1 I(Zi;Y ) represents the sum of mutual information between Zi and Y , referred to relevancy, which is utilized to establish the bound of the internal classifier. ∑l i=2 I(Zi;Z1:i−1) is defined as redundancy, assessing the interdependence among classifiers. A diminished value indicates an elevated scope of dissimilitude among classifiers.
Following Sun et al. (2021), relevancy(Lrel) and redundancy(Lred) can be defined as −Lce(zi, y) and −minj<i Lce(zi, zj) respectively. Therefore, the objective to be optimized as follows:
Lz = −(Lrel − Lred) (3)
= α L∑
i=1
Lce(zi, y)− β L∑
i=2
min j:j<i
Lce(zi, zj),
(4)
where Lce is standard cross-entropy loss. zi is the internal representation of the input in layer i, and y is its real label. The α and β represent hyperparameters for optimization. Reduce Open Space Risk There is no prior knowledge about OOD as mentioned earlier, we cannot directly optimize the objective of OOD. Previous research (Scheirer et al., 2012; Zhou et al., 2022) has proved that the ability of the model to detect OOD can be also improved indirectly by reducing the open space risk. Therefore, in addition to the above effective training paradigms, we also propose a way to reduce open space risk. Following Scheirer et al. (2012); Bendale and Boult (2016), open space risk can be defined as:
RO(f) = ∫ O f(x)dx∫ S f(x)dx , (5)
where O is the open space, f is a measurable function that f(x) = 1 (or f(x) > 1) for IND intents, otherwise f(x) = 0 (or f(x) <= 0), S is whole semantic space containing the open space O.
Using training samples (IND samples) X , open space (Bendale and Boult, 2016) can be further defined as O = S−⋃z∈X σ(z), σ(z) is the minimum semantic space around the feature z.
RO(f)(↓) = ∫ O f(x)dx∫ S f(x)dx = ∫ S−⋃σ(z) f(x)dx∫ S f(x)dx
= 1− ∫ ⋃
σ(z) f(x)dx∫ S f(x)dx
= 1− X∑
z
∫ σ(z) f(x)dx(↑)∫
S f(x)dx (6)
According to Eq.(6), to reduce the open space risk, only need to increase ∫ σ(z) f(x)dx. Intuitively, according to the definition of f , we only need to make the semantic space around the input feature z be recognized as IND as far as possible. However, even in the infinitesimal space enclosing z, there exist a boundless number of sample points. As such, we select representative sample z̃ that are hard to be differentiated by the classifier to make their labels consistent with the label of z (i.e., recognized as IND). Inspired by adversarial examples (Zhu et al., 2020), z̃ is satisfied the following requirements:
z̃ = z + τ∗, τ∗ = argmax ∥τ∥≤ε Lz+τ . (7)
Final finetune Objective By bringing z̃ into Eq.(4),
we can get the loss Lz̃:
Lz̃ = α̃ L∑
i=1
Lce(z̃i, y)− β̃ L∑
i=2
min j<i
Lce(z̃i, z̃j).
(8)
The final finetune objective as follows:
Lobj = Lz + Lz̃. (9)
An overview of the whole training process can refer to Figure 2(a).

heading: 3.2 Inference Strategy
text: During inference, each internal classifier through which the input sample xi passed will give a corresponding prediction, the prediction of the l-th internal classifier Ŷl as follows:
Ŷl = { OOD, G(zli) < θl, argmaxk∈[K]Hk(zli), G(zli) ≥ θl. , (10)
where G is the scoring function (LOF (Breunig et al., 2000) is used in this paper) whose value is used to distinguish IND (>= θl) and IND (< θl). zli is the feature of xi as layer l. H is softmax function. K is the number of IND intents.
The final decision is determined by the joint vote of all the internal classifiers which the sample has passed through. Only when a certain class (OOD or specific IND class) reaches or exceeds the preset threshold, the model will specify the class as the final result and early exit. At l-th layer, the inference strategy generalizing Sun et al. (2021) is
as follows:
φ(Ŷ1:l) = maxk∈H
{∑l j=1 I(Ŷj = yk) }
lγ , (11)
where H is the set including IND categories and OOD (H=[K] ∪ {OOD}), γ ∈ [0, 1) is a hyperparameter. When φ(Ŷ1:l) is greater than a certain threshold, the sample exits at the current l-th layer and is given an IND label (i.e., k) or recognized as OOD. See Figure 2(b) for the inference.

heading: 4 Experiments

heading: 4.1 Datasets
text: To verify the effectiveness and universality of our proposed method, we conducted exhaustive experiments on three widely used intent datasets. These datasets are summarized as follows: CLINC-FULL (Larson et al., 2019) is a very popular dataset, which encompasses a broad range of intents, totaling 150 across 10 domains. The entire dataset consists of 22500 in-domain samples and 1200 Out-of-domain samples. BANKING (Casanueva et al., 2020) is a kind of dataset about the banking business, with 77 categories. The data is characterized by the imbalance of samples in different categories. The training set, validation set, and test set contain 9003, 1000, and 3080 samples respectively. StackOverflow (Xu et al., 2015) is a dataset about programming languages released by Kaggle.com. The dataset is subdivided into 20 categories and has 20000 samples. The number of samples in the training set, validation set, and test set is 12000, 2000, and 6000 respectively. The detailed statistics of datasets are listed in Table 1.

heading: 4.2 Baselines
text: We compare with the following mainstream OOD intent classification methods: Softmax, MSP (Hendrycks and Gimpel, 2017), DOC (Shu et al., 2017), SEG (Yan et al., 2020), LMCL (Lin and Xu, 2019), OpenMax (Bendale and Boult, 2016), ADB (Zhang et al., 2021b), SCL (Zeng et al., 2021), KNN-CL (Zhou et al., 2022). The above baselines are described in detail in Related Work (Section 2).
For a fair comparison, BERT is the backbone network of all methods (The other backbone is also explored in Appendix C). We supplement the results of ADB (Zhang et al., 2021b), SCL(GDA+LOF) and KNN-CL (Zhou et al., 2022) based on their
released codes, which are considered the state-ofthe-art OOD classification methods at present, on three datasets. The other baselines adopt the results reported in respective papers.

heading: 4.3 Evaluation Metrics
text: We follow the metric adopted in previous work (Zhang et al., 2021b; Zeng et al., 2021; Zhou et al., 2022) and regard all OOD classes as a single rejected class. Accuracy and F1-score are calculated in the same way as Zeng et al. (2021); Zhou et al. (2022). F1-IND and F1-OOD represent macro F1-score values of IND and OOD classes respectively. To measure the overall effectiveness of the model, we also calculate the accuracy score and macro F1-score on all classes (including all IND and OOD classes) and denote them as ACC-ALL and F1-ALL respectively. For the efficiency savings of the inference, we follow the settings in the research literature about early exit (Xin et al., 2020; Zhou et al., 2020; Liu et al., 2020; Sun et al., 2021). The specific calculation is as follows:
Speedup = ∑L
l=1 L×Nl∑L l=1 l ×Nl , (12)
where L represents the number of layers, Nl is the number of samples which exit at layer l.

heading: 4.4 Experimental Setting
text: Similarly, following the previous work (Zhang et al., 2021b; Zhou et al., 2022), after splitting the dataset into the training set, validation set, and test set, we randomly retain 25%, 75% of the whole intent classes as IND classes and the remaining as OOD class. At the same, the samples with OOD class are excluded from the training set and preserved in the test set. All datasets undergo in the same processing to ensure the absence of OOD samples in the training set. For each configuration, we conduct at least three rounds of experiments with different random seeds and report the final average value.
We employ the widely adopted BERT model (bert-uncased, 12-layer transformer) provided by the Huggingface Transformers2. We finetune the model with the most commonly recommended hyperparameters. We adopt AdamW optimizer (Loshchilov and Hutter, 2019) and try learning rate in {1e-5, 2e-5, 5e-5, 5e-6}, the training
2https://github.com/huggingface/transformers
batch size in {16, 32}, the epochs 30 or 50. We fix the α and α̃ as 1.0, and fix the β and β̃ as 0.1. We choose LOF (density-based and independent of the assumption of any distribution) as our detection method. According to Zhou et al. (2022), there is no distance to calculate LOF (Breunig et al., 2000) that is more advantageous than others. In this paper, we choose Cosine distance to calculate LOF. We use the same method as suggested in Zhou et al. (2022) to select the threshold of LOF by validation set. In addition, we also conduct experiments on other detection methods, such as KNN (Sun et al., 2022), to verify the generality and effectiveness of our method. All experiments are conducted on the NVIDIA GeForce RTX 3090 Graphical Card with 24G graphical memory.

heading: 4.5 Main Results
text: Table 2 shows the comparison between our method and baselines on BANKING and StackOverflow datasets, while the comparison between our method and baselines on CLINC-FULL is shown in Table 3. We highlight the best of all methods in bold. On the whole, our method can effectively improve the speed of inference on the premise of ensuring the accuracy of IND and OOD recognition.
Firstly, we observe the results in Table 2. We notice that our method performs particularly well on the BANKING dataset. With a speed-up ratio of more than 2 times, our method can not only ensure the accuracy of IND recognition but also effectively detect OOD, which fully demonstrates our method is both fast and good. The same phenomenon occurs in the comparison of the Stackoverflow and CLINC-FULL (Table 3) datasets. Under different settings, we exceed the existing baselines in all indicators. At the same time, We also maintain excellent inference efficiency.
On closer observation, we find that when the acceleration significantly increases, the effect also significantly improves, such as in BANKING and Stackoverflow(75%) datasets, and when the acceleration slows down, the effect also slows down, such as in CLINC-FULL(25%). This seems to
show that the later the decision of the model is made, the closer the effect will be to the last level. This also verifies the necessity of introducing dynamic decision-making. The model itself can make decisions independently without needing to rely on the last layer to make decisions. We also conduct experiments with other backbone networks, see Appendix C for results.

heading: 5 Analysis

heading: 5.1 A Closer Look at Internal Layers
text: Performance of Different Layers To better and more effectively show the effect of our method in a more fine-grained way, we have shown the performance of each layer of the model and the result of our method as shown in Figure 3(a)(b). First of all, we observe a more general form of “overthinking”. After an internal layer of the model reaches the best performance, the performance begins to fluctuate, or even begins to decline(Figure 3(a)). At the
same time, the early exit mechanism introduced in our method (orange) can effectively alleviate “overthinking” and ensure accuracy. Distribution of Exiting Samples Figure 3(c)(d) further shows the distribution of exiting samples in detail. We can observe that most samples can exit in advance and maintain high accuracy. Figure 3(c)(d) further verify the effectiveness of our method, which can not only make samples exit early but also ensure the accuracy of intent recognition.

heading: 5.2 Importance of Training Methods
text: In this section, we explore the effectiveness of our training strategy. We compare different training methods with our method on the premise of keeping the inference strategy unchanged. Average (AVG.) refers to directly adding the losses of the internal classifiers as the training objective. Joint follows the training method in Zhou et al. (2020) and adds a weight coefficient to the loss of each layer. Ensemble is proposed by Sun et al. (2021) and divides the training loss into two parts, taking into account not only the relevance but also the diversity between different layers. See Appendix A
for specific expressions of different training objectives. Our training methods are shown in Section 3.1. The comparison results are presented in Table 4. It can be seen from the table that our method can better adapt to the strategy adopted in inference. The comparison with Ensemble also shows the importance of reducing open space risk (Section 3.1) in our method. Detailed performance and discussion are the available in Appendix A.

heading: 5.3 Effect of Inference Strategy
text: In this section, we try to shed light on the effectiveness of our dynamic ensemble inference strategy. Under the same training strategy, we compare our strategy with other different inference strategies. Random means to the sample randomly early exits from a layer (the selection of layers follows uniform distribution). Concat. refers to concatenating the output representations of the internal
layers, and takes the concatenated representation as the final representation. This approach is based on the previous research (Clark et al., 2019) that different layers of the model can capture the semantics of different levels of samples, and the fusion of different semantic representations may lead to better representation. Pabee is a widely used inference strategy proposed by Zhou et al. (2020). Refer to Appendix B for more details on strategies. The comparison results are presented in Table 5, from which it can be concluded that our inference strategy can achieve better results on each dataset. Detailed performance and discussion are available in Appendix B.

heading: 5.4 Compatible with Other Detection methods?
text: In the above experiments and analysis, we have been using LOF (Breunig et al., 2000) as the OOD detection method (also known as a scoring function) and proved the effectiveness of our method based on it. To verify the generality of our proposed method proposed, i.e., whether it is compatible with other detection methods, we try another widely used distance-based OOD detection method–KNN (see (Sun et al., 2022) for details). After replacing it with KNN, we compare it with KNN-CL. As shown in Figure 4, our method also achieve better results in different datasets, which further proves the generality of our method.

heading: 6 Conclusion
text: In this paper, we explore whether the model would be overthinking in the open-world scenario and
demonstrate how to affect the discrimination between IND and OOD. On this basis, we propose a two-birds-one-stone method, i.e., during inference, let the model independently choose whether to exit early without going to the last layer, which not only ensures the accuracy of recognition but also accelerates the speed of inference. At the same time, we also introduce a training method that can adapt to the dynamic inference of the model. Detailed experiments and analysis show that our method can not only accelerate inference but also establish substantial improvements.
Limitations
To further inspire the follow-up work, we summarize our limitations as follows: 1) We only preliminarily reveal the overthinking phenomenon in the open-world scenario, and explore how to mitigate and utilize it during inference. We do not continue to conduct more in-depth research on the broader forms of overthinking in the open-world scenario and do not explore whether there are differences in its performance in different models. In addition, whether it can be solved or alleviated by other ways, such as training methods. 2) From the results of Sections 5.2, 5.3 and the corresponding Appendix A, B, it seems that there is room for further improvement in the speedup of our method. We leave how to achieve the best accuracy-speed trade-offs to subsequent research. 3) We have preliminarily verified that our method can be compatible with more detection algorithms and models, and look forward to exploring more methods and models.

heading: Acknowledgements
text: This work was supported by the National Key Research and Development Program of China (No.2022CSJGG0801) and National Natural Science Foundation of China (No.62022027).

heading: A More Results on the Comparison of Training Methods
text: In this section, we will show more detailed results of the comparison of different training strategies. First, we list the detailed training objectives in Table 6. Then, In Table 7, Table 8 and Table 9, we show the detailed comparison results of our methods and different training strategies on CLINC, BANKING and StackOverflow datasets. From the above comparison, we can find that our training strategy is not only effective but also general, which achieves better results in all datasets. From the comparison, our method seems to have room for improvement in the acceleration of inference, and we leave how to achieve better accuracy-speed trade-offs to subsequent research.

heading: B More Results on the Comparison of Inference Strategy
text: In this section, we will show more detailed results of the comparison of different inference strategies. First, Let us review these strategies again. Random means to the sample randomly early exits from a layer, i.e., randomly select a layer of BERT according to the uniform distribution, and then the sample early exits from this layer. Concat. refers to concatenating the output representations of the internal layers and takes the concatenated representation as the final representation. This approach is based on the previous research (Clark et al., 2019) that different layers of the model can capture the semantics of different levels of samples, and the fusion of different semantic representations may lead to better representation. Pabee is a widely used inference strategy proposed by Zhou et al. (2020) and makes a decision according to whether the predictions of the just passed continuous k (called patience) classifiers (not all classifiers) are consistent, see Zhou et al. (2020) for details. Then, In Table 10, Table 11 and Table 12, we show the detailed comparison results of our methods and different inference strategies on BANKING, CLINC, and StackOverflow datasets.
From the above comparison in tables, we can find that our inference strategy can achieve better results in all datasets. The speed-up ratio of Random is fixed around the 1.84× because the average number of exit layers of each sample is fixed equal to 6.5 in Random strategy so the speed-up ratio is about 12/6.5 ≈ 1.846. Concat. requires the output representation of all layers of the model, so it does not speed up (1.00×). At the same time, we find that the Pabee strategy can also achieve good results (compared with KNN-CL in the Table 2 and Table 3), which also verifies the generality of the overtaking phenomenon and the rationality of making a decision early, which is in line with our expectations. We also observe that our method seems to have room for the acceleration of inference, and we leave how to achieve the best accuracy-speed trade-offs to subsequent research.

heading: C Compatible with Other Backbone?
text: In Section 5.4, we have verified that our method could be compatible with other detection methods. In this section, we explore whether our method can adapt to other Backbones. Taking ALBERT (Lan et al., 2020) as an archetype, we compare our
method with the comparable baseline methods. The comparison results are shown in Table 13, which demonstrate that our method can also obtain im-
provements while accelerating inference. We establish preliminary exploration and left research on other models for future work.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations" (7th Section)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section "Limitations" (7th Section)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: "Abstract" and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? These datasets are available for all researchers in the NLP community.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? These datasets are only for scientific research and are available for all members of the NLP research community. We have adhered to the typical method of utilizing these resources.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These datasets are only for scientific research and are available for all members of the NLP research community. We have adhered to the typical method of utilizing these resources.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4 and Section 4.5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We establish preliminary exploration and left research on other models for future work. ACL 2023 Responsible NLP Checklist