Accelerating Transformers by Sparsifying Information Flows (2024)

Abstract: Transformers have become the de-facto standard for natural language processing. However, dense information flows within transformers pose significant challenges for real-time and resource-constrained devices, as computational complexity grows quadratically with sequence length. To counteract such dense information flows, we propose SPARSEFLOW, a novel efficient method designed to sparsify the dense pathways of token representations across all transformer blocks. To this end, SPARSEFLOW parameterizes the information flows linking token representations to transformer blocks. These parameterized information flows are optimized to be sparse, allowing only the salient information to pass through into the blocks. To validate the efficacy of SPARSEFLOW, we conduct comprehensive experiments across diverse benchmarks (understanding and generation), scales (ranging from millions to billions), architectures (including encoders, decoders, and seq-to-seq models), and modalities (such as language-only and vision-language). The results convincingly demonstrate that sparsifying the dense information flows leads to substantial speedup gains without compromising task accuracy. For instance, SPARSEFLOW reduces computational costs by half on average, without a significant loss in accuracy1.

heading: 1 Introduction
text: Transformers (Vaswani et al., 2017) have brought about a paradigm shift in diverse research areas such as NLP (Brown et al., 2020; Chowdhery et al., 2022) and computer vision (Dosovitskiy et al., 2021; Liu et al., 2021). However, their undeniable effectiveness often comes with a non-negligible computational burden, scaling quadratically with the length of the input sequence. This bottleneck poses a critical challenge in realizing the full po-
1Our code is available at https://github.com/ yeachan-kr/sparseflow
tential of Transformers, particularly in resourceconstrained or real-time systems (Sun et al., 2020).
A primary contributor to this inefficiency is the dense information flows2 between all token representations, specifically, the pairwise computations between every pair of tokens. While these dense information flows are a source of strength in transformers, due to their minimal inductive bias, they often lead to redundancy in token representations (Ethayarajh, 2019), resulting in different tokens carrying similar information (Goyal et al., 2020). Such redundancy implies that the current pre-trained language models involve a large amount of expensive yet unnecessary computations. This observation underscores the potential benefits of enhancing efficiency by simplifying the complexity of the current dense information flows.
In this paper, we propose SPARSEFLOW, a novel efficient method designed to sparsify the current dense information flows across all transformer blocks. To this end, SPARSEFLOW parameterizes the information flows linking token representations to each transformer block. These parameters are subsequently optimized to be sparse such that only
2Following (Abnar and Zuidema, 2020), we represent a information flow as DAG (Directed Acyclic Graph) where nodes are input tokens and its representations and edges are the interaction between them (e.g., feed-forward or self-attention).
5937
the salient information flows are preserved. To effectively optimize the sparsified patterns for diverse inputs, we extend SPARSEFLOW through sparsely activated mixture-of-experts (Fedus et al., 2022). By engaging each sample with the optimized sparse pathways, SPARSEFLOW streamlines the transformer inference while preserving the task accuracy, offering practical value across a broad spectrum of applications.
To confirm the efficacy of SPARSEFLOW, we conduct extensive experiments, encompassing diverse tasks (natural language understanding and generation), scales (from millions to billions), architectures (including encoder, decoder, and sequence-to-sequence models), and modalities (language and vision). We further perform a thorough analysis to gain insights into the behavior of SPARSEFLOW by examining the information flow of task-relevant tokens and the sparsified patterns of this flow. In summary, the contributions of this paper include the following:
• We propose SPARSEFLOW, a novel efficient method that learns to sparsify dense information flows, thereby enhancing the efficiency of pre-trained language models.
• We demonstrate experimentally that SPARSEFLOW achieves substantial speedup gains while preserving the task accuracy in comparison to strong baselines.
• We confirm the general applicability of SPARSEFLOW on different modalities, architectures, and its scales, showcasing the practical usefulness in a wide range of applications.

heading: 2 Related Work
text: Representation Removal in Transformers Dense information flows in self-attention often lead to redundancy in token representations (Ethayarajh, 2019; Goyal et al., 2020). This facilitates numerous studies aimed at removing these redundant representations during a forward pass. The major strategy behind those studies is to reduce computations in transformer blocks depending on the input representations, and they can be categorized in two-fold: (i) depth reduction (also known as early-exit) (ii) width reduction (also known as token pruning).The former approach reduces computations by decreasing the number of computing layers (Zhou et al., 2020; Zhang et al., 2022), while the latter does so by shortening
the number of tokens forwarded into transformer blocks (Goyal et al., 2020; Kim et al., 2022; Guan et al., 2022; Kim et al., 2023). Recent works have more focused on token pruning approach as it can optimize the computational resources in a more fine-grained way (Kim et al., 2023). Given these categorizations, SPARSEFLOW aligns more closely with the token pruning approach.
Token Pruning The earliest token pruning approach is PoWER-BERT (Goyal et al., 2020) that maintains only the pre-determined ratio of tokens based on the attention weights while removing others. However, due to the re-training costs of PoWER-BERT for different environments, LAT (Kim and Cho, 2021) extends it to effectively fit diverse configuration of computational budgets without the retraining. Subsequently, LTP (Kim et al., 2022) removes the tokens by optimizing the trainable threshold for the attention values to identify which tokens are significant to tasks. Beyond manual analysis on attention weights, recent works move to automatic methods that learn to select tokens to be removed during training. For example, TR-BERT (Ye et al., 2021) and Transkimmer (Guan et al., 2022) have suggested token removal strategies that can be learned during training, by using reinforcement learning and re-parameterization, respectively. AdapLeR (Modarressi et al., 2022) and LoT(Kim et al., 2023) have proposed a saliencybased strategy that eliminates tokens by estimating the gradients of the token representations with respect to the predictions.
Compared to token pruning, SPARSEFLOW has distinct characteristics. The token pruning methods learn individual sparse patterns for each data. In contrast, SPARSEFLOW learns the generalized patterns across all data by directly parameterizing information flows. Furthermore, token pruning methods necessitate additional procedures (e.g., attention analysis) or specialized layers (e.g., token classifiers) to identify which tokens to remove. However, SPARSEFLOW eliminates these processes, as it pre-determines the sparsified pathways during the training phase, enabling streamlined inference. Lastly, while token pruning methods progressively discard token information from the input sequence, SPARSEFLOW maintains all token information but reduces connections in the flows. Such a difference renders SPARSEFLOW less susceptible to the information loss commonly associated with token pruning methods (Zhong et al., 2023).

heading: 3 SPARSEFLOW
text: In this section, we elaborate on SPARSEFLOW, which is designed to sparsify the dense information flow in current pre-trained language models. We first revisit the dense and sparse information flows in transformer (§3.1). To construct the sparse information flows, we directly parameterize the information flow and introduce a binarization method to sparsify these parameterized flows (§3.2). To cover the diverse inputs, we extend SPARSEFLOW through adaptively optimizing multiple sets of parameterized information flows (§3.3). These parameters are lastly optimized to be sparse (§3.4), allowing only the salient flows to be preserved.

heading: 3.1 Information Flow in Transformer
text: Dense Information Flow. We start by revisiting the dense information flows in the transformer. Due to the parallel computation over the sequence, the information flows can be decomposed by its sequence index (i.e., token position). Let the token representations of the input x in the l-th transformer block be denoted as X(l), l ∈ [1, L] where L is the number of layer, and its representation at i-th position be denoted as X(l)i , the information flow at the i-th position starts by passing through the selfattention layer as follows:
X(l)i = softmax
( qiK⊤√
dk
) V, (1)
where qi = X (l) i Wq, K = X (l)Wk, and V = X(l)Wv are the query vector of the i-th position, key, and value matrices derived from the input tokens, and dk is the dimensionality of the keys. Subsequently, the updated representation is then forwarded to the following feed-forward networks3.
X(l+1)i = σFF(X (l) i WF1)WF2 , (2)
where WF1 ,WF2 are learnable weight matrices, and σFF(·) denotes an activation function of the feed-forward layers. The above information flows occurs at every token position, and it results in the redundancy with quadratic computational costs with the sequence length (Goyal et al., 2020).
3For simplicity, we denote the process of information flows only within self-attention and feedforward layers. In practice, the information flows include processes such as layer normalization and skip connections.
Sparse Information Flow. To counteract such dense information flows, we propose SPARSEFLOW that sparsifies the dense information flows to be sparse by reducing the number of positions used for computations (i.e., Eq. (1) and Eq. (2)). Let the reduced subset of the information flows in the l-th block of the transformer be denoted as P(l), the sparse information flow is defined through the computation over this subset:
X(l) ← { softmax ( qiK⊤√
dk
) V | i ∈ P(l) } , (3)
Similarly, the resulting representations of the subsets are forwarded to the subsequent feed-forward layers as follows:
X(l+1) ← { σ(X(l)i WF1)WF2 | i ∈ P(l) } , (4)
By including only a subset P(l) forwarded to each transformer block, SPARSEFLOW reduces the computation and memory requirement. Specifically, let the size of the subset P(l) is m≪ n where n is the length of the input sequence, the self-attention complexity is reduced from O(n2d) to O(mnd), and the complexity of the position-wise feed-forward is reduced from O(nd2) to O(md2).

heading: 3.2 Parameterized Information Flows
text: Given the definition of SPARSEFLOW, its efficacy hinges on how we optimize the subset P(l). Specifically, the subset should remain sufficiently small to enhance efficiency without compromising task accuracy. To achieve this, we directly parameterize this subset to control the complexity of information flows. Let the maximum length of transformers be denoted as Nmax, the subset of information flows in the l-th block is parameterized as p(l) ∈ RNmax = [p(l)1 , p (l) 2 , . . . , p (l) Nmax ] where the value of p(l)i is a scalar indicator of the information flow. To represent existence in the subset, the parameters are constrained to have binary values (i.e., 1 for the utilized information flows, 0 for the skipped ones). These parameters are then multiplied to token representations to bypass the computation in each block, i.e., p(l) ⊙ X(l).
However, optimization with the binary constraints is non-trivial as the binarization function is not differentiable. Inspired by the continuous relaxation trick for the discrete function (Jang et al., 2017), we approximate the binary constraints through the Gumbel trick with sigmoid function,
i.e., Gumbel-sigmoid. Specifically, as the sigmoid function can be viewed as a 2-class softmax where logits are learnable parameter p(l)i and the fixed value of 0, the Gumbel-sigmoid is calculated as follows:
p (l) i =
exp((p (l) i + g ′)/τ)
exp((p (l) i + g
′)/τ) + exp(g′′/τ) (5)
= σ((p (l) i + g ′ − g′′)/τ), where σ(·) denotes the sigmoid function, g′ and g′′ are independent samples drawn from a standard Gumbel distribution4, and τ is a temperature parameter controlling the steepness of the sigmoid function. As the temperature τ approaches zero, the sigmoid function becomes more like a step function, enabling the optimization of the subset parameters using gradient-based methods.

heading: 3.3 Mixture of Sparse Information Flows
text: The parameterized information flows are optimized to generalize across all training samples. However, employing a single sparse pattern to cover all data can be sub-optimal, as different inputs may require different patterns. Inspired by the concept of sparsely activated mixture-of-experts (MoE) (Fedus et al., 2022), we extend SPARSEFLOW to a dynamic method where multiple sparse patterns are selectively activated for different inputs.
To this end, we first duplicate the subset parameters, ensuring that p(l) ∈ RK×Nmax where K is the number of sets of parameterized information flows (i.e., the number of experts in MoE). We then introduce a learnable router w(·), which consists of a single feed-forward layer. The router takes the average of input embeddings X(0) over the sequence as input and produces the routing probability to select the subset parameters, as follows:
w(x) = softmax(W⊤X(0)) (6)
where W ∈ Rd×K is a learnable weights. With the routing probability, the selected subset parameter is derived through the soft weighted average of the subset parameters5:
p (l) i =
K∑
k=1
w(x)k · p(l)k,i (7)
4Gk = − log(− log(Uk)), Uk ∼ U(0, 1) with the uniform distribution U .
5While we adopt learning-based routing, there exists various alternatives, such as Top-k selection (Zhou et al., 2022) and Random selection (Wang et al., 2022). Moreover, semantically-driven routing can be a promising approach. We leave the exploration of these routing variants as future work.
where p(l)k,i indicates the i-th connection parameter of the l-th transformer block in the k-th pattern. These derived parameters are subsequently binarized using the Gumbel-sigmoid function (as in Eq (5)).

heading: 3.4 Information Flow Optimization
text: Sparse Regularization To inject the sparsity into the parameterized information flows, we regularize the subset parameters to be zero, thereby resulting in the sparse subset P . This is achieved by adding L2 regularization term to the loss function, defined as follows:
Lsparse = 1
L
L∑
l=1
( α(l) − 1
n
n∑
i=1
p (l) i
)2 , (8)
where α(l) is the target sparse ratio in the l-th block to prevent the model from overly removing the information flow, and λ is a hyper-parameter that controls the degree of sparsity. This regularization sparsifies the information flow until having the minimum size of subsets (i.e., α(l)) while learning the given task.
Overall Objective The overall loss function comprises a task-specific loss (such as cross-entropy for classification tasks) and sparsity regularization. Additionally, we add a consistency regularization loss to ensure predictions remain consistent before and after sparsification, as follows:
Lcons(x, y) = CE(p(y;x), p(y;xsparse)) (9)
where CE(·) denote the cross-entropy, p(y;x) and p(y;xsparse) are the prediction over classes produced by the model with the dense information flow and SPARSEFLOW, respectively. With this regularization, the total loss function for a given input x and its label y can be expressed as:
L(x, y; θ) = Ltask(x, y; θ) (10) + λsparseLsparse + λconsLcons(x, y),
where Ltask(x, y; θ) is the loss function for the primary task (such as classification or translation), λsparse and λcons are hyper-parameters that balance the importance of sparsity and consistency regularization, respectively. Through the above optimization procedures, SPARSEFLOW learns to use the most salient information flows for each transformer blocks, thereby leading to improved efficiency.

heading: 4 Experiments
text: In this section, we experimentally demonstrate the efficacy of the proposed method. Specifically, we answer the following three questions through extensive experiments and analysis:
Q1 (Efficiency) Does SPARSEFLOW offer better efficiency than token pruning methods across a wide range of tasks? (§4.2)
Q2 (Generality) Can SPARSEFLOW be generally applicable to larger scales, different architectures, and even other modalities? (§4.3)
Q3 (Insights) What flow patterns and behaviors are learned from SPARSEFLOW? (§4.4)

heading: 4.1 Experimental Setups
text: Baselines As SPARSEFLOW is closely related to the token pruning, we mainly compare our method with following token pruning methods with a backbone model (denoted as Baseline in the tables): LTP (Kim et al., 2022) which utilizes the attention maps to eliminate the tokens; Transkimmer (Guan et al., 2022) that removes tokens with the learnable token predictors; LoT (Kim et al., 2023) that learns to skip per-token computations through gradient-based routers. We follow the original setups for each baseline. As for SPARSEFLOW, the hyper-parameters and ablation results including the effects of the number of mixtures (K) are described in Appendix.
Tasks and Datasets To confirm the broad applicability of the methods, we evaluate each baseline on both natural language understanding (NLU) and natural language generation (NLG) tasks. For NLU tasks, we evaluate each baseline on five tasks, which are Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) for sentence classification,
OpenbookQA (Mihaylov et al., 2018) for multiplechoice question answering, Question-answering NLI (QNLI) (Rajpurkar et al., 2016) for natural language inference, Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005) for paraphrasing task, and Semantic Textual Similarity (STS-B) (Cer et al., 2017). For NLG tasks, we perform summarization tasks on SamSum (Gliwa et al., 2019) and XSum (Narayan et al., 2018).
Models For NLU task, we compare all baselines on the BERTLarge (Devlin et al., 2019) due to the suitability of the encoder model for understanding tasks. For the generation tasks, we utilize T5Small (Raffel et al., 2020) which consists of the encoder and decoder transformers. To confirm the scalability and general applicability, we additionally perform experiments on the larger models, which includes RoBERTaLarge (encoder, 337M parameters) (Liu et al., 2019), T5Large (encoder-decoder, 770M parameters), and GPT2XL (decoder, 1.5B parameters) (Radford et al., 2019). Note that, for models comprising a decoder transformer, such as T5 and GPT-2, SPARSEFLOW is applied to the context token processing components (i.e., the encoder in T5 and conditional token computation in GPT-2) because our primary focus is on the quadratic costs
associated with encoding. The exploration to the decoding phase is left as future work.
Evaluation To evaluate the efficiency of each baseline in maintaining task accuracy while reducing computational demand, in every experiment, we report the speedup gains achieved by each method up to the point where it retains 99% of the original accuracy for each task. Following the previous work (Guan et al., 2022; Kim et al., 2023), these speedup gains are calculated as the ratio of reduced FLOPs compared to the backbone model.

heading: 4.2 Main Results
text: Comparison to baselines Table 1 shows the overall comparison results. It is noticeable that SPARSEFLOW achieves substantial speedup gains better than other token pruning methods on NLU tasks (i,e., best speedups among four out of five datasets). This result demonstrates the efficacy in sparsifying the dense information flows. In addition, the evaluation result of the generation tasks in Table 2 presents that the strength of SPARSEFLOW can be generalized to generation tasks. Regarding the superiority to token pruning methods, we believe that a significant contributing factor lies in the preservation of all token information while decreasing computations, preventing the model from losing semantic information (Zhong et al., 2023). Furthermore, it is worth noting that the minimal overhead compared to other approaches also contributes to its superior efficiency. These overall results clearly verify our hypothesis that sparsifying information flow works quite well to improve the efficiency in transformers.
Trade-off between Accuracy and Speedup In Figure 2, we also analyze the trade-off between the task accuracy and computational costs while retaining 95% and 99% of original task accuracy. Our results suggest that SPARSEFLOW shows a superior performance over a wide range of speedup gains with a smoother decline. This result also supports the efficacy of our method, demonstrating that the proposed method better preserves the tasksignificant information than other token pruning methods.
Throughput Increase and Memory Reduction To confirm the actual speedup on a specific GPU hardware, we measure the throughput increase and memory reduction achieved by SPARSEFLOW. Ta-
ble 3 presents the improvement in each metric on an NVIDIA RTX 2080Ti. Our results provide that SPARSEFLOW indeed brings speedup gains in actual GPU setups while lowering the memory requirements, similar to the reduced ratio of FLOPs. Such reduction implies that the memory and computationally-constrained environments can indeed benefit from SPARSEFLOW.

heading: 4.3 General Applicability of SPARSEFLOW
text: Different Modality SPARSEFLOW can be easily applicable to any transformer-based models with minor modifications. We thus investigate whether the effectiveness of SPARSEFLOW can be generalized to the different modalities other than text. We specifically test a multimodal (visionand-language) transformer, as such multi-modality often necessitates longer sequence lengths to en-
code various types of inputs (e.g., image patches and textual tokens). For this purpose, we employ the Vision-and-Language Transformer (ViLT) as the backbone model. Evaluations are conducted on Visual Question Answering (VQAv2) (Goyal et al., 2017) and Natural Language for Visual Reasoning (NLVR2) (Suhr et al., 2019). VQAv2 is to predict the answers for the questions requiring an understanding of vision, language, and commonsense knowledge. NLVR2 is to predict whether the given sentence is true about two input images. To confirm the competitiveness of SPARSEFLOW, we compare ours with PuMer (Cao et al., 2023), a token pruning method specifically designed for vision and language tasks.
Table 3 shows the evaluation result on two tasks. Interestingly, without a modality-specific design, SPARSEFLOW demonstrates substantial speedup gains on both tasks, comparable to or even surpassing those of modality-specific methods. This result suggests that sparsifying dense information is a generally effective strategy for enhancing the efficiency of pre-trained transformers. In Section 4.4, we further explore how SPARSEFLOW works
differently on image and text representations.
Different Scales and Architectures As the size of pre-trained models keeps increasing recently, it is important to confirm whether the proposed method can be scaled to larger models. We therefore examine the scalability of SPARSEFLOW by applying ours to RoBERTaLarge (Liu et al., 2019), T5Large (Raffel et al., 2020), and GPTXL (Radford et al., 2019). Table 4 provides the evaluation results on this setup. This shows that the performance trend in ther other architecture is similar to the main experiments, demonstrating the general applicability of SPARSEFLOW to larger and even different architectures.

heading: 4.4 Learned Patterns of SPARSEFLOW
text: Sparse Distribution SPARSEFLOW learns to remain crucial information flow throughout the training. To investigate whether the remaining information flows have interpretable patterns or not, we analyze the distribution of sparse information flows through the lens of token positions6 in Figure 4. In the language understanding tasks (i.e., SST-2 and QNLI), we observe that initial positions tend
6In Appendix, we include the analysis on the remaining information flows across different layers.
to be more pruned than the later positions. We believe the frequent optimization on initial positions contributes this result since the parameters of the latter positions are rarely optimized during training (More discussion can be found in the Limitation part). We also observe the distinct patterns from the vision-language tasks (i.e., VQAv2). It shows that image tokens (positions after 40) are more pruned than text tokens (positions before 40), indicating the significant of textual tokens to solve the given tasks or the potential existence of short-cut in textual inputs. As for the pruned positions, we also find that the pruned image tokens are noticeably located in the near start and end positions which are corresponding to the edge of an image. This suggests that SPARSEFLOW can learn the different behavior depending on the modality and task.
Detailed examples of SPARSEFLOW To get a deeper understanding of the SPARSEFLOW’s behaviors, we analyze how the information of tasksignificant tokens are flowed through the remaining positions of the information flows. Here, we trace the similarity between the task-significant tokens and remaining tokens (i.e., remaining information flows) on the example of "the movie fails to live up to the sum of its parts" from SST-2 (sentiment analysis) where the task-significant word is fails. As token representations in transformers grow increasingly similar across layers, the degree of similarity between important tokens and others can indicate the effectiveness with which task-specific features are transferred throughout the network. Figure 5 shows the similarity changes compared to the base model. Interestingly, we observe that the similarity of the task-significant token (i.e., fails) to remaining tokens is higher than that of the base model. It implies that the the sparsified patterns (i.e., remaining tokens) can convey task-significant features well. In contrast, the similarity to the insignificant word (e.g., the) also support our findings by showing a similar or even lower similarity compared to the base model on the less significant word.
We also see the sparsified positions on specific images to see how the model differently performs on image modality. Here we sample the image from VQAv2, and Figure 6 shows the pruned positions (i.e., image patch) on the given image. Noticeably, in the lower layers, it is evident that the removed information flows are primarily found from the edge positions. In other words, the model learns to decide that tokens in center positions are crucial

heading: 1-3 Layers 7-9 Layers
text: to understand the image. This result shows that the model automatically learns center bias (Tseng et al., 2009) in image processing, and SPARSEFLOW is able to learn modality-specific information during the training.

heading: 5 Conclusion
text: In this paper, we have introduced SPARSEFLOW, a novel efficient method that sparsifies the dense information flow to improve efficiency in transformers. To this end, we parameterize the information flow of the pre-trained models and learn to sparsify the information flow such that only the necessary flows are preserved. We have performed extensive
experiments on diverse benchmarks, scaled models, and modalities to verify the efficacy and general applicability. Comprehensive result convincingly demonstrates that sparsifying the information flows brings significant speedup gains without compromising the task accuracy. Moreover, we have also shown that SPARSEFLOW can learn modalityspecific information automatically (e.g., center bias in an image processing).
Limitations
While SPARSEFLOW enables the streamlined inference of pre-trained language models, there exist potential limitations. First, since the sparsified patterns are pre-determined during the training, the effectiveness of SPARSEFLOW can be limited to the samples deviating from the training distributions, such as those longer than the maximum sequence. However, we believe that composing sufficiently diverse inputs can mitigate the limitation. Second, in this work, we mainly focus on the encoder part of the transformers and the context encoding part of the decoder model, which necessitate quadratic computations in the forward pass. Therefore, the efficiency of SPARSEFLOW during the decoding phase is yet to be verified. However, we believe that integrating SPARSEFLOW with KV-caching tricks effectively reduces the computation costs with memory requirements. We leave this direction of improvement as a promising avenue for future research.

heading: Acknowledgment
text: This work was supported by the Basic Research Program through the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2021R1A2C3010430) and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019II190079, Artificial Intelligence Graduate School Program (Korea University)).

heading: A Sparse Distribution of SPARSEFLOW
text: In Figure 7, we additionally analyze the number of remaining information flows across layers. We first notice that SPARSEFLOW learns to prune more information flows located in the deeper layers. This indicates the large redundancy in the deeper layer, and it is closely aligned with the previous finding that token representations get similar through multiple layers (Ethayarajh, 2019; Goyal et al., 2020).

heading: B Hyper-parameter Setup of
text: SPARSEFLOW
In our experiments, we control the sparsity of SPARSEFLOW through adjusting α in Eq. (8), and we find that α = 0.3 generally works well. For the number of sparsified patterns in MoE, we empirically set K to 3 as it generally performs well over the experiments.

heading: C Ablation on SPARSEFLOW
text: To confirm what components of SPARSEFLOW are crucial to achieve better efficiency, we perform an ablation study of individual components, which are
consistency regularization (Eq. (9)) and setting a target ratio (Eq. (8)) and different configurations of MoE. Table 5 shows the ablation results. Consistent with other experiments, we maintained the 99% original accuracy of each method to evaluate speedup gains. The findings indicate that omitting consistency regularization and the target ratio results in reduced speedup gains, thereby empirically justifying the importance of each component. In terms of MoE patterns, we observed that integrating MoE into SPARSEFLOW significantly enhances speedup gains. These results confirm the effectiveness of our proposed method in accommodating diverse inputs, which in turn leads to enhanced efficiency.

heading: D Additional Experiments with Large Models
text: we have performed additional experiments with larger models on other datasets (MRPC and STSB). Table 6 shows the evaluation results when applying LoT (Kim et al., 2023) and the proposed method to T5Large and GPT2XL. Similar to the experiments in others, SPARSEFLOW shows superior trade-off between accuracy and speedups compared to recent token pruning method, highlighting the scalability of the proposed method.

heading: Future Work
text: We leave this direction of improvement as a promising avenue for future research