PAD-Net: An Efficient Framework for Dynamic Networks
Year: 2023

Abstract:
Dynamic networks, e.g., Dynamic Convolution (DY-Conv) and the Mixture of Experts (MoE), have been extensively explored as they can considerably improve the model’s representation power with acceptable computational cost. The common practice in implementing dynamic networks is to convert the given static layers into fully dynamic ones where all parameters are dynamic (at least within a single layer) and vary with the input. However, such a fully dynamic setting may cause redundant parameters and high deployment costs, limiting the applicability of dynamic networks to a broader range of tasks and models. The main contributions of our work are challenging the basic commonsense in dynamic networks and proposing a partially dynamic network, namely PAD-Net, to transform the redundant dynamic parameters into static ones. Also, we further design Iterative Mode Partition to partition dynamic and static parameters efficiently. Our method is comprehensively supported by large-scale experiments with two typical advanced dynamic architectures, i.e., DYConv and MoE, on both image classification and GLUE benchmarks. Encouragingly, we surpass the fully dynamic networks by +0.7% top-1 acc with only 30% dynamic parameters for ResNet-50 and +1.9% average score in language understanding with only 50% dynamic parameters for BERT. Code will be released at: https://github.com/Shwai-He/PAD-Net.

heading: 1 Introduction
text: Deep neural networks have been continuously pushing the state-of-the-art performance in the tasks of computer vision (Girshick et al., 2014; Dosovitskiy et al., 2021) and natural language processing (Dai and Le, 2015; Brunet et al., 2019; Zan et al., 2022; Zhong et al., 2022b) in past years, at the cost of increasing training cost (Shen et al., 2023). However, most prevalent architectures perform infer-
∗Corresponding author
ence statically where both the computational graph and network parameters are fixed once after training, which limits the representation power. Dynamic networks (Han et al., 2021), as opposed to static ones, adapt their parameters or architectures to each specific input, improving the model representation power with acceptable computational cost, e.g., Switch Transformers (Fedus et al., 2021). The common practice of implementing dynamic networks is transforming static networks (or modules) with counterpart dynamic ones. For example, Dynamic Convolution (DY-Conv) (Chen et al., 2020b) replaces traditional convolution by adopting k adaptive convolutional kernels; Mixture of Experts (MoE) (Shazeer et al., 2017) replaces a single fully connected layer with multiple feed-forward neural networks in a parallel manner.
The success of dynamic networks motivates practitioners to design dynamic networks (Fedus et al., 2021; Li et al., 2021a), which often follow a fully dynamic approach, where all parameters are dynamic (at least within a single layer) and change with the input. Previous works (Chen et al., 2020b) show that dynamic networks often outperform their static counterpart, and using more dynamic layers intriguingly leads to ever-increasing performance. For instance, dynamic convolution promotes the performance on the ImageNet when more static convolution layers turn into dynamic ones. However, these prior works do not explain the need for a fully dynamic mechanism, and it remains unclear whether to convert static parameters into dynamic and to what extent if yes.
On the other hand, such a fully dynamic manner is resource expensive and may cause redundancy, limiting the applicability of dynamic networks. For instance, the total parameters of BERTBase equipped with MoE are ~506.3M (with 8 experts) compared to only ~108.9M for vanilla BERT-Base. In addition, an MoE layer also multiplies the computation. It seems more is better
14354
when transforming static layers into dynamic ones, but how about the dynamic parameters within a dynamic network: Do all of them lead to the promotion? This urges us to reflect whether there exist redundant dynamic parameters, in fully dynamic network layers? Based on the above scrutinization, we hypothesize that less is more for dynamic parameters in fully dynamic networks.
Motivated by this hypothesis, we propose the Iterative Mode Partition (IMP) algorithm to progressively convert less important dynamic parameters into static ones for higher efficiency, while maintaining performance at a competitive level. Given a fully dynamic network initialized with all parameters in dynamic mode, we attempt to partition a subset of static parameters out from them. Specifically, we iteratively transform dynamic parameters based on their impact on loss values. If the transformation of the i-th element of dynamic parameters results in only a minimal loss difference, we safely make it static. Given a desired dynamic ratio (the proportion of dynamic parameters), we can balance the trade-off between dynamic and static parameters. Since static parameters are less costly to deploy, we prune redundant parameters after mode partition, obtaining a lightweight architecture, namely Partially Dynamic Networks (PAD-Net), which contains two modes of parameters (dynamic parameters that vary with inputs and static parameters that are fixed during inference).
Empirically, we extensively validate this hypothesis and our proposed PAD-Net, including GLUE benchmark (Wang et al., 2019) for MoE and visual image classification (Deng et al., 2009) for dynamic convolution. Experiment results reveal that we successfully converted redundant dynamic parameters into static ones and PAD-Net achieves the highest performance in all tasks with lightweight architectures. Given the superiority of PAD-Net in both effectiveness and efficiency, we show that less dynamic is more efficient in fully dynamic networks, successfully verifying the above hypothesis. The inspiration of partially dynamic can be extended to other dynamic networks and even inform future efficient architectures designation.
In short, our contributions are threefold:
• We hypothesize that a fully dynamic network contains partially dynamic subnetworks that maintain or exceed the representation power of the original network.
• Following our hypothesis, we propose the
novel PAD-Net framework to achieve a partially dynamic mechanism and devise an Iterative Mode Partition (IMP) algorithm to partition static and dynamic parameters.
• We empirically validate our hypothesis and PAD-Net on both NLP and CV tasks across two representative dynamic networks, including MoE and dynamic convolution.

heading: 2 Related Work
text: Dynamic Networks. The dynamic neural network is an emerging research topic in deep learning, which adapts structures or parameters to different inputs, leading to notable advantages in terms of accuracy, and computational efficiency. Han et al. (2021) classify dynamic networks into two categories: dynamic architecture networks and dynamic parameter networks. Dynamic architecture networks adaptively adjust architectures conditioned on each sample. Specifically, they adjust the network depth (Wang et al., 2018), width (Mullapudi et al., 2018), or route based on the input (Huang et al., 2018). Instead of changing the model architecture, dynamic parameter networks boost representation power by adapting parameters or activation functions to the input (Yang et al., 2019; Liu et al., 2021). Existing works often transform various types of static parameters into dynamic versions (Chen et al., 2020b). Among them, dynamic convolution and mixture-of-experts are the typical examples that aggregate multiple convolution parameters (and experts) dynamically based on the input, leading to significant improvement with negligible computational cost.
Network Pruning. Past works in network pruning have explored effective techniques to find efficient subnetworks (Lee et al., 2019; Evci et al., 2020; He et al., 2022) and zero out redundant parameters. According to the lottery ticket hypothesis (LTH) pioneered by Frankle and Carbin (2019), dense, randomly initialized, feed-forward networks contain the subnetwork (winning tickets) that maintains comparable test performance of the original network after training for the same iterations. This hypothesis inspires a series of follow-up works in network pruning. However, these methods always sacrifice performance because of pruned parameters. As for dynamic networks, instead of directly pruning dynamic parameters, we considered changing them to static ones. In Section 5.4, we show
our approach significantly and consistently outperforms fully dynamic networks in the GLUE benchmark (Wang et al., 2019), while the pruned model performed worse than the original network.

heading: 3 Review of Fully Dynamic Networks
text: Basic Concept. Dynamic networks first adjust computational parameters and then compute the input using adjusted parameters, rather than directly using intrinsic parameters to compute the input. In a fully dynamic network, all intrinsic parameters are used as dynamic factors to generate computational parameters Θ̂, which are dependent on two parts: the input x and the intrinsic parameters Θ. Let us denote W as the dynamic function, computational parameters is formulated as Θ̂ = W(x,Θ). Given an input sample x, the output of is y = F(x,Θ) for a conventional network with static parameters and y = F(x, Θ̂) for a dynamic network.
Existing dynamic networks, though using different dynamic functions, tend to follow a fully dynamic manner: Networks take all intrinsic parameters to generate the computational parameters where all elements are dynamic and vary with the input. We call such networks fully dynamic networks and, in the following, introduce instantiations coming from dynamic architecture networks, i.e., Mixture of Experts, and dynamic parameter networks, i.e., Dynamic Convolution, respectively.
Mixture of Experts. We talk about dynamic architecture networks by taking the Mixture of Experts (MoE) (Jacobs et al., 1991; Shazeer et al., 2017) as an instantiation. MoE prepares m parallel static experts with parameters Θ(i)(i = 1, 2, . . . ,m) and only selects n experts with the highest scores (n ≤ m). Given a specific input, we denote G(x) as the output scores of gating and T as the indices of the selected experts. For the i-th selected expert, we denote the combination of the score GTi(x) and parameters Θ
(Ti) as w(Ti) = { GTi(x),Θ
(Ti)}. The dynamic function of MoE can be represented as:
W(x,Θ) = {w(T1), . . . , w(Tn)}, where w(Ti) = {GTi(x),Θ(Ti)}.
(1)
Dynamic Convolution. As a typical example of dynamic parameter networks, Dynamic Convolution (Chen et al., 2020b) prepares k parallel static kernels Θ(i)(i = 1, 2, . . . , k) as intrinsic parameters and utilizes the linear combination of them
as the aggregated kernel. The linear scale is aggregated dynamically via a channel-wise attention block (Hu et al., 2018) denoted as Attention, so the dynamic function can be written as:
W(x,Θ) = k∑
i=1
πi(x) ·Θ(i),
where π(x) = Attention(x).
(2)
Limitation Discussions. Mainstream dynamic networks usually replace static layers with fully dynamic layers, where all elements of dynamic parameters require corresponding dynamic factors co-working with input samples. However, this situation causes redundant parameters and high deployment costs, limiting the applicability of dynamic networks to a border range of resource-constrained situations and large-scale models. For this fully dynamic manner, we raise two questions: (1) Is it necessary to pay the cost of enormous parameters and computations to aggregate dynamic parameters? (2) Is it necessary to make all computational parameters dynamic, to maintain the performance improvement? We propose the Partially Dynamic Network (PAD-Net) that mixes dynamic and static parameters to answer the above questions.

heading: 4 Methodology

heading: 4.1 PAD-Net: Partially Dynamic Network
text: In response to the limitation of fully dynamic networks, we question whether it is necessary to make all parameters dynamic. To this end, we try to detect the less important dynamic parameters and transform them into input-agnostic static parameters. Specifically, we utilize a mask Mi(i = 1, 2, . . . ,m) to indicate whether the i-th element of Θ̂ is dynamic or static: Mi = 1 means the i-th element of Θ̂ is dynamic and vice versa. We use Θ̃ ∈ Rm to denote the dynamic parameters and Θ̄ ∈ Rm to represent the static parameters, then computational parameters Θ̂ are reformulated as:
Θ̂i = { Θ̃i =Wi(x,Θ) if Mi = 1 Θ̄i otherwise , (3)
where Θ̂i(i = 1, 2, . . . ,m) represents the i-th element of Θ̂, and Θ denotes the dynamic factors. In our architecture, intrinsic parameters include dynamic factors Θ and static parameters Θ̄. Note that M partitions the computational parameters into two non-overlapping parts, forming a network with
only a part of the parameters dynamic, i.e., Partially Dynamic Network (PAD-Net). Details of the procedure of generating the computational parameters from intrinsic are visualized in Figure 1.
To overcome the aforementioned challenges and limitations, we propose a novel network architecture, Partially Dynamic Network (PAD-Net). We also devise a new algorithm Iterative Mode Partition (IMP) to build this model efficiently.
In addition, we set two scale factors to describe the relative intensity of these subnetworks separately in terms of magnitude, namely λs and λd. With these scale factors, we factorize our method into a more general formulation:
Θ̂i = { λd · Θ̃i if Mi = 1 λs · Θ̄i otherwise , (4)
where we constrain λs + λd = 2(λs, λd > 0), and Equation 3 is the special situation when both λs and λd are equal to 1. Similar to the constraint ∑k i=1 πi in dynamic convolution (Chen et al., 2020b), this constraint compresses the parameters space and significantly simplifies the joint optimization of scale factors and the counterpart parameters.

heading: 4.2 Iterative Mode Partition
text: In the above section, we present the architecture of PAD-Net, which includes dynamic parameters and counterpart static parameters. Next, we further discuss our method in how to generate indicator masks to partition dynamic and static parameters. Let us first formulate this partition as an optimization problem, where our goal is to minimize loss values L. Given a dataset D = {(xi,yi)}ni=1 and
a desired dynamic ratio κ of M, we briefly formulate mode partition as the following constrained optimization problem:
min M L(Θ̂,M;D) = min M
1 n
n∑
i=1
ℓ(Θ̂,M; (xi,yi)),
s.t. M ∈ {0, 1}m, ∥M∥0 ≤ κ ·m, (5)
where ℓ(·) denotes the standard loss function (e.g., cross-entropy loss), Θ̂ is the set of computational parameters of the neural network, ∥ · ∥0 is the standard L0 norm, m is the total number of parameters. The conventional approach to optimize the above problem is adding sparsity enforcing penalty term M (Carreira-Perpinán and Idelbayev, 2018), while it often requires heavily tuned hyperparameter settings and several trials. On the other hand, LTH-based (Chen et al., 2020a; Evci et al., 2020) methods find the mask by several iterations, but it is prohibitively time-consuming. Also, considering the large-scale dynamic networks, it is unnecessary to deploy redundant parameters.
We tend to partition the two modes before training to prune redundant parameters and avoid timeconsuming training iterations. Inspired by Lee et al. (2019)’s gradient-based pruning strategy, we propose an algorithm to make excessive dynamic parameters static. We resort to mini-batches of training data Db = {(xi,yi)}bi=1 ∼ D to detect redundant dynamic parameters. Given a dynamic parameter Θ̂j at the j-th element of Θ̂, we compute its importance of being dynamic based on the loss difference ∆Lj caused by making Θ̂j static (changing the value of Mj from 1 to 0):
∆Lj(M, Θ̂;Db) = L(M, Θ̂;Db)−L(M− tj , Θ̂;Db), (6)
where tj is the indicator vector of j-th element of M (i.e., zeros everywhere except at the index j where it is one). We only consider transforming redundant dynamic parameters into static ones, so the loss difference ∆Lj is zero if Θ̂j is static. Note that computing ∆Lj for each dynamic parameter is prohibitively expensive, as it usually requires millions of forwarding passes over the dataset, so we resort to a simple and effective approximate alternative. Specifically, we release binary constraints of M and make it differentiable and utilize the derivative of L with respect to Mj to approximate ∆Lj :
∆Lj(M, Θ̂;Db) ≈ gj(Θ̂;Db) = ∂Lj(M, Θ̂;Db) ∂M
= lim δ→0 Lj(M, Θ̂;Db)− Lj(M− δtj , Θ̂;Db) δ ∣∣∣∣∣ t=1 , (7)
where gj(Θ̂;Db) denotes the j-th element in derivative g(Θ̂;Db). We accumulate the derivatives for all j by one forward-backward pass using automatic differentiation. Note that if the magnitude of gj is high, it essentially means that making parameter Θ̂j static has a considerable effect on the loss, and it has to be dynamic. In contrast, the parameter should be static if the magnitude of gj is low. Therefore, We take the normalized magnitude of the derivatives of g as the criteria:
sj = ∣∣∣gj(Θ̂;Db) ∣∣∣ /∑m
k=1
∣∣∣gk(Θ̂;Db) ∣∣∣. (8)
Given the dynamic ratio κ, we take the sκ (the κ-th percentile of s) as the threshold and transform the mask elements whose scores are below zero:
M = 1 [s− sκ ≥ 0] , (9)
where 1[·] is an element-wise indicator function where the output will be 1 if the condition [·] meets
else it will be zero. Note that the indicator mask M prunes out redundant parameters in dynamic parameters Θ̃ and static parameters Θ̄ respectively. Also, for fewer dynamic parameters to generate, we can also utilize the binary mask to prune redundant dynamic factors. Taking MoE as an example, M can be directly applied to parallel experts: Θ(i) ← M⊙Θ(i),∀i ∈ {1, 2, . . . , k}. In addition, we can decrease the computational cost of generating based on dynamic factors.
Inspired by the success of the iterative strategy in pruning at initialization (Verdenius et al., 2020; de Jorge et al., 2021), we start from a fully dynamic network and adopt an iterative strategy shown in Figure 2 to transform dynamic parameters into static parameters iteratively, where we increase the zero ratios of M exponentially. The effectiveness of the mode partition and the iterative mode partition is experimentally verified in Section 5.3.

heading: 5 Empirical Evaluation

heading: 5.1 Implementation Details
text: Mixture of Experts. We use Adam (Kingma and Ba, 2015) as the optimizer with β1, β2 = 0.9, 0.98. For regularization, we set the weight decay as 0.1 and grid-search the learning rate from {1e-5, 5e-5, 1e-4, 5e-4}, where we warm up the learning rate in the first 10% steps (of the total training steps). For different data scales, we grid-search training epoch and batch size from {5, 10, 15, 20} and {8, 16, 32, 64}, respectively. The maximum length is 128 for all tasks. Following Shazeer et al. (2017), we initialize dynamic and static parameters with pretrained parameters.
Dynamic Convolution. We use an SGD optimizer (Ruder, 2016) with 0.9 momentum, follow-
ing cosine learning rate scheduling and warmup strategy. The learning rate rises to the maximum linearly in the first ten epochs and schedules to arrive at zero within a single cosine cycle. We follow Chen et al. (2020b)’s temperature annealing strategy to avoid the unstable output values of the softmax function in the first epochs. We train ResNet for 100 epochs with the max learning rate of 0.1. We train the MobilenetV2 for 300 epochs with the max learning rate of 0.05. The weight decay is 1e-4 for ResNet and 4e-5 for MobilenetV2. The training batch size is 256 for all models.

heading: 5.2 Main Results
text: Natural Language Understanding. We evaluate the performance of PAD-Net for MoE on various datasets from the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019).
Like previous works (Lee et al., 2020; Dodge et al., 2020; Zhong et al., 2022a), we fine-tune pretrained models, e.g., BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), RoBERTa (Liu et al., 2019), ELECTRA (Clark et al., 2020) on the training set and directly report results on validation set using the last checkpoint, since the test results are only accessible by the leaderboard with submission limitation.
Following Shazeer et al. (2017); Gao et al. (2022), we replace feed-forward layers with MoE layers where we prepare 8 experts and select the top-2 experts for each input. We set the dynamic ratio κ = 50% because it is close to the optimal value. Table 1 shows that PAD-Net outperforms MoE on the GLUE benchmark with a 0.95% average increase for four backbones. Specifically, PAD-Net improves BERT by 1.9% and RoBERTa by 0.6%
on average. Equipped with PAD-Net, MoE reduces both parameters and computation significantly, and we provide a theoretical analysis of the reduced computation in Section 5.4.
Visual Image Classification. We also report the superiority of PAD-Net in visual image classification. In Table 2, we compare PAD-Net with static convolution (Krizhevsky et al., 2012), CondConv (Yang et al., 2019) and Dynamic Convolution (Chen et al., 2020b) on ImageNet (Deng et al., 2009) classification for ResNet (He et al., 2016) and MobileNetV2 (Sandler et al., 2018) in the same experimental setting with previous works, by adjusting all convolution layers except the first layer. Before training, we first partition two modes of parameters with a given dynamic ratio κ using ten batches of examples.
PAD-Net improves accuracy with significantly
lighter architecture and fewer FLOPs (Floating Point Operations). For instance, PAD-Net outperforms DY-Conv by 0.7% top-1 accuracy with 33.9% parameters and 0.1G fewer FLOPs in ResNet-50.

heading: 5.3 Ablation Study
text: Effect of Dynamic Ratio. Inspired by Wettig et al. (2022), we investigate the impact of different dynamic ratios κ, and the results are shown in Table 3 For MoE and Figure 3 for DY-Conv. Because PAD-Net with low dynamic ratios significantly outperforms fully dynamic networks, we only consider ratios of less than 70%, allowing for more sparsity and efficiency. We empirically find that κ = 50% is nearly the optimal ratio for MoE to achieve the highest performance, while the best performance of DY-Conv is achieved when κ = 30%. We believe that different dynamic functions contribute
to different optimal dynamic ratios, and an efficient way, e.g., hyper-parameter optimization or meta-learning, to search them will be necessarily explored in the future.
Effect of Scale Factors. We also conduct an ablation study on the proposed scale factors and verify their necessity. Table 4 summarizes the impact of scale factors on different architectures. We initially tried to gain scale factors from a SENet structure (Hu et al., 2018), while it did not contribute to the improvement of performance. So we just set scale factors as trainable parameters to avoid redundant parameters and operations. Besides the setting “λs+λd = 2” in Equation 4, we consider other situations: only using one factor (“λs” and “λd”) , and no scale factors used (“–”). We conduct experiments on CIFAR-10 (Krizhevsky, 2009) and ImageNet for ResNet-50, RTE, and STS-B for BERT. λs and λd enhance performance substantially, and their coexistence leads to further improvement. To explore the impact of the summation constraint, we release it and denote this setting as “λs, λd”. Clearly, without summation constraint, the performance of ResNet-50 and BERT decreases significantly, i.e., -0.4% and -0.35% on average.
Effectiveness of Iterative Mode Partition. We compare different partition strategies in Figure 4. Compared to fully dynamic networks, accuracy degrades when we partition two modes randomly, which means this naive partition method mistakes some important dynamic parameters. In contrast, mode partition contributes to a better combination of dynamic and static parameters, improving the accuracy. IMP shows its effectiveness by achieving the best performance.

heading: 5.4 Detailed Analysis
text: Reduced Computation. We show the computation cost of PAD-Net in Figure 5. Compared to vanilla MoE, PAD-Net reduces the computation between selected experts and the input, yTi = ETi(x), where ETi denotes the i-th selected experts. Because the two methods share the same gating mechanism, we temporally ignore its computation for simplicity. We denote the computation of the i-th expert as CTi where CT1 = · · · = CTn = c, and the total computation of multi-experts is nc if we select n experts within m ones. In PAD-Net, given the dynamic ratio κ, it is reduced to nκc. Together with the computation (1− κ)c, the computation of a PAD-Net layer is nκc+(1−κ)c. Integrated with PAD-Net, an MoE layer can reduce the computation by (n − 1)(1 − κ)c. When κ is low enough, the computation of PAD-Net can be close to static networks. For DY-Conv, the reduced computation lies in the linear combination of parallel kernels∑k
i=1 πi(x) ·Θ(i), which is sparse in PAD-Net. In short, the degree of reduced computation depends on the specific dynamic function.
Difference with Model Pruning. Mode partition maintains important dynamic parameters while making redundant ones static, which may be similar to network pruning. In Table 5, we compare mode partition with network pruning (Lee et al., 2019) on the GLUE benchmark for BERTbase and reveal their difference empirically. PADNet achieves the best performance among all tasks listed, with 1.2% average improvements over vanilla MoE. In contrast, we discover that network pruning lowers the performance of MoE significantly by 1.1% on average. Considering maintain-
ing the performance of a fully dynamic network, it is preferable to convert unimportant dynamic parameters into static ones than to prune them.
Dynamic Property. Dynamic property refers to variant numerical characteristics of a dynamic network caused by different input samples. The ideal dynamic network maintains two capacities: assigning specific parameters for the input and making counterpart output discriminative. Inspired by Li et al. (2021b), we take two levels of variance as metrics (parameter variance and output variance) to measure the dynamic property and show the result in Figure 6. Static convolution, dynamic convolution, and PAD-Net (κ = 30%) show different properties given the same samples from ImageNet. We see that dynamic convolution retains a high degree of parameter variance while it has the lowest output variance. Static convolution performs the opposite. The outputs of PAD-Net are discriminative, which may contribute to its superiority in performance.

heading: 6 Conclusion and Future Work
text: In this work, we first reveal parameter redundancy and high deployment costs of fully dynamic networks. To resolve these problems, we proposed the partially dynamic network (PAD-Net) to advance both performance and efficiency. PAD-Net
demonstrated its superiority on MoE and DY-Conv frameworks. Extensive experiments on both NLP and CV tasks empirically show its effectiveness and efficiency against fully dynamic networks, significantly improving performance with much fewer dynamic parameters and less computation. Our proposed method could be extensively integrated with other mainstream architectures and inspire future work in efficient neural network designation and other fields.

heading: Acknowledgements
text: We are grateful to the anonymous ACL reviewers and the area chair for their insightful comments and suggestions.

heading: 7 Limitations
text: Despite the progress we made, there still exist limitations in our work. On the one hand, we only investigated some classic dynamic networks and found that the proposed method contribute to the best performance in selected criteria. However, other advanced partition methods that further improve the performance and efficiency may exist, which deserve exploration in future work. On the other hand, since we only consider MoE and DY-Conv in limited tasks, it would be valuable to consider other architectures (e.g., Switch Transformer (Fedus et al., 2021)), machine learning methods (e.g., reinforcement learning (Li et al., 2022)) and tasks (e.g., machine translation (Ding et al., 2020, 2021)).
Ethics Statement
We take ethical considerations seriously and strictly adhere to the ACL Ethics Policy. This paper focuses on the higher efficiency of dynamic networks, e.g., the mixture of experts. Both the datasets and models used in this paper are publicly available and have been widely adopted by researchers. We ensure that the findings and conclusions of this paper are reported accurately and objectively.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Both the methods we cite and the improvement we make involve no potential risk.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
7 B1. Did you cite the creators of artifacts you used? Left blank.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments?

heading: 5.1 5.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5.1 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: However, other advanced partition methods that further improve the performance and efficiency may exist, which deserve exploration in future work. On the other hand, since we only consider MoE and DY-Conv in limited tasks, it would be valuable to consider other architectures (e. g. , Switch Transformer (Fedus et al. , 2021)), machine learning methods (e. g. , reinforcement learning (Li et al. , 2022)) and tasks (e. g. , machine translation (Ding et al. , 2020, 2021))