NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models
Authors: Jongwoo Ko
Seungjoon Park
Yujin Kim
Sumyeong Ahn
Du-Seong Chang
Euijai Ahn
Se-Young Yun
Venue: EMNLP 2023 Findings (2023)
Keywords: Model Compression
Structured Pruning
Encoder-Decoder Language Model
TL;DR: We conduct an empirical analysis to investigate the consideration for applying structured pruning to encoder-decoder language models and propose a new framework, NASH, considering the insight gained from the analysis.

Abstract: Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures such as Transformers. Despite the versatility of encoder-decoder models in numerous NLP tasks, the structured pruning methods on such models are relatively less explored compared to encoder-only models. In this study, we investigate the behavior of the structured pruning of the encoder-decoder models in the decoupled pruning perspective of the encoder and decoder component, respectively. Our findings highlight two insights: (1) the number of decoder layers is the dominant factor of inference speed, and (2) low sparsity in the pruned encoder network enhances generation quality. Motivated by these findings, we propose a simple and effective framework, NASH, that narrows the encoder and shortens the decoder networks of encoder-decoder models. Extensive experiments on diverse generation and inference tasks validate the effectiveness of our method in both speedup and output quality.

NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models
Jongwoo Ko Seungjoon Park2* Yujin Kim
Sumyeong Ahn
Du-Seong Chang² Euijai Ahn² Se-Young Yun
*KAIST AI ²T
Michigan State University
Abstract
Width pruning
Depth pruning
Structured pruning methods have proven effec- tive in reducing the model size and accelerating inference speed in various network architec- tures such as Transformers. Despite the versa- tility of encoder-decoder models in numerous NLP tasks, the structured pruning methods on such models are relatively less explored com- pared encoder-only models. In this study, we investigate the behavior of the structured pruning of the encoder-decoder models the decoupled pruning perspective of the encoder and decoder component, respectively. Our find- ings highlight two insights: (1) the number of decoder layers is the dominant factor of infer- ence speed, and (2) low sparsity in the pruned encoder network enhances generation quality. Motivated by these findings, we propose a sim- ple and effective framework, NASH, that narrows the encoder and shortens the decoder networks of encoder-decoder models. Extensive experi- ments on diverse generation and inference tasks validate the effectiveness of our method in both speedup and output quality.
Enc,
Dec,
Doc,
Enc.
Dex
Enc,
Deck
Decr-1
Deck-1
Ene;
Enc,
Det-
Enc,
Dec,
Dec,
Encoder
Shallow
Figure I: Brief illustration the proposed algorithm, NASH: NArrow encoder and SHallow decoder. is com- posed of two main components, width and depth pruning for encoder and decoder, respectively.
On the other perspective of LM researches rather than performances, efficiency of LMs co.pputtttaai and memory cost) have been intensively studied because of their huge com- putational requirements. This research direction is called model compression. Among the various model compression techniques (Jiao et al., 2020; Yao et al., 2022), pruning (Frankle and Carbin, 2018; Sanh et al., 2020; Wang et al., 2020c; Xia et al., 2022) promising method that aims to remove redundant weights from networks, result- ing in improved efficiency by saving storage ca- pacity and enhancing inference speed. Between structured pruning and unstructured pruning ap- proaches, structured pruning is typically preferred in practice due to its relative ease of deployment on various types of hardware platforms compared to unstructured pruning (Han al., 2016; Gupta and Agrawal, 2020).
1 Introduction
In recent years, ore-trained language models (LMs) have demonstrated their effectiveness in various downstream tasks, such as natural language un- derstanding (NLU) and natural language gener- ation (NLG). Especially, there have been three main types of research, e.g.eencderr-nll LMs (De- vlin et al., 2019; He et al., 2023), decoder-only LMs (Touvron al., 2023; OpenAI, 2023), and encoder-decoder LMs (Lewis et al., 2020; Raffel et al., 2020; Chung et al., 2022b; Tay et al., 2023), which aim for their specific expertise. Among these various types of LMs, we will focus on the widely studied and utilized encoder-decoder LMs due to their flexibility in application across range of tasks (Guo al., 2022: Wang al., 2023b).
Therefore, we focus on the structured pruning method specifically tailored for encoder-decoder LMs. Despite the remarkable advancements in encoder-decoder models, little attention has been given to structured pruning methods for encoder- decoder LMs. This can be attributed to the inherent
'denotes equal contribution. denotes equal advising. denotes working done at KAIST AI. Correspondence to Jongwoo Ko <jongwoo. ko@kaist... st. kr.>

---

differences in the components that enhance prun- ing efficiency between encoder and decoder net- works. Consequently, traditional structured prun- ing methods that rely on encoder-only models may not effectively optimize encoder-decoder models. For instance, CoFi (Xia et al., 2022), one of the SoTA encoder-only pruning methods, demonstrates a maximum speedup improvement of 1.53x on the CNN/DailyMaiil (See et al., 2017) dataset, with ROUGE-L drop of 7.36%. This gain is consider- ably lower compared to the original result achieved on the encoder-only model applicd to the worst QQP case in the GLUE (Wang et al., 2018), where the speedup reaches 11.0x with an accuracy drop of 1.20%. Thus, it becomes crucial to investigate structured pruning methods that are specifically tailored for the encoder and decoder networks.
2 Preliminary
Transformers.
We focus on the Transformer net- work (Vaswani al., 2017), which consists of the encoder and decoder architecture. The encoder ar- chitecture is composed of L blocks, and each block consists of a multi-head attention (MHA) layer and a feed-forward (FFN) layer. An MHA layer in the i-th Transformer layer with N₂. heads outputs:
(HA( (Q. K. V)- Au( KW VW
NA
() Q V)
MHA QQ K, VIW Σ
where Att represents a dot product attention head, and Q K, and V are the input sequences for query, key, and value, respectively. In self-attention layers, all of the keys, values, and queries come from the outputs of the previous layer. On the other hand, cross-attention layers, the queries come from the previous decoder layer, while the memory keys and values come from the output of the encoder. It is important to note that the j-th head parameterized by will.), wildd) wlf)), K
To this end, in this paper, we pose the following question: How can we design a structured pruning method that effectively accelerates the encoder- decoder model while maintaining its performance? To the best of our knowledge, this study represents the first attempt to address this question. In order to accomplish this, we conduct systematic studies examine the impact of structured pruning on the encoder and decoder networks, respectively.
and wilij) E Idxd,, which represent the query, key, value, and output matrices, respectively, Here, d and d, denote the hidden state dimension and attention head dimension, respectively.
Contribution. In this study, we propose an algo- rithm, NASH, which is strongly motivated by two findings derived from our preliminary experiments. (1) The number of decoder layers is the primary factor for inference speedup. (2) The sparsity of the encoder network is a key factor affecting the output quality of encoder-decoder LMs.
The output of the MHA layer, denoted as X, is then fed into the FFN layer in the i-th Transformer layer:
FFN ( (X) GELU(XW1)
Based on these findings, we propose an algo- rithm, illustrated in Figure 1, that consists of two parts: the encoder network, which enhances output quality by gradually reducing the width of each layer, and the decoder network, which achieves faster inference speed by uniformly selecting lay- ers to reduce depth.
Here, the two fully-connected layers are parameter- ized by W1 E Rdxdf and W₂ E of with dy representing the dimension of the FFN layer.
Structured Pruning. Structured pruning gradu- ally removes unnecessary parameters from a model, targeting width-related components (HA heads, FFN hidden units) and depth-related ele- ments (e.g.,Transs Transformer layers) during training, Re- cent advancements have demonstrated significant speedups with minimal output quality reduction. For example, block pruning (Lagunas al., 2021) and CoFi (Xia al., 2022) have enhanced flexibil- ity, optimization, and enabled simultancous prun- ing at multiple levels.
We empirically evaluate the performance of NASH on various NL. datasets including standard fine-tuning on single task (Gliwa et al., 2019; Xiong et al., 2019), multi-task learning scenarios, and recent instruction-tuninng datasets (Conover et al., 2023; Wang al., 2023a). Notably, in our ex- periments using T5-base, NASH achieves a speedup of 2.5-4.2x while preserving 95% of the output quality. Our experimental results show that NASH can be a unified framework which is regardless of task difficulty and model type.
Pruning the components of the i-th layer related

---

RTE (Mocel rize vs Accuracy)
RTE (Speecup Accuracy)
SAMSum (Model cize vs ROUGE-L
SAMSSm (Speedup ve ROUGE-L
80
Ere: Agz.
40
10:03
45
40
86
86
60-
83
60-
45
45
Dwc Apc
40
Diec Rd
3
30
100M
200M
2,0
100M
200M
1.0
1.2
1.6
1.0
Model siz0
Speedup (Seclexamppk)
Speocup (Seotokan)
Baselne (TG Bace)
Encoder (wl Layer) Enooder (wl Layer)
Decoder (wl Layer) d Decader (w// Layeri
Figure 2: Comparing model size (or speedup) output performance for four pruning options: with without depth pruning applicd to either the encoder or decoder network individually. The results emphasize that (1) the number of layers the decoder network the primary factor contributing to speedup improvements. and (2) the sparsity of the encoder network is the key factor of output quality.
to MHA can be formulated as follows:
of cases, we aim to address the following ques- tions: (1) Does depth pruning exhibit different phc- nomena in each case? (2) What is the key factor for accelerating inference speed while preserving sufficient output quality? We provide detailed an- swers to cach question by training T5-Basc with target sparsities of {60%, 70%, 80%, 90%, 95%} for the decoder cases, and {20%, 40%, 60%, 70%, 80%, 90%, 95%) for the encoder cases. I
No.
() QQ,K, V)
MMIIA"
(
bead"
(i.)
Au(Qw())) (15) kW K
vw w()))
(6) where "MHA and head E {0, 1 and to mask MHA layer and individual head of MHA.
(1.))
The FFN layer, which is another major compo- nent of the Transformer network, is also known to be over- parammaraeteriee (Dong al., 2021). Strate- gies for pruning include pruning an entire FFN layer and pruning intermediate dimensions at a more granular width level. This can be achieved (6)
Before delving into the detailed answers, we briefly address the first question: the impact of depth pruming when applied the encoder and de- coder, respectively. As depicted in Figure 2, depth pruning exhibits a significant influence on the de- coder (as indicated by A and O), while the encoder shows negligible effects (as observed in A and O), Consequently, the appropriate utilization of depth pruning becomes crucial. In the following para- graphs, we outline our key findings related to the second question establish effective structured pruning mechanism for encoder-decoder LMs.
_(i) by introducing mask variables, "FFN and z E
{0, 1345, with the following formulation:
( FFN GELU(XW;) -iwW:Wi)ddi((( (2)) ) WW2
FFN X)=
Various techniques have been employed to learn these mask variables used in structured pruning. For example, Wang al. (2020c) and Xia al. (2022) utilized LO regularization to eliminate re- dundant parameters. On the other hand, Lagunas et al. (2021) adopted the movement score intro- duced by Sanh et al. (2020) as a measurement for their pruning approach.
Finding 3.1. The number of layers in the decoder network is the dominant factor affecting the infer- ence speed, while the decoder width does not have significant impact.
We evaluate the findings regarding the decoder net- work from two perspectives: (1) the effectiveness of layer-wise pruning and (2) the ineffectiveness of width pruning. Firstly, demonstrated in the second and fourth plots of Figure 2, the decoder exhibits a significant speedup with minor degra- dation (when comparing A and O), whereas the encoder shows no such effect (when comparing A and O). This indicates that layer-wise pruning plays dominant role in pruming the decoder. On the other hand, when comparing the model size
3 Experimental Motivations
In this section, we separately investigate the behav- ior of the encoder and decoder when depth prun- ing is applied or not, using CoFi-T5, the modi- fied version of CoFi (Xia et al., 2022) tailored for T5 (Raffel al., 2020). Particularly, in Figure we study the results of four cases: encoder with depth-pruning (A), encoder without depth-pruning (O), decoder with depth-pruning (A), and decoder without depth-pruning (O). From these four types
'In the case of high level of target sparsity, we observed that CoFi-T5 T5 is unable to achieve the desired sparsity. A de- tailed explanation of this phenomenon is in the Appendix A.

---

Relative Process Time per Transformer Layer
Relative Process Time
(Dem - Ny (Dra 1224, N,
Relative Time (%)
RTE
Theiii
100
SAMsum
Relative Time
75
20
50
25
Enc. Total
Dec.
Dec.
Dec.
Dec. LN
Dec.
SA
CA
FF
DO
128
256
512
Sequence Length
Network compomenet
Figure 3: Processing time per one Transformer layer depending on the model configuration and the sequence length. As depicted in the sequence length case, the factors, such as the number of attention heads and FFN dimensions not affect the processing time.
Figure 4: Componentwise processing time of the T5- base. The layer normalization and dropout contribute 20-25% of the total inference time.
nificant performance degradation. These findings emphasize the significance of considering pruning in both the decoder and encoder networks. Further- more, they provide insights into the necessity of employing distinct pruning strategies for these two networks, considering their unique characteristics.
and speedup (as shown in the first and second plots of Figure 2 with O), width pruning reduces the model size but leads to both performance degra- dation and negligible speedup. This suggests that width pruning not effective for the decoder.
Comparison with Prior Observations. Our key findings provide valuable insights: the appropriate strategy for encoder encoder-decoder decoder models involves us- ing small number of layers for the decoder and minimal pruning for the encoder networks. Impor- tantly, our observations offer more generalized understanding compared to previous works (Kasai et al., 2020; Tay et al., 2021). Unlike prior studies that manually determined model configurations for specific tasks such as machine translation (Kasai et al., 2020) or NLU (Tay et al., 2021), our con- clusions are derived automatically through gradual structured pruning and have been validated across both NLG and NLU tasks. Furthermore, while the DeepNarrow strategy proposed by Tay al. (2021) demonstrates effectiveness in NLU tasks with short output sequences, it exhibits computational ineffi- ciency when applied to NLG tasks. Similarly, the contribution of processing time for encoder net- works varies, necessitating the use of narrower encoder architecture contrary to the approach pro- posed by Kasai et al. (2020).
To further investigate Finding 3.1, we investigate the inference speed of Transformer layers, with specific focus understanding why width prun- ing is ineffective. This analysis involves two key observations: (1) finding the metric that synergizes with width pruning, and (2) identifying the compo- nent that predominantly consumes computational resources. According to Figure 3, width pruning can have a significant impact on the computational cost as the sequence length increases. However, due to the inherent nature of the autoregressive de- coding process, the decoder network is constrained to sequence length of 1. As result, width prun- ing cannot effectively improve the speed of the decoder network. Furthermore, as illustrated Fig- ure 4, Layer Normalization (LN) and dropout (DO) collectively contribute approximately 20-25% to the overall inference time. Consequently, the time allocated to these fixed operations remains constant, leading to diminished efficiency in terms of infer- ence speed. In conclusion, width pruming is not an appropriate approach for optimizing the decoder.
Finding 3.2. From the perspective of encoder prun- ing, while achieving high-level sparsity may not be desirable, attaining low-level sparsity not only slightly accelerates inference speed but also en- hances performance.
Narrow Encoder and Shallow Decoder
Based on the findings presented in Section 3, we propose structured pruning framework called NASH (Narrow encoder and Shallow decoder) that is specifically optimized for encoder-decoder LMs. Our approach focuses on enhancing inference speed by utilizing uniform layer selection in the decoder network, deviating from the gradual prun- ing technique commonly employed in encoder-only models. Additionally, we improve generation per-
By comparing the O points and * in the second and fourth plots of Figure 2, we observe that en- coder pruning yields a slight speedup along with improved performance. However, when the en- coder network is heavily pruned, experiences sig-

---

formance by applying gradual LO regularization pruning specifically to the encoder network, in- ducing low sparsity instead of solely prioritizing inference speed improvement.
in practice is achieved by enforeing an equality con- straint between the target sparsity and the current sparsity:
== A1 1) 12 1)2
4.1 Shallow Decoder:
Uniform Layer Selection for Decoder Networks
where A1. 22, s, and denote the learnable La- grange multipliers, current sparsity, and target spar- sity, respectively. The detailed derivation of R is described in Appendix B. The current sparsity, 8, is calculated as follows:
For given number of selected layers dg we can generate sub-network of the decoder network with a set of selected layers as follows:
L, { 1 |
l 1+1/ do
d. -111
dd, d
N
M dp Σ
(i.)
head +
M
ini
We match the hidden states of the sub-network to those of unpruned decoder networks:
where M, L. Nr. and d indicate the number of model parameters, encoder layers, attention heads, and feed-forward layer dimensions, respectively, We only conduct the pruning of individ- attention heads and intermediate layer
e+1
Cdoe
Σ
MSB(H dec dec,t
xE(T....d.)
While uniformly selecting layers work well on var- ious domains such as knowledge distillation (Jiao et al., 2019; Shleifer and Rush, 2020) or structured pruning of encoder-only model (Hou et al., 2020), work first proposes using uniform layer selec- tion of decoder network for structured pruning of encoder-decoder LMs.
(i) dimensions by introducing variables 2 and Zint-
No
MHA K, V) =
Σ
head (O
FFN(X) = GELU(XW,)) diag(Zint) W2.
The key philosophy of our proposed module is twofold: (1) As shown in Finding 3.1, , the num- ber of layers in the decoder network is the main factor affecting inference speedup. (2) Uniform selection is proven to be an effective approach for selecting layers (Hou al., 2020). To verify this second statement, we compare various candi- dates, including uniform selection, selection with lower layers, selection with higher layers, and the LO regularization-base approach (Louizos et al.,
We further use hidden states distillation by match- ing the hidden states of pruned and unpruned networks at the same layers as follows:
Ln
nene
Σ (T........)
MSE(H onc.e H Hen..).
As we demonstrated in Finding 3.2, structured pruning with low sparsity enables output quality enhancement rather than inference speedup gain, Motivated by this finding, unlike previous meth- ods (Wang al., 2020c; Xia al., 2022) that mainly LO regularization to achieve high in- ference speedup, we use such LO regularization accomplish improvement output quality.
2018). Through our empirical evaluation, we con- firm that uniform selection is the best approach among these candidates (see Section 5.3 and Ta- ble 3 for details). Based this philosophy, construct shallow decoder pruning by selecting the layers using uniform selection.
4.3 Training Loss Function of NASH
4.2 Narrow Encoder: Gradual
We combine hidden states distillation with prediction-layer distillation by using Kull- back-Leibler divergence (KLD) function.
LO-regularization with Low Sparsity
Among various structured pruning methods (Hou al., 2020; Lagunas et al., 2021). utilize the LO regularization-batt pruning method, which has shown the state-of-the-ee performances encoder- only language models (Wang et al., 2020c; Xia et al., 2022). The application of L0 regularization
Ppred KLD (f()))la()),
where the f(-) and (+) are softmax outputs for the sub-network of pruned model and unpruned model,

---

Table I: The summary of Figure 5 which compares the generation quality and latency speedup of NASH against other acceleration methods TweetQA (Xiong al., 2019), XSum (Narayan al., 2018), SAMSum (Gliwa et al., 2019), and CNN/DailyMail (See et al., 2017). The numbers of parameters for all models are around 60M, except for T5-Base. The best and second-best results of sharing dataset are highlighted bold and underline.
Task
Question Answering
Summarization
TweetQA
XSum
SAMSum
CNN/DailyMail
Dataser
METEOR
Speedup
ROUGE-L
Speedup
ROUGE-L
Speedup
ROUGE-L
Speedup
T5-Base
57.16
1.00x
28.66
1.00x
43.33
1.00x
37.47
1.00x
TS-Small
$1.42
x.92
25 23,04
1.96x
38.14
2.14x
35.59
2.11x
CoFi-T5
49.99
1.42x
25.04
1.12x
37.35
1.45x
34.71
1.53x
NASH decoder layers)
4.02
26.96
3.74x
38.89
5.53
33.70
4.91
NASH (3 decoder layers)
50.77
2.99
28.20
2.42X
41.09
4.24x
36.02
3.37
NASH (4 decoder layers)
55.08
2.52
28.64
1.64
41.34
2.99
36.78
2.69x
TweeGGA
XSum
SAMSum
CMNVDaiyMal
44
301
METEOR
42
ROEGUL
Speedup
Speedup
Speodap
Speedup
CoF1-15
T5-Mml
Figure 5: METEOR (or ROUGE-L) vs. speedup on abstractive question answering (TweetQA) and summariza- tion (XSum, SAMSum, and CNNiDailyMail) tasks. We compare NASH on 220M T5-Base against CoFi-T5 on T5-Base, 60M T5-Small, 31M T5 15-Mini (Tay et al., 2021). On datasets, NASH is able to outperform CoFi-T5.
Lagend
2019). For the text summarization task, we ex- periment on XSum (Narayan et al., 2018), SAM- Sum (Gliwa et al., 2019), and CNN/DailyMail (See et al., 2017). We evaluate the output quality using METEOR (Banerjee and Lavie, 2005) for abstrac- tive question answering and ROUGE (Lin, 2004) for the summarization tasks. We conduct experi- ments on multi-task scenario that consists of SAM- Sum, TweetQA, and five tasks from GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks. The detailed explanations for the datasets used are described in Appendix C.
Tharsal TSbeee
Prured mostic
com-Ts
0.88
Croo aeae
7 NA8444
3 comirs
0.15
g RAsae
Figure 6: Comparison of the unpruned layers between NASH and the other methods SAMSum. The values on the right of each row indicate the corresponding sparsity compared to the part (encoder or decoder) of TS-base, respectively.
respectively. Then, the total training objective for pruned model is
Implementation. First, fine-tune model and perform uniform layer selection on the decoder net- work of the fine-tuned model to generate sub- network model. Subsequently, we further train the sub-network model with the pruning objective, uti- lizing scheduler to gradually increase the sparsity until reaching the desired target value. In our experi- ments, we calculate sparsity by dividing the number of pruned parameters (excluding the embedding) by the size of the full model. Following the ap- proach of Xia et al. (2022), we continue fine-tuning the pruned model until convergence. We set the target sparsity of the encoder networks as 30% for all experiments. The reported results are based on
- pred + cnc Lone + + A cdee R,
where Anne and doc are coefficients for controlling the contribution of hidden state distillation for the encoder and decoder network, respectively.
5
Experiments
5.1 Experimental Setup
Dataset. We evaluate our proposed method on various tasks using the versatility of encoder- decoder LMs. For abstractive question answering, we conduct experiments on TweetQA (Xiong et al.,

---

bese
TS-case Singo
Ts-mmall
Nastie
Nash- Tasko
Nash-4 Seven
SAM4ium
60
WETEOR
12
ROUGEL
40-
38-
1.5
2.5
Sprecuu
Speedoup
Spendup
Figure 7: Evaluation of generation quality and latency speedup of NASH multi-task learning scenario. We compare NASH on 220M T5-BBase against T5-Base and 60M TS-Small. On all datasets, NASH is able to outperform T5-Small.
Table 2: Evaluation generation quality and latency speedup of NASH on instruction-tunii scenario. Note that NASH-L indicates that conduct NASH with decoder layer number of L.
Dolly Evaluation
Self-Instruce ROUGE-L
Vicuna Evaluation
Method
GPT Eval
ROUGE-L
Speedup
GPT Eval
Speedup
GPT Eval
ROUGE-L
Speedup
T5-Base
61.32
32.53
1.00x
40.15
14.20
1.00x
59.26
20.47
1.00x
T5-Small
47.03
29.29
1.96x
29.82
11.67
1.91 x
34,98
17.40
2.21x
NASH-2
41.28
28.09
5,15x
26.74
10.58
5.27
45.53
20.09
5,90x
NASH-3
52.82
30.87
3,62
32.18
11.91
3.79
50.59
21.07
4.24x
NASH-4
56.85
31.43
2.67
36.44
13.01
2.94x
54.21
21.83
3.17x
the validation sets of all datasets. Additionally, our models are implemented using Huggingface (Wolf et al., 2020) library.
rithm (NASH) exhibit fluctuations in accuracy ac- cording to the number of tasks. However, it is worth noting that our proposed method demon- strates significantly less variation in generation performance when compared T5-Small. This robustness in multi-task learning consistently observed across all datasets. Overall, NASH con- sistently outperforms T5-Small in terms of both generation performance and speedup, similar to standard fine-tuning. Through the results shown in complex scenarios, we verify that the generation performance of encoder-decoder models is robust to the number of decoder layers.
5.2 Main Results
Standard Fine-tuning. Table 1 and Figure summarize that the proposed method outperforms CoFi-T5, 6-layer "F-Small,(Rafiee et al.. 2020), and 4-yayer T5-mini, et al., 2021) in terms of output quality and inference speedup. Our results consistently demonstrate the superiority of NASH with three decoder layers over the baselines both performance metrics, although there trade-off between output quality and inference speedup with our method. Moreover, our method is particularly effective in improving inference speed, especially for tasks involving longer sequence outputs, such SAMSum CNN/DailyMail. However, CoFi- T5 falls short in achieving comparable speedups T5-Small while maintaining the same output quality. Figure 6 illustrates the pruned model struc- ture trained on SAMSum, following the method described Table to provide a deeper under- standing of the results. Despite CoFi-T5 removing more modules in the encoder network compared to NASH, it cannot remove as many layers in the decoder layer as NASH does.
Instruction Tuning. To verify versatility of pro- posed NASH, we consider the instruction-tuniin sce- nario with the sttabrickks-olll-1 ( (Conover et al., 2023) dataset. By following Gu et al. (2023), split the total dataset into 14k train sam- ples and 1k evaluation samples. We evaluate the trained model (with 14k of train samples) on three instruction- instruction-follooinng datasets to check the general- izability of the model across the different datasets: (1) 1k dolly evaluation; (2) Self-Instruct (Wang et al., 2023a); (3) Vicuna evaluation (Chiang et al., 2023). Similar to previous instances of task-specific fine-tuning and multi-task scenarios. algorithm with three decoder layers consistently outperforms T5-Small across various metries, including GPT- evaluation, ROUGE-L, and speedup, as shown in Table 2. These results suggest that our pro- posed method well-suited for developing general-
Multi-task Learning. We conducted an anal- ysis to observe how performance trend change with varying numbers of tasks. As depicted in Figure 7, both fine-tuned Ba-Base and our algo-

---

Table 3: The performance of various layer selection strategies different datasets is presented. "Gray" indi- cates a failure to achieve the target sparsity. Additionally, we report the number of remaining SA, CA and FF lay- ers for the automatic selection method.
Table 5: Comparison of results achioved by CoFi-T5 T5 and NASH on deeper models (Tay et al., 2021) and SAMSum.
Task
16
NL
R-L
Specdup
R-L
Speedup
R-L
Speedup
Rase
41,31
1.0x
41.73
1.0x
[1.83
1.0x
Task
RTE
BoolQ
CB
SAMSum
TweetQA
CoFi-I5
33,53
1.6x
35.87
1.6x
36.65
1.5%
Uniform (Ours)
76.17
79.51
91.07
47.37
52.74
NASH (2 DL)
36.50
4,6%
35.45
5.2%
35.29
5.9%
Low
75.81
79.82
89.29
46.14
49.70
NASH DL)
39.05
3.6x
3.8%
38.25
4.3x
High
75:45
78.99
87,50
39.45
47.40
Lonizos et (2018)
78:20
79.20
92.86
48.52
56.41
Table 6: Comparison of results of NASH and Tao et al. (2023) on BART-Base and CNN//ailyMail. Results of Tao et al. (2023) are from the original work.
#SA, CA. FF
2,4.2
0,3.2
4,7.4
6,7,5
11,10.8
Table 4: Comparison of pruning strategy on encoder network. We conduct our method on T5-base with a uniform selection of 4 decoder layers.
BART
NASH
Tao et al. (2023)
Base
NASH-2
NASH-3
50%
27%
ROUGE-L
41.68
40.87
41.14
39.91
40.39
Dutaset
RTE
SAMSum
TweetQA
Acc
Speedup
R-I.
Speedup
MTR
Speodup
Speedup
1.0x
3.2x
2.1x
~1.5x
# Params
139M
80M
89.8M
70.9M
102.4M
None
37.97
2.12x
41.06
2.78x
54.73
2.42
Unit, (S 0.25)
74,36
2.35K
40.42
2,86x
53.62
2.55
Unif, (S 0.5)
71.84
2,59x
39.36
2.94%
49.62
2.68 x
lected uniformly. The results presented in Table 4 clearly demonstrate that our chosen approach, with low sparsity, outperforms both the unpruned base- line and the uniform layer selection. We also ob- serve that the advantage of this approach is only noticeable at low sparsity, as evidenced by the com- parison between 30% and 60% sparsity.
LO Reg. (S 0.3)
78.70
x.24
2.99
55.08
2.52x
1.0 Rog. (S 0.6)
76,89
2.61
40.49
3,12%
49,48
2.75
purpose language models, a usage pattern widely adopted in recent large language models (Chung et al., 2022a; Tay et al., 2023).
5.3 Ablation Studies
NASH on Different LMs. We also conducted a deeper model experiment using T5-Small- efficient,(Tay et al., 2021), which is a variant of T5-Small with up to four times more layers while maintaining the same configuration. This experi- ment aimed to determine the effectiveness of our method regardless of the model architecture. The results presented in Table 5 consistently demon- strate that NASH improves inference speed without significantly compromising the quality of gener- ated outputs, regardless of the depth of the decoder networks. It is noteworthy that the acceleration compared to the original model increases as the number of decoder layers increases. Furthermore. NASH exhibits faster inference and higher output quality compared to CoFi-T5, which consistent with the results presented in Table 1.
Different Layer Selection. To validate the effec- tiveness of uniform layer selection in shrinking the decoder network, we investigate other layer selec- tion methods in a two-layer selection problem. We compare four selection methods: lower selection (first 2 layers), higher selection (last 2 layers), LO regularizaaion--ased automatic selection (Louizos et al., 2018; Xia et al., 2022), and our uniform selec- tion. The results presented in Table demonstrate that uniform selection consistently outperforms the other manual selection methods. The performance margin becomes more pronounced in NLG tasks. Notably, we observe that automatic selection fails to achieve the target sparsity consistently across all tasks, except for BoolQ. This instability in auto- matic selection aligns with our preliminary findings discussed in Appendix A.
Comparison with Tao et al. (2023). We applied NASH to BART-Base (Lewis et al., 2020) using the CNN/DailyMail dataset, conducting direct com- parison with SIMPLE (Tao et al., 2023). SIMPLE introduced a structured pruning method for genera- tive LMs, which is relevant to our work. Notably, NASH exhibits higher ROUGE-L scores than SIM- PLE when both models are at 27% sparsity. Ad- ditionally, despite having larger parameters, NASH
Different Pruning on Encoder. To evaluate the effectiveness of our pruning strategy on the encoder network, investigate the following strategies at different levels of sparsity: (1) without encoder pruning, (2) uniform layer selection (similar to the decoder network), and (3) the proposed L.0 regu- larization approach. We prune the encoder network of the T5-Base, which has four decoder layers se-

---

outperforms SIMPLE with 50% sparsity in terms of speedup. Our approach achieves more than three times the speedup, while SIMPLE reaches a maxi- mum of 1.5 times on the GPU.
ing, but their method can only achieve up to 1.4x speedup for encoder-decoder based BART (Lewis et al., 2020). Yang al. (2022) released an open- source toolkit that combines structured pruning and vocabulary pruning for various pre-trained lan- guage models, but only vocabulary pruning is ap- plicable to T5 and BART.
6 Related Works
Language Model Compression. With the ad- vancement of NLP, LMs have grown in size, making difficult to deploy them on edge de- vices and resulting in slower inference speed. As a result, there has been active research on lan- guage model compression which has three main approaches: quantization, knowledge distillation, pruning. Quantization (He et al., 2016; Alom et al., 2018; Zafrir et al., 2019; Shen al., 2020; Yao al., 2022) minimizes the storage requirements for weight values by reducing the number of bits needed to represent them. Knowledge distil- lation (Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2019, 2020; Wang et al., 2020b,a) transfers the knowledge of large-scale teacher model with high performance to smaller-scale student model, enabling the student model to replicate the behavior of the teacher model. Pruning (Chen et al., 2020; Sanh et al., 2020; Kwon et al., 2022; Frantar and Alistarh, 2023) reduces the size of a model by re- moving unnecessary parts of large networks such as neurons, weights, or layers.
7 Conclusion
We propose NASH to address the lack of exploration in structured pruning of encoder-decoder LMs. To design structured pruning method suitable for encoder-decoder models, we first examine the be- havior of pruned models with different strategies, focusing inference speed and generation perfor- mance. Our findings reveal that (1) the number of decoder network layers is the key factor in accel- erating inference speed and (2) low sparsity prun- ing on the encoder network can enhance model performance. Based on these insights, we develop NASH, which constructs narrow encoder and a shallow decoder network for encoder-decoder LMs through gradual LO regularization pruning and uni- form layer selection, respectively. We demonstrate the superiority of NASH in terms of speedup and output quality across various tasks. We strongly be- lieve this work lays a strong foundation for further investigation into effective pruming approaches for encoder-decoder LM.
Pruning. Pruning be categorized into two parts: (1) unstructured pruning and (2) structured pruning. In unstructured pruning (Chen al., 2020; Prasanna et al., 2020), weights, which are connec- tions between neurons, are removed from the net- work based on various criteria. However, this line of methods produces sparse weight matrices, requir- ing specific hardware support. On the other hand, structured pruning (Xia et al., 2022; Kwon al., 2022; Kurtic et al., 2023), prunes away structures such as neurons, weight matrix blocks, or layers. Most previous works on structured pruning have focused on encoder-based models (Xia al., 2022: Kwon et al., 2022; Kurtic et al., 2023), which re- move attention heads, columns, and rows of weight matrices using different importance score metrics, including magnitudes or Hessians of weight ma- trices, and LO loss. However, structured pruning on generative models has been significantly under- investigated, with only few available works (Lagu- nas al., 2021; Yang et al., 2022; Santacroce et al., 2023). Lagunas et al. (2021) extended movement pruning (Sanh et al., 2020) into structured prun-
Limitations
Although we were unable conduct research on unstructured pruning due to device limitations, collaboration with devices could facilitate perfor- mance enhancements. Furthermore, owing to the motivating analysis and algorithm construction of this paper, i.e., analysis of separate encoder and decoder networks, further exploration of co- optimized method is necessary, and there is po- tential for improvement in this aspect.
Acknowledgment
This work was supported by the "Research on model compression algorithm for Large-scale Lan- guage Models" project funded by KT (KT award B220002586, 90%) and Institute of Information & communications Technology Planning & Eval- uation (LITP) grant funded by Korea government (MSIT) [No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), 10%].

---

References
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng. Xiaozhe Ren, and Xiaodan Liang. 2021. Efficient- BERT: Progressively searching multilayer percep- tron via warm-up knowledge distillation. In Find- ings of the Association for Computational Linguis- ficx: EMNLP 2021, pages 1424-1437, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Md Zahangir Alom, Adam T Moody, Naoya Maruyama, Brian C Van Essen, and Tarek M Taha. 2018. Er- fective quantization approaches for recurrent neural networks. In 2018 international joint conference on neural networks (IJCNN), pages 1-8. IEEE
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan, Association for Computational Linguis- tics.
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: #rXiv:/803.0383 1803.03635.
Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Mas- sive language models can be accurately pruned in one-shot.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin, 2020. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural informa- tion processing systems, 33:15834 15846.
Xinyang Geng, Amav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Bogdan Gliwa, Iwona Mochol, Macicj Biesek, and Alck- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Knowledge distillation of large language mod- els. arXiv preprint arXiv:2306. 08543.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022a. Scaling cesructioonfinetunee language mod- els. arXiv preprint arXiv:2210.11 11416.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang. Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross- modal pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7212-7225, Dublin, Ireland. Associa- tion for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- hert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Sungun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Costro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022b. Scaling cneruuction-inntunee language models.
Manish Gupta and Punect Agrawal. 2020, Compression of deep learning models for text: A survey, ACM Trans. Knowl. Discov. Data, 16:61:1161:55.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally, 2016. Eie: Efficient inference engine on compressed deep neural network. SIGARCH Comput. Archit News, 44(3)2243-254.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction- tuned Ilm.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Newmodel: Improving deBERTa using ELECTRA- style pre-training with diideenn-disettagleedd embed- ding sharing. In International Conference on Learn- ing Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HHLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-41 1-86. Association for Computational Linguistics.
Qinyao He, He Wen, Shuchang Zhou, Yuxin Wu, Cong Yao. Xinyu Zhou, and Yuheng Zou. 2016. Effective quantization methods for recurrent neural networks. arXiv preprint
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu, 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances Neural Information Processing Systems, 33:9782-9793.

---

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language under- standing. arXiv preprint Xrviv 1e::9900 1909. 1035I.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744:
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
TinyBERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4163-
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, 7eming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
4174, Online. Association for Computational Lin- guistics.
Jungo Kasai, Nikolaos Pappas, Hao Peng. James Cross, and Noah A Smith. 2020. Deep encoder, shallow decoder: Reevaluating non non-autoregressive machine translation. arXiv preprini arXiv:2006.10535 10369.
Sai Prasamna, Anna Rogers, and Anna Rumshisky. 2020.
When bert plays the lottery, all tickets are winning. arXiv preprint 2rXi::2005.00560
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former.
Eldar Kurtic, Elias Frantar, and Dan Alistarh. 2023. Zi- plm: Hardware-awan structured pruning of language models.
Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022. fast post-training pruning framework for transformers.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021. Block pruning for faster trans- formers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 10619-10629, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine- tuning.
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. 2023. What matters the structured pruning of generative language models? arXiv preprint 232.7333...
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeeee to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7888 Online. Association for Computa- tional Linguisties.
Abigail See, Peter J. Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 34, pages 8815-8821.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through to regularization.
Sam Shleifer and Alexander M Rush. 2020. Pre- trained summarization distillation. arXiv preprint arXiv:2010.13002.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model com- pression. arXiv preprint arXiv:1908.09355.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: compact agno-agi bert for resource-limited de- vices. arXiv preprint arXi:
OPenAI. 2023. Gpt-4 technical report.

---

Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023. Structured pruning for efficient generative pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10880- 10895, Toronto, Canada. Association for Computa- tional Linguisties.
Hajishirzi. 2023a. Self-instruct: Aligning language models with self- geferreed instructions. In Proceed- ings of the vt Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 33484-13505 Toronto, Canada. Association for Computational Linguistics.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi, 2023b. Codet5+: Open code large language models for code understanding and generation. arXiv preprint 2xXiv2305500922...
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar. Hyung Won Chung. Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2021. Scale efficiently: Insights from pre- training and fine-tuning transformers. arXiv preprint arXiv:2109. 10686.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020c. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 6151-6162 Online, Association for Computa- tional Linguistics.
Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng. Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh Inter- national Conference Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of- ofate-of-thh-arr the-hhh-rr natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1513-1528, Dublin, Ireland. Association for Computational Linguisties.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: stickier benchmark for general-purpose language understand- ing systems. Advances in neural information pro- cessing systems, 32.
Wenhan Xiong, Jiawei Wu, Hong Wang. Vivek Kulka- mi, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5020- 5031, Florence, Italy. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: mullt-taakk benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguisties.
Ziqing Yang, Yiming Cui, and Zhigang Chen. 2022. Textpruner: A model pruming toolkit for pre-trained language models. arXiv preprint arXiv:2203.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020a. Minilmy2: Multi-head self-attention relation distillation for compress- ing pretrained transformers. arXiv preprint arXiv:2012.15828.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168- 27183.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- sormation Processing Systems, 33:5776-5788.
Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8hit bert. In 2019 Fifth Workshop Energy Efficient Machine Learning and Cognilive Computing-Nuurlii Edition (EMC2-NIPS), pages 36-39. IEEE
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch

---

Tianyi Zhang, Felix Wu, Arzoo Katiyar. Kilian Q Wein- berger, and Yoav Artzi. 2021. Revisiting saw-semple [bert] fine-tuning. In International Conference on Learning Representations.

---

A Instability of CoFi-T5
this section, we observe that pruning of the T5 shows more varied accuracy and pruned spar- sity than those of BERT under the same condition of pruning (e.g., target sparsity, warm-up epochs) which means instability of encoder-decoder model pruning.
DEAT (ITC)
TG (ITE) T3 (SAMISu
DERT (RTE)
TO (5AMeii
Vide (Dirso))
Figure The pruned sparsity (left) and output perfor- mance (right) distributions for 90% of target sparsity across 20 random trials on RTE (Wang et al., 2018) with BERT-Base RTE with T5-Base, and SAMSum (Gliwa et al., 2019) with TS-Base. Unlike the results of BERT on RTE, which show that the accuracy and pruned spar- sity concentrated around the target sparsity, we ob- serve that both output performance and pruned spar- sity vary across different trials. This variation confirms the instability of pruning- decoder models. It is important to note that accuracy and ROUGE-L are used as output performance metrics for RTE and SAMSum, respectively.
Experimental Setup. We study the instability of CoFi-T5 with T5-Base compared to the orig- inal CoFi with BERT-Base (Devlin et al., 2019). To compare the T5 and BERT, we conduct the ex- periments on the RTE task of the GLUE bench- mark (Wang et al., 2018) with 90% of target spar- sity² for both models. Additionally, we extend our investigation the SAMSum (Gliwa et al., 2019) which contains messenger-like conversations with summaries, utilizing T5- 15-Base to observe the train- ing instability of the encoder-decoder model on more challenging tasks. We prune with 20 random seeds to compare different settings.
training for encoder-
B Pruning with LO Regularization
In this section, we give the details of the pruning with Lo regularization. Structured pruning through LO regularization (Louizos et al., 2018) 8) is proposed to construct the sparse neural network efficiently. In addition, this scheme using Lo regularization is widely applicd to prune LMs (Xia et al., 2022; Wang et al., 2020c). With given LM f(: 0) that is parameterized by 0 /- we can define binary mask variable and 2j denote a unit of weights (e.g., weights of attention heads, or column of an MLP layer) and mask variable corresponding to 0j. respectively.
Results. Figure 8 presents main observations. Firstly, the output performance of pruned T5 mod- els exhibits more variability than pruned BERT models at high-level target sparsity. Secondly, CoFi- T5 fails to achieve the target sparsity at high-level sparsity in both RTE and SAMSum. In the case of RTE, we observed high proportion of over- pruming, while in the case of SAMSum, all ex- periments were significantly under-pruned. These results indicate the instability of Coli-T5 in terms of sparsity and output quality, which aligns with the findings discussed by Zhang et al. (2021).
{z)]ー=1- Note that 6g
In this formulation, a pruned LM is written as (-s 0). where = O z and the pruning is a prob- lem to find optimal mask variables and weights. In the Lo regularization-based pruning, these masks and weights are learned based the following objective function:
LO regularization based structured pruning methods (Wang et al., 2020c; Xia et al., 2022) commonly incorporate linear warm-up up to gradu- ally increase the target sparsity during the training process, aiming ensure stable training of mask variables. Based on this understanding, we employ longer warm-up epochs for gradual structured prun- ing and observe that this approach partially miti- gates the instability of CoFi-T5. While this miti- gates the training instability to some extent, it does not completely address the challenge associated with CoFi-T5. These results motivate us to investi- gate the appropriate strategy for structured pruning in encoder-decoder models."
[ D (((()))) + All011o min E, D D
In the objective function above, every mask vari- able 2j is chosen based on the prior distribution q |Tt) = Bernoulli(TT). Considering the num- ber of binary masks, n, the possible choices of z can be represented as 2n. The discrete feature of the mask and this tremendous amount of choices make mask optimization practically intractable. Louizos et al. (2018) mitigate this issue with a re-parameterization trick, enabling to be differ- entiable and updates with the model parameter 0. In detail, the masks z are defined as continuous
"The sparsity computed as the number pruned param- eters divided by the full model size (embeddings and classifier excluded).
"As longer warm-up epoch is shown to be effective, we applicd it to hoth CoFi-T5 and NASH in our experiments.

---

Table Description and label NLU datasets in GLUE and SuperGLUE benchmarks.
Task
Description
Label
CoLA
To determine lingaistic accepiability single sentence
d'accoptable', "not_ noo__cccpeablee
MRRC
To answer whether human annotators paraphrase input sentence pairs
l'equivalent' equuisaal)l
STS-B
To answer how much input sentence pairs are semantically similar
0cx-33 (regression)
GLUE
KTB
determine logical relationship berween inpur sentence pairs
I'entailment' 'not__ntailmmnnt
S
To answer whether the input sentence contains positive negative sentiment
[" ["negative", pusilivee] pusiive
QNLI
To determine whether a given context entails the answer to a correspoeding question
I'entallment' , 'not cnt__entalmmmn" ]
CB
To determine the Linguistic uccepiabiity of single sentence
l'uccopiable', _ aaceptabllll
COPA
To comprahend the logical connections hetween events and make accurate judgments
l'choicel'. "choios2)]
SGLUE
WIC
To deternine the linguistic neceptability of single sentence
['tmae', 'false']
BoolQ
to determine the linguistic ucceptability of a single sentence
I'tro'. 'lalse']
variables determined by min(1, max s)), where continuous random variable is sampled from the range of [o, 1]. Note that it is equivalent to sam- ple u from the uniform distribution, U(0, 1) and calculate s as follows:
two tasks: summarization and abstract question answering. We employed the XSum (Narayan et al., 2018), SAMSum (Gliwa et al., 2019), and CNN/DaaIlyMal (See et al., 2017) datasets to as- sess the summarization capability of our proposed method. These datasets are widely used in evaluat- ing the effectiveness of summarization techniques. Regarding abstractive question answering, we em- ployed the TweetQA (Xiong al., 2019) dataset to evaluate method.
sigmoid(logu - log(1 - u) a), x (r - 1) + 1,2 min(1, max(0, s))
where and r are constant values that satisfy I < and r > 0, and a is a learnable paramcter. From this formulation, the learning objective can be re- written as:
XSUM (Summarization): XSUM (Narayan al., 2018) comprises articles sourced from BBC. along with corresponding single sentence summaries. More specifically, each article be- gins with an introductory sentence that serves as a summary. These summaries are professionally written and are usually provided by the article's ss author.
D
min E U(O,))
U(0.1)
D E
C(p(ax;0).0 + +110110
This process obtains = where every 2j is in the range of [0. 1J. However, Wang et al. (2020c) observes that optimizing with those relaxed reg- ularizations makes models converge to different- size subnetworks depending on a learning rate and pruning schedule. To mitigate this problem, they suggest using Lagrangian relaxation instead of the LO regularizer Al101o follows:
SAMSum (Summarization); SAMSum (Gliwa al., 2019) consists of 16K messenger-lie con- versations annotated with summary for provid- ing a concise overview of the conversation's 's con- tent in third person. The conversations encom- pass a variety of styles and registers, ranging from informal to semi-formal and formal. Addi- tionally, they may include slang words, emoti- cons, and typographical errors.
(s 1) + Az 1)2
where A and A2 are learnable Lagrange multipli- ers. $ represents the current sparsity calculated by the masks while represents the target sparsity. Motivated by these works, we also utilize the re- laxed regularization term, R, for gradually struc- tured pruning on the encoder network.
CNN/Daily (Summarization): CNN / Dai- lyMail (See et al., 2017) consists of over 300K English news articles that were originally de- signed for machine-reading and comprehension well abstractive question answering, but now also supports extractive and abstractive summarization. this work, we utilize the 3.0.0 version.
Description of Datasets
Natural Language Generation Tasks. Since we study the structured pruning for encoder-decoder models, a sort of generative model, we conduct comprehensive experiments on the NLG tasks. The NLG datasets used in our study encompass
TweetQA (Abstract Question Answering): TweetQA (Xiong et al., 2019) is the first large- scale dataset for question answering (QA) over

---

Table 8: Hyperparameters for NLG datasets
of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. In this work, we employ six tasks in GLUE benchmarks: CoLA, MRPC, STS-B, RTE, SST-2, and QNLI
Hyperparameter
Value Description
20 (TQA, SAMSum), (XSum, CNNIDM)
Training epochs
how many spochs are trained.
3 x 10-5
Leamiin
learning rate by Adam optimizer
Evaluation steps
1000
evaluation frequency
Batch size
quantity of samples per update
SuperGLUE: SuperGLUE (Wang ct al., 2019)
512
Max input knith
is a new benchmark styled after GLUE with a new set of more difficult NLU tasks. It incorpo- rates improved resources to address the fact that performance GLUE has surpassed the level of non-expert humans, thereby indicating the limited potential for further research progress. In this work, we adopt some of SuperGLUE tasks: CB, BoolQ, WiC, and COPA.
the maximum length for the training
128
Max target longth
masimum knnth generated
16 (TM),SAM), (X5m, CNNDM) spochs that target spirsity moets
Wirm--up spochs
0.01
Reg leaming rite
learning for and A Eo regularization
X the
0.001
weightt layer disill
20 (TQA, SAMSum), (XSum, CNNIDM) epochs For fine- ane-tuning after the pruning
Fine tune epochs
Instruction- Tuning Tasks. We apply batabricks-dolly-te (Conover et al., 2023), Self-Instruct (Wang et al., 2023a), and Vi- cuna (Chiang et al., 2023) for evaluating on instruction-tuning tasks.
Table 9: Hyperparameters for NLU datasets
Hyperparameter
Value & Description
150 (smail)) 100 (midklle). 20 (high) how many epochs trained.
Training spachs
3 x 10
Leaming
learning rate by Adam optimizer 20 (small), 50 (middie)) 500 (high) evaluution frequency
databricks-ddlly-15k.
Evaltation steps
databricks-ddllly-1kk (Conover et al., 2023) is an open-source dataset of instruction- following records generated by thousands of Databricks employees in several of the behavioral categories outlined in the In- structGPT (Ouyang al., 2022), including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
32
Batch
quantity of samples per update
128
Max input length
maximum length for the training
Max target longth
maximum longth to generated 120 (small), 50 (midkle)) 16 (high) upochs that target sparsity meets
Wimt-op spoche
0.01
Reg kaming mic
learning
Ay /. regelarization
Jome Aahe weight layer disiil
0.001
The authors of Self- Instruct (Wang et al., 2023a) have introduced a dataset comprising 52,000 instructions matched with 82,000 instance inputs and outputs. This dataset serves as a resource for fine-tuning language models to improve their adherence to instructions. Additionally, they've ve provided 252 expert-created tasks and instructions designed for user-centric applications, which are used in the human evaluation section of their research. Furthermore, the Self-Instruct dataset includes 50,000 examples from the P3 and Super Natural Instructions datasets for the purpose of facilitating comparisons with existing public datasets.
20
Fine-tune spochs
epochs for fine tuning after the pruning
social media by addressing that previous QA datasets have concentrated on formal text like news and Wikipedia.
Natural Language Understanding Tasks. We apply GLUE (Wang et al., 2018) and SuperGLUE benchmarks (Wang et al., 2019) for evaluating on NLU tasks. While these benchmarks consist of clas- sification datasets, we generate the phrase related to the label instead of the class index. The detailed descriptions and labels of each task are described in Table 7.
Vicuna: Vicuna (Chiang et al., 2023) utilized approximately 70,000 multi-round conversa- tions between users and ChatGPT collected
GLUE: GLUE (Wang et al., 2018) is a collec- tion of datasets for evaluating the performance

---

from the ShareGPT website (Geng et al., 2023), which allows sharing of ChatGPT dialogues, as dataset for fine-tuning. this work, we uti- lize 80 challenging questions used in the Vicuna evaluation.
We would like to request your feedback on the performance of two Al assistants in response to the user instruction and input displayed above.
Please rate the helpfulness. relevance, accuracy. and level of detail of their responses. Fach assistant receives an overall score on scale of to 10, where a higher score indicates better overall performance.
D Hyperparameters
Please first output single line containing only two values indicating the scores for Assistant I and 2, respectively. The two scores separated by space.
In this section, we describe the hyperparame- ter setup of experiments. We report the hyper- parameters that utilized in Table 8 and for NLG and NLU tasks, respectively. We use differ- ent hyperparameter sets for small NLG datasets (TweetQA, SAMSum) and large NLG (XSum, CNN/DailyMail) datasets. Similarly, for NLU tasks, we use different hyperparameters depend- ing on the dataset size. For the small-size NLU datasets (CB, COPA) that the number of sam- ples is smaller than 1,000, we use the hyperpa- rameters (small) described in Table 9. We espe- cially train our model for 150 epochs because the data size is too small to learn the weights for the L0 regularization. We use the hyperparame- ters (middle) described in Table 9 for the middle- size NLU datasets (MRPC, RTE, STS-B, CoLA, WIC, BOOLQ) that the number of samples is more than 1, 000 but less than 10, 000. We use the hy- perparameters (high) described in Table 9 for the large-size NLU datasets (MNLI, QQP, QNLI, SST- that the NLU datasets whose size is larger than 10, 000.
In the subsequent line, please provide comprehen- sive explanation of your evaluation, avoiding any po- tential bias and ensuring that the order in which the responses were presented does not affect your judg- ment.
Figure 9: GPT-4 evaluation prompt.
3090 GPU and AMD EPYC 7502 32-Core Pro- cessor CPU. For each inference prediction, we uti- lized batch size of 32. Additionally, we generated output sequences using a beam size of 4. The time taken for the measurements included all decoding steps until completion.
Table 10: Comparison with Tay et al. (2021) using same decoder layer number (# DL) on TweetQA (TQA) and SAMSum (SAM).
DL2
DL.1
DL6
DL
TQA
SAM
TQA
TQA
Thy at (2001)
51,16
39.30
51.67
40.96
51.68
41.07
53.67
41.18
MASH
16.00
38.90
85.11
41.37
56.92
42233
58.02
42,11
G Additional Experiments
E Instruction-tuning Details
G.1 Comparison with Efficient-T5
In the evaluation process of GPT-4, feedback is solicited by instructing the model compare its generated responses with the authentic, reference answers and assign numerical score ranging from to 10 to each response. Drawing upon the method- ology outlined by Gu et al. (2023), we calculate the ratio between the cumulative scores assigned to the model's responses and those of the ground truth answers. Further details regarding the specific prompt employed for this evaluation are presented Figure 9.
We compare our algorithm, NASH, with models designed to have a shallow decoder depth orig- inally from the pre-training stage, as proposed by Tay et al. (2021). In our evaluation, we exam- ine the performance of our method on two tasks, namely TweetQA and SAMSum, using 2, 4, 6, and 8 decoder layers. As shown in Table 10, NASH demonstrates superior performance in most cases. This result is noteworthy as our method can con- struct a small yet effective model without requiring any costs to make the small pre-trained language model.
F Speedup Evaluation Metric
G.2 Results on NLU Tasks
To measure the inference speed, we conducted inference predictions for each dataset and exam- ined configuration using the PyTorch (Paszke et al., 2019) compiled function. This was done on a sin- gle server equipped with a NVIDIA GeForce RTX
We also compare NASH to the bascline methods on the GLUE and SuperGLUE benchmarks, which are focused on NLU tasks. Since these tasks in- volve relatively longer input sequences and shorter

---

(Buper
BugerBLUEE
60
55
Actumaay
70
70
3.0
2.0
Speedup
Speedup
Speedup
Speedup
- Ts. Base
85% T5-Bade
CaFi-T5
NASH-TS
T5-5mal
Figure 10: Accuracy vs. speedup on various natural language understanding tasks (CoLA, RTE, BoolQ and CB). We compare NASH on 220M T5-Base against CoFi-T5 on Base and 60M T5-Small. On all datasets, NASH is able to outperform CoFi-T5.
The summary Figure which compares understanding accuracy and latency speedup of NASH against other acceleration methods on GLUE (Wang al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks. The number of parameters for all models is around 60M, except for TS-Base. For NASH, we apply two layers of decoder sub-network. The best results of sharing the dataset are highlighted bold.
Table II:
Task
GLUE
SuperGLUE
Dataset
QNLI
SST-2
CoLA
MRPC
RTE
STS-B
BoolQ
WiC
COPA
CB
T5-Base (Teacher)
93.26
95.53
63.14
89.71
78.70
90.85
79.45
72.57
70.00
91.07
T5 75-lmlll
91.03
92.32
44,93
88.73
68.95
89.27
75.87
69.28
55.00
82.14
Speedup
2.5x
2.8x
2.5x
2.4x
2.3x
2.6x
3.0x
2.0x
2.0x
2.6x
CoFi-T5"
87.88
91.28
42.92
75.74
66.96
86.49
78.69
68.97
65.00
87.50
Speedup
2.1x
2.1x
2.0x
2.1x
1.7x
1.9x
2.0
2.0x
2.2x
2.2x
NASH (Ours)
91.19
92.74
56.56
86.27
76.17
90.39
79.51
71.16
69.00
89.29
Speedup/"
3.0x
2.8%
3.0%
3.3x
3.0x
2.9
2.9x
2.6x
3.0x
2.7x
output sequences compared to NLG tasks, our pro- posed method exhibited less effectiveness. How- ever, NASH still demonstrates superiority over the baselines, as depicted in Figure 10. It is impor- tant to note that the performance of our method remains robust across different compression rates. We also provide detailed performance results of our proposed method for the full GLUE and Su- perGLUE benchmarks in the Appendix G.2 Ta- ble 11. The results demonstrate the effectiveness of our proposed NASH method in achieving high output quality while significantly improving in- ference speed. The superior performance of NASH across both GLUE and SuperGLUE benchmarks highlights its potential an efficient acceleration method for NLU tasks as well as for NLG tasks.


Reviews:
[{"review_id": "KOHkJ7qwN9", "reviewer": "Reviewer_7MBx", "summary": "", "questions": "The Sec.\"Preliminary,\" covers Transformers and Structured Pruning, both of which discuss the work of others. In what aspects does the author's improvement primarily manifest across these two parts?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged that you found our paper well-organized, clear motivation and methods, and our experimental results comprehensive. The answers for your concerns are addressed as follows.\n\n***Q1-1. Evaluation of shallow decoder on complex scenarios.***\n\n**A1-1**.   \n**Summary:**  \n\nAs described in Figure 2 and Section 3, we figured out that the most important factor for the generation quality is the sparsity of the encoder network, and the number of the decoder layers is relatively unimportant. \n\nTo address your question, we additionally conduct experiments on two complex scenarios: (1) multi-task scenario with seven tasks and (2) instruction-tuning task. ***The results on such complex scenarios demonstrate that our finding does not originate from the low difficulty of target tasks, but validates generally.*** The detailed experimental setup and results are below.\n\n**Detailed Results:**\n\n**(1) Multi-task scenario:**   \n We consider the multi-task scenario that includes seven tasks (BoolQ, TweetQA, SAMSum, CoLA, STS-B, SST-2, WiC). We applied our method on multiple downstream tasks in the training stage, and evaluated the model on each individual task. Following settings of generation tasks, we set sparsity ratio as 30% for encoder, and set 2, 3, and 4 layers for decoder layer. NASH with 2, 3, or 4 layers outperforms T5-small in terms of both generation performance and inference speed.\n\n|          | BoolQ (Acc) | (Speedup) | TweetQA (Meteor) | (Speedup) | SAMSum (Rouge-L) | (Speedup) | CoLA (Mcc) | (Speedup) | STS-B (Corr) |(Speedup) | SST-2 (Acc) | (Speedup) | WiC (Acc) | (Speedup) |\n|----------|:---------------:|:-----:|:--------------------:|:-----:|:--------------------:|:-----:|:--------------:|:-----:|:----------------:|:-----:|:---------------:|:-----:|:-------------:|:-----:|\n| T5-base  |      77.03      | ×1.00 |         57.65        | ×1.00 |         40.93        | ×1.00 |      52.56     | ×1.00 |       87.86      | ×1.00 |      92.78      | ×1.00 |     66.14     | ×1.00 |\n| T5-small |      72.02      | ×1.98 |         50.45        | ×2.06 |         39.50        | ×2.32 |      20.52     | ×2.14 |       86.05      | ×2.06 |      91.51      | ×2.71 |     65.20     | ×2.64 |\n| NASH 2   |      76.82      | ×2.66 |         51.69        | ×4.67 |         39.02        | ×5.36 |      43.38     | ×3.46 |       86.79      | ×3.15 |      91.63      | ×2.71 |     65.20     | ×2.64 |\n| NASH 3   |      76.94      | ×2.28 |         52.21        | ×3.46 |         39.85        | ×4.14 |      47.13     | ×2.73 |       87.67      | ×2.57 |      91.86      | ×2.38 |     66.46     | ×2.33 |\n| NASH 4   |      77.40      | ×2.03 |         53.70        | ×2.81 |         40.40        | ×3.30 |      50.36     | ×2.30 |       87.51      | ×2.21 |      92.66      | ×2.08 |     65.52     | ×2.02 |\n\n**(2) Instruction-tuning:**   \n We consider the instruction-tuning scenario with the databricks-dolly-15k dataset. We split the total dataset into 14k of train samples,1k of evaluation samples. We evaluate the trained model (with 14 of train samples) on three instruction-following datasets to check the generalizability of model across the different datasets: (1) 1k dolly evaluation samples; (2) SelfInst [1]; and (3) Vicuna evaluation [2] datasets.\n\n|          | Dolly Eval (ROUGE-L) | (ms/example)  | (speedup) | SelfInst (ROUGE-L) | (ms/example)  | (speedup) | Vicuna Eval (ROUGE-L) | (ms/example)  | (speedup) |\n|----------|:----------:|:------:|:-----:|:--------:|:------:|:-----:|:-----------:|:------:|:-----:|\n| T5-base  |    32.53   | 104.57 | ×1.00 |   14.20  | 113.39 | ×1.00 |    20.47    | 146.52 | ×1.00 |\n| T5-small |    29.29   | 53.408 | ×1.96 |   11.67  |  59.19 | ×1.91 |    17.40    |  66.39 | ×2.21 |\n| NASH 2   |    28.09   | 20.307 | ×5.15 |   10.58  |  21.50 | ×5.27 |    20.09    |  24.83 | ×5.90 |\n| NASH 3   |    30.87   | 28.856 | ×3.62 |   11.91  |  29.92 | ×3.79 |    21.83    |  34.51 | ×4.24 |\n| NASH 4   |    31.43   | 39.212 | ×2.67 |   13.01  |  38.53 | ×2.94 |    21.07    |  46.29 | ×3.17 |\n\nThrough the results shown in above complex scenarios, we strongly argue that the generation performance of encoder-decoder models is quite robust to the number of decoder layers. These results contradict conventional insights and disprove that our findings are not trivial or obvious, but rather very surprising and valuable.\n\n***Q1-2. Analysis for the impact of the number of encoder layers on inference speed***\n\n**A1-2.**   \n**Summary:**\n\nAs shown in Figure 2-4 and Section 3, we discussed the impact of the number of encoder layers as a less important factor based on our experimental results. The main reasons that we referred but did not emphasize the relationship between the number of encoder layers and the inference speedup are twofold: Firstly, depth pruning on the encoder networks is not effective for the inference speedup (see Figure 2, 4); Secondly, applying high-sparsity depth pruning to the encoder networks is impractical, given the crucial role played by the encoder network in maintaining performance (see Figure 2 and Finding 3.2). Considering the trade-off between small advantage (inference speedup) and the enormous disadvantage (performance degradation), we decided to exclude the high-sparsity depth pruning on the encoder network. More detailed description are below:\n\n**Detailed description:**\n\n*Firstly*, as depicted in Figure 2, although we conducted pruning on the encoder networks with sparsity up to 95%, the maximal speedup gain was ×1.3. This is because, the most of processing time is spent on the autoregressive generation process as summarized in Figure 4, even though we conducted experiments on NLU tasks which have relatively larger proportional of encoder processing time.\n\n*Secondly*, as shown in the first row of Figure 2, the sparsity of the encoder network is the primary factor for maintaining the output quality of encoder-decoder models, hence, we cannot enlarge the sparsity of encoder networks. In contrast to the recent literature concerning encoder-only models [3], it is surprising that the encoder network of encoder-decoder models are vulnerable to the pruning even though the sparsity is not severely high.\n\nTo address this phenomenon, we conjecture that the generation performance of the encoder-decoder model depends largely upon the encoded context rather than the depth of the decoder network. The similar trends discovered in early-exiting frameworks [4,5] that only add the multiple classifiers in intermediate decoder layers.\n\nMoreover, when the sparsity is moderate (\\~60% sparsity), we observe that the difference between pruning encoder w/ and w/o layer are negligible in terms of both inference speed and generation quality. This result is consistent with the previous study [3] in the structured pruning on the encoder-only model that “in the moderate regime, the number of the layers does not affect the model performance and inference speedup”. Although there is a small amount of difference in the high sparsity regimes (60~95%), we can say that such difference is meaningless because of their low feasibility and useless generation performances.\n\n***Q2. Suggestion for adding a notation table.***\n\n**A2.** Thank you for your constructive comment. To improve the clarity, we will add a following notation table in the finalized version. \n\n|        Type       |      Notation      |                                Description                               |\n|:-----------------:|:------------------:|:------------------------------------------------------------------------:|\n|  Layer selection  |       $L_{s}$      |                      a set of selected layer indices                     |\n|                   |       $d_{s}$      |        the number of selected layers for the decoder depth pruning       |\n|  Model components |  $H_{dec, s}^{l}$  | a hidden representation of the $l$ th student decoder layer              |\n| L0 regularization | $z_{head}^{(i,j)}$ | a mask for a $j$ th attention head of an $i$ th layer                    |\n|                   | $z_{int}^{ (i,j)}$ | a mask for a $j$ th dimension of feed-forward network of an $i$ th layer |\n|                   |      $\\hat s$      |  current sparsity                                                        |\n|                   |         $t$        |  the target sparsity we set                                              |\n|                   |    $\\mathcal{R}$   |  the L0 regularization loss (pruning loss)                               |\n\n***Q3. Request for absolute inference time (specific computational complexity).***\n\n**A3.** Thank you for your constructive comment. As it is not possible to post a chart or graph in OpenReview, we post a table reporting the absolute values of inference time. We reported the absolute inference time for T5 (base, small for Figure 5, 7 and mini for Figure 5), CoFi-T5 with sparsity of 60%-90%, and NASH-T5 with decoder layers of 2-8. We use 30% of encoder sparsity on NASH.\n\n- Absolute values for Figure 5 (milliseconds/example)\n\n|               | Original T5 |       |      | CoFi-T5 (Sparsity) |      |       |       | NASH-T5 (# decoder layers) |      |      |      |      |\n|---------------|:-----------:|:-----:|:----:|:------------------:|:----:|:-----:|:-----:|:--------------------------:|:----:|:----:|:----:|:----:|\n|               |     base    | small | mini |         90%        |  80% |  70%  |  60%  |              2             |   3  |   4  |   6  |   8  |\n| TweetQA       |     9.93    |  5.16 | 3.68 |        6.25        | 7.00 |  8.75 |  8.93 |            2.47            | 3.33 | 3.94 | 5.42 | 8.66 |\n| XSum          |     41.1    |  21.0 | 17.2 |        34.0        | 36.8 |  37.4 |  40.3 |            11.0            | 17.0 | 25.1 | 30.1 | 36.0 |\n| SAMSum        |     62.4    |  29.2 | 22.9 |        43.0        | 50.7 |  54.3 |  61.8 |            11.3            | 14.7 | 20.9 | 32.2 | 42.1 |\n| CNN/DailyMail |    105.7    |  50.1 | 36.2 |        65.1        | 84.4 | 102.6 | 104.6 |            21.5            | 31.3 | 39.3 | 59.7 | 78.5 |\n\n\n- Absolute values for Figure 7 (milliseconds/example)\n\n|       | Original T5 |       | CoFi-T5 (Sparsity)     |  |      |      | NASH-T5 (# decoder layers)     |      |  |      |      |\n|:-----:|:-----------:|:-----:|:----:|:------------------:|:----:|:----:|:----:|:----:|:--------------------------:|:----:|:----:|\n|       |     base    | small |  90% |         80%        |  70% |  60% |   2  |   3  |              4             |   6  |   8  |\n| CoLA  |     4.61    |  1.84 | 2.30 |        2.66        | 3.14 | 4.14 | 1.54 | 2.26 |            2.72            | 3.64 | 4.28 |\n| RTE   |     4.52    |  2.00 | 2.10 |        2.56        | 3.18 | 4.15 | 1.54 | 2.26 |            2.67            | 3.31 | 4.38 |\n| BoolQ |     3.79    |  1.48 | 1.75 |        2.33        | 2.71 | 3.26 | 1.40 | 1.81 |            2.19            | 2.63 | 3.51 |\n| CB    |     4.59    |  1.77 | 2.13 |        2.83        | 3.30 | 3.97 | 1.70 | 2.21 |            2.67            | 3.20 | 4.28 |\n\nIn the final version, we would like to depict the above values on graphs.\n\n***Q4. Additional comparison with recent SOTA.***\n\nPreviously, research on the structured pruning of encoder-decoder language models was limited. Therefore, we initially compared our method to a smaller baseline model and applied the SOTA encoder pruning method to the encoder-decoder model for reference.\nHowever, in response to your concerns, we carried out an additional experiment. In this experiment, we applied our method to the BART-base model for the CNN/DM summarization task, directly comparing it to the method proposed in the very recent paper [6].\nAs illustrated in the table below, we ensured that the model sizes of both methods are similar. In this context, our method demonstrated superior performance, surpassing the method proposed in [6], in terms of generation quality. Our NASH-BART-2 which only uses two decoder layers outperforms the method proposed in [6](with 27% sparsity), despite using smaller parameters. Moreover, it is difficult to directly compare because there is no code repository of [6], but we achieve more than three times while [6] achieves maximially 1.5 times the reported speedup on the GPU.\n\n\n| CNNDM    | BART-base | NASH-BART-2 | NASH-BART-3 | [6] (Sparsity 50%) | [6] (Sparsity 27%)  |\n|----------|-----------|-------------|-------------|--------------------|---------------------|\n| Rouge-L  | 41.68     | 40.87       | 41.14       | 39.91              | 40.39               |\n| Speedup  | ×1.00     | ×3.18       | ×2.11       | Maximally ×1.5     | Maximally ×1.5      |\n| # Params | 139M      | 80M         | 89.8M       | 70.9M              | 102.4M              |\n\n\n***Q5. Ablation studies on the feasibility with considering factors such as task complexity, data volume, real-time requirements, and model robustness.***\n\n**A5.**   \n**Summary:**\n\nTo verify the feasibility of our proposed approaches, the results of ablation study we provided in our paper are: (1) task types (4 NLG tasks, 10 NLU tasks; see Table 1, Figure 5, Figure 7, Table 8); (2) model configuration robustness (feasibility on deeper decoder network; see Table 4), and (3) Comparison with efficient-T5 (see Table 9). \n\nTo intrigue the reviewer 7MBx, we additionally conduct the experiments for (1) task complexity; (2) data volume; (3) absolute value of processing time. Moreover, we emphasize the results for (4) model type. All results in (1) - (4) support that our proposed NASH has a high level of feasibility. The detailed experimental setup and results are below.\n\n**Detailed Results:**\n\n**(1) Task complexity:** \n\nFor the task complexity, we conducted the multi-task experiments with the number of tasks three and seven. We applied our method on multiple downstream tasks in the training stage, and evaluated the model on each individual task. All experiments were conducted on T5-base, T5-small, and NASH-T5. As we already reported the absolute inference time in A3, we only report the relative inference speedup.\n\n- Multi-task learning using 3 downstream tasks (BoolQ, TweetQA and SAMSum)\n\n- Multi-task learning using 7 downstream tasks (BoolQ, TweetQA, SAMSum, CoLA, STS-B, SST-2, WiC)\n\nAs the number of tasks increases (see BoolQ, TweetQA, SAMSum), the decline in performance is very large in T5-small, while the NASH-T5-base are robust. This means that our proposed method is more usable in the complex scenario.\n\n- Multi-task learning using 3 downstream tasks (BoolQ, TweetQA, and SAMSUM)\n\n|          | BoolQ (Acc) | (Speedup) | TweetQA (Meteor) | (Speedup) | SAMSum  (Rouge-L) | (Speedup) |\n|----------|:-----------------:|:-----:|:----------------------:|:-----:|:----------------------:|:-----:|\n| T5-base  |       78.23       | ×1.00 |          58.39         | ×1.00 |          41.12         | ×1.00 |\n| T5-small |       71.41       | ×1.96 |          50.70         | ×2.00 |          39.47         | ×2.22 |\n| NASH 2   |       77.52       | ×2.65 |          53.03         | ×4.82 |          39.32         | ×4.84 |\n| NASH 3   |       77.40       | ×2.28 |          53.12         | ×3.68 |          40.59         | ×3.74 |\n| NASH 4   |       76.73       | ×1.96 |          55.99         | ×2.75 |          40.50         | ×2.93 |\n\n\n- Multi-task learning using 7 downstream tasks (BoolQ, TweetQA, SAMSUM, CoLA, STS-B, SST-w, and WiC)\n\n|          | BoolQ (Acc) | (Speedup) | TweetQA (Meteor) | (Speedup) | SAMSum (Rouge-L) | (Speedup) | CoLA (Mcc) | (Speedup) | STS-B (Corr) |(Speedup) | SST-2 (Acc) | (Speedup) | WiC (Acc) | (Speedup) |\n|----------|:---------------:|:-----:|:--------------------:|:-----:|:--------------------:|:-----:|:--------------:|:-----:|:----------------:|:-----:|:---------------:|:-----:|:-------------:|:-----:|\n| T5-base  |      77.03      | ×1.00 |         57.65        | ×1.00 |         40.93        | ×1.00 |      52.56     | ×1.00 |       87.86      | ×1.00 |      92.78      | ×1.00 |     66.14     | ×1.00 |\n| T5-small |      72.02      | ×1.98 |         50.45        | ×2.06 |         39.50        | ×2.32 |      20.52     | ×2.14 |       86.05      | ×2.06 |      91.51      | ×2.71 |     65.20     | ×2.64 |\n| NASH 2   |      76.82      | ×2.66 |         51.69        | ×4.67 |         39.02        | ×5.36 |      43.38     | ×3.46 |       86.79      | ×3.15 |      91.63      | ×2.71 |     65.20     | ×2.64 |\n| NASH 3   |      76.94      | ×2.28 |         52.21        | ×3.46 |         39.85        | ×4.14 |      47.13     | ×2.73 |       87.67      | ×2.57 |      91.86      | ×2.38 |     66.46     | ×2.33 |\n| NASH 4   |      77.40      | ×2.03 |         53.70        | ×2.81 |         40.40        | ×3.30 |      50.36     | ×2.30 |       87.51      | ×2.21 |      92.66      | ×2.08 |     65.52     | ×2.02 |\n\n\n**(2) Data volume:** \n\nFor the data volume, we conducted the experiments on the SAMSum dataset with the number of data points of 1k, 5k, and 14k. Similar to the task complexity experiments, all experiments were conducted on T5-base, T5-small, and NASH-T5. Similar to above, we report the relative inference speedup.\n\n- SAMSum with 1k training samples\n\n|         | T5-base | T5-small | NASH-T5 (2 layers) | NASH-T5 (3 layers) | NASH-T5 (4 layers) |\n|:-------:|:-------:|:--------:|:------------------:|:------------------:|:------------------:|\n| ROUGE-L |  39.00  |   35.60  |        30.44       |        36.21       |        36.49       |\n| SpeedUp |  ×1.00  |   ×2.12  |        ×5.19       |        ×3.97       |        ×3.07       |\n\n\n- SAMSum with 5k training samples\n\n|         | T5-base | T5-small | NASH-T5 (2 layers) | NASH-T5 (3 layers) | NASH-T5 (4 layers) |\n|:-------:|:-------:|:--------:|:------------------:|:------------------:|:------------------:|\n| ROUGE-L |  40.55  |   38.77  |        39.41       |        41.88       |        41.89       |\n| SpeedUp |  ×1.00  |   ×2.15  |        ×5.13       |        ×4.23       |        ×3.14       |\n\n- SAMSum with 14k training samples\n\n|         | T5-base | T5-small | NASH-T5 (2 layers) | NASH-T5 (3 layers) | NASH-T5 (4 layers) |\n|--------:|:-------:|:--------:|:------------------:|:------------------:|:------------------:|\n| ROUGE-L |  43.33  |   38.14  |        38.89       |        41.09       |        41.34       |\n| SpeedUp |  ×1.00  |   ×2.14  |        ×5.53       |        ×4.24       |        ×2.99       |\n\n\nNASH-T5 beats the T5-small for all setups in terms of both model performance and inference speed, except for the model performance of NASH-T5 with two layers and 1k data points. Although our proposed method is more effective with a large number of data points, the results depict that our proposed method is quite robust to the number of data points. \n\n**(3) Absolute value of processing time:**\n\nWe understand the term “real-time requirements” to mean “the absolute values of inference time”. We have reported such values in A3. Please refer to the corresponding answer.\n\n**(4) Model types:**\n\nWe understand the term “model robustness” as “consistent performance according to the different model configuration”, and we have reported such values in Table 4 and Section 5.3. The results show that our proposed method NASH has robustness in terms of output performance even with shallower decoder than the original models (i.e., unpruned models) Moreover, since the original model has deeper decoder networks, NASH can achieve higher inference speedup.\n\nTo address your concern, we additionally conducted experiments on BART, which is one of the encoder-decoder language models. The results demonstrate that our proposed method also works well on BART, which means the robustness to the model types.\n\n|         |  BART |        |          | NASH-BART |          |\n|:-------:|:-----:|:------:|:--------:|:---------:|:--------:|\n|         | large |  base  | 2 layers |  3 layers | 4 layers |\n| ROUGE-L | 43.24 |  42.39 |   40.47  |   42.33   |   42.85  |\n| Speedup | ×1.00 | ×2.05  |  ×5.56   |   ×3.38   |  ×2.43   |\n\n\n***Q6. Authors’ improvement related to contents in preliminary section***\n\n**A6.** The preliminary section, we introduced the background knowledge to understand our works (or contributions) which are described in Section “Experimental Motivations”, Section “Narrow Encoder and Shallow Decoder”, and Section “Experiments”.\n\nThe main contributions of our paper are: (1) While the existing structured pruning methods were mainly considered on encoder-only models [3,7,8] or decoder-only models [9], we conduct systematic analysis of structured pruning on encoder-decoder models. (2) Based on findings from our analyses, we design a simple yet effective method which fully exploits the property of different behaviors of encoder and decoder networks during structured pruning. None of these contributions were present before, which encompasses the previously mentioned works outlined in the “Preliminary” section. \n\n***References***\n\n[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL. 2023   \n[2] Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023   \n[3] Structured Pruning Learns Compact and Accurate Models. ACL. 2022   \n[4] Depth-Adaptive Transformer. ICLR. 2020   \n[5] Confident Adaptive Language Modeling. NeurIPS. 2022  \n[6] Structured Pruning for Efficient Generative Pre-trained Language Models. ACL findings. 2023   \n[7] Block Pruning For Faster Transformers. EMNLP. 2021   \n[8] A Fast Post-Training Pruning Framework for Transformers. NeurIPS. 2022   \n[9] What Matters In The Structured Pruning of Generative Language Models?. ArXiv. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The illustration of motivation and method is clear. The ablation\nstudy is reasonably thorough and acceptable results are claimed.", "reasons_to_reject": "1.The author concludes that the number of decoder layers is the primary factor for accelerating inference speed. The depth of the decoder affects the model's complexity and capability, as deeper decoders can generally capture more intricate semantics and relationships, thereby enhancing the quality of tasks such as generation and translation. The author should quantitatively assess and present experimental results regarding semantic relationships in complex scenarios. Additionally, the number of encoder layers can also impact the inference speed to varying degrees, yet the author fails to analyze why this aspect has been excluded.\n2.The math presentation can be significantly improved. I think to add a notation table can help.\n3.Compared to the SOTA methods, the proposed approach showcases its advantages in various aspects, such as reduced computational complexity and enhanced efficiency. The article should include charts and graphs to visually depict the method's strengths, including its specific computational complexity.\n4.The author should incorporate more comparative results with SOTA methods in the experimental tables.\n5.The author lacks a qualitative analysis of the feasibility of this approach considering factors such as task complexity, data volume, real-time requirements, and model robustness.", "questions_to_authors": "The Sec.\"Preliminary,\" covers Transformers and Structured Pruning, both of which discuss the work of others. In what aspects does the author's improvement primarily manifest across these two parts?", "paper_topic_and_main_contributions": "This study explores impact of structured pruning on encoder-decoder models, revealing the importance of decoder layers for inference speed and the role of sparse pruning in enhancing encoder network quality. Building on these insights, the NASH framework is introduced, effectively refining encoder and decoder networks. Experiments across diverse tasks confirm NASH's success in improving both computational efficiency and output quality. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z1pB5pNT03", "reviewer": "Reviewer_h8AR", "summary": "", "questions": "1. Why do the authors choose L0 regularization to instantiate their method? Does it provide any advantage? How about the gradient-based pruning methods?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments, and we are encouraged that you pointed out the simplicity and effectiveness of our proposed method based on our rigorous analysis. The answers for your concerns are addressed as follows.\n\n***Q1. The main contributions of observations in Section 3***\n\n**A1.**   \n**Summary:** \n\nThe observations in our paper provide (1) the ***counter-intuitive perspective on the recent direction of structured pruning methods*** [1-5] ***(see First in detailed description)*** and (2) ***general understanding to the encoder-decoder model architecture which is still controversial*** [8,9] ***(see Second in detailed description)***. Hence, they are not trivial and obvious, rather valuable and explainable. Moreover, we provide exciting results on the multi-task and instruction-tuning datasets, that our proposed method can be possible to apply for making general language models. We expect such additional results to intrigue you. The detailed descriptions are demonstrated below.\n\n**Detailed Descriptions:** \n\n*Firstly*, while the recent studies [1-5] showed fine-grained structured pruning (prune individual head in MHA or individual dimension in FFN) [1-4] can achieve both higher model performance and inference speedup than the traditional layer-dropping based methods [6, 7], we empirically showed that the layer-dropping methods can achieve competitive performances as fine-grained structured pruning [3], while highly increasing inference speed.\n\n*Secondly*, while our discovery of the number of decoder layers somewhat consistent with traditional methods [8], this discovery can be shown as not only revisiting the superiority of such a traditional approach which is conflicts with more recent argument [9] that are deep and narrow decoder network is computationally efficient and achieves higher performance, but also suggest more general perspective. We also figured out that the moderate level of sparsity imposed on the encoder slightly improves the quality although it is memory-efficient.\n\nTo further demonstrate that our findings are applicable to various settings, we conducted experiments on two complex scenarios: (1) multi-task scenario (2) instruction-tuning scenario. We first describe our experimental setup and then describe the experimental results.\n\n**(1) Multi-task scenario:**   \n We consider the multi-task scenario that includes seven tasks (BoolQ, TweetQA, SAMSum, CoLA, STS-B, SST-2, WiC). We applied our method on multiple downstream tasks in the training stage, and evaluated the model on each individual task. Following settings of generation tasks, we set sparsity ratio as 30% for encoder, and set 2, 3, and 4 layers for decoder layer. NASH with 2, 3, or 4 layers outperforms T5-small in terms of both generation performance and inference speed.\n\n|          | BoolQ (Acc) | (Speedup) | TweetQA (Meteor) | (Speedup) | SAMSum (Rouge-L) | (Speedup) | CoLA (Mcc) | (Speedup) | STS-B (Corr) |(Speedup) | SST-2 (Acc) | (Speedup) | WiC (Acc) | (Speedup) |\n|----------|:---------------:|:-----:|:--------------------:|:-----:|:--------------------:|:-----:|:--------------:|:-----:|:----------------:|:-----:|:---------------:|:-----:|:-------------:|:-----:|\n| T5-base  |      77.03      | ×1.00 |         57.65        | ×1.00 |         40.93        | ×1.00 |      52.56     | ×1.00 |       87.86      | ×1.00 |      92.78      | ×1.00 |     66.14     | ×1.00 |\n| T5-small |      72.02      | ×1.98 |         50.45        | ×2.06 |         39.50        | ×2.32 |      20.52     | ×2.14 |       86.05      | ×2.06 |      91.51      | ×2.71 |     65.20     | ×2.64 |\n| NASH 2   |      76.82      | ×2.66 |         51.69        | ×4.67 |         39.02        | ×5.36 |      43.38     | ×3.46 |       86.79      | ×3.15 |      91.63      | ×2.71 |     65.20     | ×2.64 |\n| NASH 3   |      76.94      | ×2.28 |         52.21        | ×3.46 |         39.85        | ×4.14 |      47.13     | ×2.73 |       87.67      | ×2.57 |      91.86      | ×2.38 |     66.46     | ×2.33 |\n| NASH 4   |      77.40      | ×2.03 |         53.70        | ×2.81 |         40.40        | ×3.30 |      50.36     | ×2.30 |       87.51      | ×2.21 |      92.66      | ×2.08 |     65.52     | ×2.02 |\n\n**(2) Instruction-tuning:**   \nWe consider the instruction-tuning scenario with the databricks-dolly-15k dataset. We split the total dataset into 14k of train samples,1k of evaluation samples. We evaluate the trained model (with 14 of train samples) on three instruction-following datasets to check the generalizability of model across the different datasets: (1) 1k dolly evaluation samples; (2) SelfInst [10]; and (3) Vicuna evaluation [11] datasets.\n\n|          | Dolly Eval (ROUGE-L) | (ms/example)  | (speedup) | SelfInst (ROUGE-L) | (ms/example)  | (speedup) | Vicuna Eval (ROUGE-L) | (ms/example)  | (speedup) |\n|----------|:----------:|:------:|:-----:|:--------:|:------:|:-----:|:-----------:|:------:|:-----:|\n| T5-base  |    32.53   | 104.57 | ×1.00 |   14.20  | 113.39 | ×1.00 |    20.47    | 146.52 | ×1.00 |\n| T5-small |    29.29   | 53.408 | ×1.96 |   11.67  |  59.19 | ×1.91 |    17.40    |  66.39 | ×2.21 |\n| NASH 2   |    28.09   | 20.307 | ×5.15 |   10.58  |  21.50 | ×5.27 |    20.09    |  24.83 | ×5.90 |\n| NASH 3   |    30.87   | 28.856 | ×3.62 |   11.91  |  29.92 | ×3.79 |    21.83    |  34.51 | ×4.24 |\n| NASH 4   |    31.43   | 39.212 | ×2.67 |   13.01  |  38.53 | ×2.94 |    21.07    |  46.29 | ×3.17 |\n\nThrough the results shown in above complex scenarios, we strongly argue that the generation performance of encoder-decoder models is quite robust to the number of decoder layers. These results contradict conventional insights and disprove that our findings are not trivial or obvious, but rather very surprising and valuable.\n\n***Q2-1. The advantage of L0 regularization***\n\n**A2-1.** Most of the research in structured pruning domains are mainly conducted experiments on encoder-only models such as BERT or RoBERTa. Among the various approaches in the research direction of encoder-only structured pruning method, recent L0 regularization method has mainly two advantages: Firstly, L0 regularization is easier to control the sparsity precisely. [1,3] We also find that such methods controls well in the low sparsity regime of encoder networks, although it did not work well on severe sparsity of the encoder or decoder networks. Secondly, In the recent work of L0 regularization, CoFi [3], achieved SoTA performance compared to gradient-based method [2,12] or layer-dropping method [6,7]. To exploit such an advantage of L0 regularization shown in encoder-only models, we chose L0 regularization to instantiate our proposed method, NASH.\n\n***Q2-2. Comparison with gradient-based pruning methods?***\n\n**A2-2.** To address your question, we conducted experiments where we replaced our gradual L0 regularization with a gradient-based pruning method. Following [12-15], we compute the importance score of individual attention head and neuron in FFN by using first-order Taylor expansion of gradient information. Due to the less flexibility of gradient-based pruning, we uniformly prune the individual heads (4 heads for all layers) and neurons (~30% dimension of FFN) for all Transformer layers.\n\n|                   |  RTE (Acc) | (Speedup) | SAMSum (ROUGE-L) | (Speedup) |\n|-------------------|:-----:|:-----:|:------:|:-----:|\n| Gradient-based    | 76.17 | ×2.27 |  40.85 | ×2.97 |\n| L0 Regularization | 78.70 | ×2.24 |  41.34 | ×2.99 |\n\nThe results show that NASH with L0 regularization can achieve higher performance than the gradient-based pruning method in both NLU (RTE) and NLG (SAMSum) tasks.\n\n***Q3. Missing reference***\n\n**A3.** Thank you for pointing out the missing reference [16]. We will add it to our revised version. \n\n***References***\n\n[1] Structured pruning of large language models. EMNLP 2020   \n[2] Block Pruning For Faster Transformers. EMNLP. 2021   \n[3] Structured Pruning Learns Compact and Accurate Models. ACL. 2022   \n[4] A Fast Post-Training Pruning Framework for Transformers. NeurIPS. 2022   \n[5] What Matters In The Structured Pruning of Generative Language Models?. ArXiv. 2023   \n[6] Reducing Transformer Depth on Demand with Structured Dropout. ICLR. 2020   \n[7] On the Effect of Dropping Layers of Pre-trained Transformer Models. ArXiv. 2020   \n[8] Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation. ICLR. 2021   \n[9] Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers. ICLR. 2022   \n[10] Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL. 2023   \n[11] Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023   \n[12] DynaBERT: Dynamic BERT with Adaptive Width and Depth. NeurIPS. 2020   \n[13] Pruning convolutional neural networks for resource efficient inference. NIPS. 2017   \n[14] Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. ACL. 2019   \n[15] LLM-Pruner: On the Structural Pruning of Large Language Models. ArXiv. 2023   \n[16] Are sixteen heads really better than one?. NeurIPS. 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Rigorous analysis and extensive experiments.\n2. A simple and effective approach.", "reasons_to_reject": "1. The observations are somewhat trivial and obvious given the previous work.", "questions_to_authors": "1. Why do the authors choose L0 regularization to instantiate their method? Does it provide any advantage? How about the gradient-based pruning methods?", "paper_topic_and_main_contributions": "In this work, the authors provide a rigorous study to explore the dominant factors that trade-off performance and efficiency during structured pruning. The key observations are that reducing the decoder layers is the best option for acceleration while an appropriate sparsity imposed on the encoder slightly improves the quality. Based on these observations, the authors propose NASH, a narrow encoder and shallow decoder pruning strategy. Extensive experiments have demonstrated the effectiveness of NASH. In general, this paper is well-written and I am satisfied with the comprehensive experiments and studies conducted by the authors. However, I find that the conclusion and method are somewhat trivial. For example, the contribution of the decoder layer to efficiency and quality in encoder-decoder models has long been observed, as also pointed out in the paper. I think the observations in the first half of this paper are expected and not very interesting. The proposed method is also straightforward: it looks to me like a direct application of L0 regularization on encoder-decoder models with a targeted granularity (layer for decoder and matrix for encoder).", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Michel, Paul, Omer Levy, and Graham Neubig. \"Are sixteen heads really better than one?.\" Advances in neural information processing systems 32 (2019)."}, "extra_text": {}}, {"review_id": "UyHqAN45Ty", "reviewer": "Reviewer_1C5A", "summary": "", "questions": "1. How the proposed method correspond with structural pruning for decoder-only models?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that you have acknowledged our well-organized paper and well-constructive experiments. The answers for your concerns are addressed as follows.\n\n***Q1. Experiments conducted on any non-T5 model, for example BART or vanilla Transformer.***\n\n**A1.** We conducted experiments on T5 models because they are more popular and widely used across various tasks, such as code generation, medical text understanding, and spell checking [1-4] than other encoder-decoder models like BART [5] and PEGASUS [6].\n\n\nTo address your concern, we conducted supplementary experiments employing another encoder-decoder model, BART, on the SAMSum dataset. Specifically, we compare NASH-BART-large (conducting NASH on BART-large), and BART-base models.\nAs illustrated in the table below, our results with the BART demonstrate consistent performance when compared to the T5 case. Notably, BART NASH 2 exhibits an impressive x5.6 faster inference speed than BART-large, while BART NASH 3 achieves faster inference compared to BART-base, all while maintaining similar output quality.\n\n|         |  BART |        | NASH-BART|           |          |\n|:-------:|:-----:|:------:|:--------:|:---------:|:--------:|\n|         | large |  base  | 2 layers |  3 layers | 4 layers |\n| ROUGE-L | 43.24 |  42.39 |   40.47  |   42.33   |   42.85  |\n| Speedup | ×1.00 | ×2.05  |  ×5.56   |   ×3.38   |  ×2.43   |\n\n***Q2. How does the proposed method correspond with structured pruning for decoder-only models?***\n\n**A2.** \nIn this work, we discussed that the well-performing structured pruning method method for the encoder-only model [7] does not work well for the encoder-decoder models, and optimized the structured pruning method for the encoder-decoder models. Similarly, we believe that further discussion is needed to find a well-functioning structured pruning method for decoder-only models. \n\nIn response to your question, when comparing decoder-only models to encoder-decoder models, the latter can be segmented into two parts. It's noteworthy that these two networks exhibit contrasting properties, as illustrated in Figure 2: while (1) the performances of LMs are vulnerable to the number of encoder layers, but robust to the number of decoder layers, (2) the number of the decoder layers is primary factor of inference speed of the LMs, but the number of the encoder layers is insignificant to inference speedup. Our proposed method fully exploits such property by conducting different pruning strategies on encoder and decoder networks. This allows us to use a maximally 6x smaller decoder network without severe performance degradation but achieving up to 5x faster inference speed.\n\nHowever, directly applying our proposed method to decoder-only models could be impossible or cause severe performance degradation, because such models process both encoding input context and generation outputs unlike encoder-decoder models. One suggested modification is (1) slightly conducting width pruning with moderate sparsity on the language model, similar to our narrow encoder (Section 4.2), then (2) only using layer skipping for auto-regressive generation process while fully utilizing the whole network for processing input context. We believe that such modification could be working on decoder-only models.\n\n***References***\n\n[1] ClinicalT5: A Generative Language Model for Clinical Text. Findings of EMNLP. 2022   \n[2] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. EMNLP. 2021   \n[3] CodeT5+: Open Code Large Language Models for Code Understanding and Generation. ArXiv. 2023   \n[4] ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. TACL. 2022   \n[5] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL. 2020   \n[6] PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. ICML. 2020   \n[7] Structured Pruning Learns Compact and Accurate Models. ACL. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although model pruning is not my primary field, I have to admit that reading the paper was quite smooth. I appreciate the split of preliminary experiments and the methods, since it helps to understand why some decision were made. The evaluation seems to be done properly.\n\nThe paper is well written.", "reasons_to_reject": "The experiments are conducted only using T5 architecture, it would make sense to make ablation on any non-T5 model, for example BART or vanilla transformer.\n\n", "questions_to_authors": "1. How the proposed method correspond with structural pruning for decoder-only models?", "paper_topic_and_main_contributions": "The proposed paper introduces NASH, a novel method to structural pruning for encoder-decoder models. NASH shows superior performance on many NLU and NLG tasks. The paper generalizes some previous insights on training of encoder-decoder models, e.g. that a rule of thumb is \"deep encoder, shallow decoder\".", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UZboJOhJ8L", "reviewer": "Reviewer_XszA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would appreciate your insightful comments based on a thorough inspection of our paper. We are encouraged that you found that our paper is well-motivated from sufficient empirical analysis and proposed algorithm is effective. The answers for your concerns are addressed as follows.\n\n***Q1. The significance and novelty of contributions (width pruning on encoder, depth pruning on decoder)***\n\n**A1.**  \n**Summary:**\n\nIn this work, we provided several observations that are (1) ***counter-intuitive perspective on the recent direction of structured pruning methods*** [1-5] and (2) ***general understanding of the encoder-decoder model architecture which is still controversial*** [6,7]. The significance of our proposed method based on such observations is as follows:\n\n*Firstly*, we believe that reducing the number of decoder layers is not a generally accepted starting point for network pruning. This is because, as we mentioned before, the efficient architectures of encoder-decoder models are still controversial topics. Moreover, before our studies, people could not guarantee to maintain the model performance by using depth pruning. The detailed description is below (see * in detailed description).\n\nSecondly, it is naturally desired to reduce the number of parameters as long as the performance does not deteriorate. Although the structured pruning on the encoder network can cause severe performance degradation (see Figure 2 and Section 3), our well-constructed width pruning scheme generally has three advantages: (1) increasing model performance, (2) slightly improving inference speed, (3) enlarging memory efficiency. While the first two advantages were depicted in our submission (see Section 5.3 and Table 3), we further discussed them in detail below (see ** in detailed description).\n\n**Detailed Description:**\n\nHere, we emphasize the contribution of our proposed method in detail.\n\n(*) There are two reasons why we believe starting with pruning the number of layers in the decoder is unnatural:  \n\n* ***Firstly, the efficient architectures of encoder-decoder models are still controversial topics.*** While our discovery on the number of decoder layers somewhat consistent with traditional methods [6], this discovery can be shown as not only revisiting the superiority of such a traditional approach which is conflicts with more recent argument [7] that are deep and narrow decoder network is computationally efficient and achieves higher performance. Furthermore, our provided findings also suggest a more general perspective which holds on a wider range of NLP tasks (i.e., both NLG and NLU tasks).\n\n* ***Secondly, before our studies, people could not guarantee to maintain the model performance by using depth pruning.*** In the recent research direction, most structured pruning methods [1-5] exploit fine-grained pruning rather than layer-dropping method [8,9] which can cause a larger performance drop. However, we systematically show that depth pruning method is quite usable in the decoder network of encoder-decoder models that is counter-intuitive to recent studies.\n\n(**) Moreover, we believe that our width pruning on the encoder network is also important, as we mentioned above. Our contributions are from three factors: \n\n* Increasing model performance: As depicted in Figure 2 and Section 3, the sparsity of encoder network is the dominant factor for model performance, and simply conducting SOTA structured pruning method [3] can cause severe performance degradation. However, as we can see from Table 3 in our paper, our proposed approach slightly improves the performance compared to without width pruning. Furthermore, we prevent severe performance degradation from naïvely using pruning on the encoder network by guiding to control encoder sparsity which has a significant impact on overall model performance.\n\n* Improving inference speed: As we can see from Table 3 in our paper, our proposed approach also improves the inference speed by about ***5~10%*** compared to w/o encoder pruning as well as increases the model performance. Furthermore, we observed that this ***relative speed gap becomes even more pronounced, increasing up to 15-30%, when we conduct experiments on a CPU*** (see below Table). This is because of the difference in effectiveness for parallel computation between GPU and CPU. However, our overall method still shows the computational efficiency as the proportion of decoder computation is still larger than encoder computation. We believe that this improvement is not insignificant.\n\n>> |                    | SAMSum |       | TweetQA |       |\n>> |--------------------|:------:|:-----:|:-------:|:-----:|\n>> | T5-Base            |  43.33 | ×1.00 |  57.16  | ×1.00 |\n>> | NASH (w/o L0 Reg.) |  41.06 | ×2.50 |  54.73  | ×1.89 |\n>> | NASH (w/ L0 Reg.)  |  41.4  | ×2.86 |  55.08  | ×2.47 |\n\n* Enlarge memory efficiency: It is naturally desirable to reduce as many parameters as possible without raising performance degradation. Combined with depth pruning on decoder network, the proportion of encoder parameters will be increased. As a result, we can achieve relatively high memory efficiency with low encoder sparsity. For example, in the above Table, the numbers of Transformer layers parameters are 112.6M and 88.7M for w/o and w/ L0 regularization, which means that our narrow encoder strategy has higher memory efficiency (***can save about 20% memory*** compared to 10% in w/o depth pruning of decoder network). \n\n***Q2. Possibility of making general purpose language models rather than task-specific language models***\n\n***A2.***   \nTo show the possibility that our proposed method NASH can be used in making general purpose language models, we conducted experiments on two complex scenarios: (1) multi-task scenario (2) instruction-tuning scenario. We first describe our experimental setup and then describe the experimental results.\n\n**(1) Multi-task scenario:**   \n We consider the multi-task scenario that includes seven tasks (BoolQ, TweetQA, SAMSum, CoLA, STS-B, SST-2, WiC). We applied our method on multiple downstream tasks in the training stage, and evaluated the model on each individual task. Following settings of generation tasks, we set sparsity ratio as 30% for encoder, and set 2, 3, and 4 layers for decoder layer. NASH with 2, 3, or 4 layers outperforms T5-small in terms of both generation performance and inference speed.\n\n|          | BoolQ (Acc) | (Speedup) | TweetQA (Meteor) | (Speedup) | SAMSum (Rouge-L) | (Speedup) | CoLA (Mcc) | (Speedup) | STS-B (Corr) |(Speedup) | SST-2 (Acc) | (Speedup) | WiC (Acc) | (Speedup) |\n|----------|:---------------:|:-----:|:--------------------:|:-----:|:--------------------:|:-----:|:--------------:|:-----:|:----------------:|:-----:|:---------------:|:-----:|:-------------:|:-----:|\n| T5-base  |      77.03      | ×1.00 |         57.65        | ×1.00 |         40.93        | ×1.00 |      52.56     | ×1.00 |       87.86      | ×1.00 |      92.78      | ×1.00 |     66.14     | ×1.00 |\n| T5-small |      72.02      | ×1.98 |         50.45        | ×2.06 |         39.50        | ×2.32 |      20.52     | ×2.14 |       86.05      | ×2.06 |      91.51      | ×2.71 |     65.20     | ×2.64 |\n| NASH 2   |      76.82      | ×2.66 |         51.69        | ×4.67 |         39.02        | ×5.36 |      43.38     | ×3.46 |       86.79      | ×3.15 |      91.63      | ×2.71 |     65.20     | ×2.64 |\n| NASH 3   |      76.94      | ×2.28 |         52.21        | ×3.46 |         39.85        | ×4.14 |      47.13     | ×2.73 |       87.67      | ×2.57 |      91.86      | ×2.38 |     66.46     | ×2.33 |\n| NASH 4   |      77.40      | ×2.03 |         53.70        | ×2.81 |         40.40        | ×3.30 |      50.36     | ×2.30 |       87.51      | ×2.21 |      92.66      | ×2.08 |     65.52     | ×2.02 |\n\n**(2) Instruction-tuning:**   \nWe consider the instruction-tuning scenario with the databricks-dolly-15k dataset. We split the total dataset into 14k of train samples,1k of evaluation samples. We evaluate the trained model (with 14 of train samples) on three instruction-following datasets to check the generalizability of model across the different datasets: (1) 1k dolly evaluation samples; (2) SelfInst [10]; and (3) Vicuna evaluation [11] datasets.\n\n|          | Dolly Eval (ROUGE-L) | (ms/example)  | (speedup) | SelfInst (ROUGE-L) | (ms/example)  | (speedup) | Vicuna Eval (ROUGE-L) | (ms/example)  | (speedup) |\n|----------|:----------:|:------:|:-----:|:--------:|:------:|:-----:|:-----------:|:------:|:-----:|\n| T5-base  |    32.53   | 104.57 | ×1.00 |   14.20  | 113.39 | ×1.00 |    20.47    | 146.52 | ×1.00 |\n| T5-small |    29.29   | 53.408 | ×1.96 |   11.67  |  59.19 | ×1.91 |    17.40    |  66.39 | ×2.21 |\n| NASH 2   |    28.09   | 20.307 | ×5.15 |   10.58  |  21.50 | ×5.27 |    20.09    |  24.83 | ×5.90 |\n| NASH 3   |    30.87   | 28.856 | ×3.62 |   11.91  |  29.92 | ×3.79 |    21.83    |  34.51 | ×4.24 |\n| NASH 4   |    31.43   | 39.212 | ×2.67 |   13.01  |  38.53 | ×2.94 |    21.07    |  46.29 | ×3.17 |\n\nThrough the results shown in above scenarios, we believe that our proposed NASH can be utilized to make more efficient general purpose language models.\n\n***References***\n\n[1] Structured pruning of large language models. EMNLP 2020   \n[2] Block Pruning For Faster Transformers. EMNLP. 2021   \n[3] Structured Pruning Learns Compact and Accurate Models. ACL. 2022   \n[4] A Fast Post-Training Pruning Framework for Transformers. NeurIPS. 2022   \n[5] What Matters In The Structured Pruning of Generative Language Models?. ArXiv. 2023   \n[6] Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation. ICLR. 2021   \n[7] Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers. ICLR. 2022   \n[8] Reducing Transformer Depth on Demand with Structured Dropout. ICLR. 2020   \n[9] On the Effect of Dropping Layers of Pre-trained Transformer Models. ArXiv. 2020   \n[10] Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL. 2023   \n[11] Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. sufficient empirical evidence is presented for motivating the proposal.\n\n2. it is the first attempt to propose a systematic strategy to prune encoder-decoder models.\n\n3. extensive ablation study demonstrates the effectiveness of the proposed algorithm.", "reasons_to_reject": "1. the separation or the improvement of increasing the sparsity of the encoder over solely pruning the decoder doesn't seem significant to me, which then makes the contribution of this submission uncertain, at least to me.  It is generally understood that the number of layers in a decoder-only model plays a large role in the latency, therefore, naturally people would start with pruning the number of layers in the decoder.\n\n2. I wonder if it is possible to distill a pre-trained language model using this proposed strategy to achieve a much simpler but also general-purpose language model rather than a task-specific language model. imo, that'd be much more beneficial to the community since it would essentially reduce the latency for all downstream computation, including finetuning and zero-shot inference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "the submission proposed a pruning strategy for encoder-decoder models that emphasizes increasing the sparsity of the encoder, and reducing the number of layers in the decoder. The proposal was founded on empirical evidence of the tradeoff among the model size, accuracy, and latency. In addition, improvements are shown in extensive experimental results. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores structured pruning techniques for encoder-decoder transformer-based pre-trained models and uncovers novel insights into pruning these models.

**Pros**:
- The approach is well-grounded, as it begins with preliminary experiments to assess the significance of each module in terms of performance and inference speedup.
- The proposed approach outperforms merely applying previous methods without thoughtful adaptation.
- The experiments conducted are thorough and comprehensive, encompassing a range of tasks, including standard fine-tuning tasks and instruction tuning.
- In summary, I believe this work lays a strong foundation for further investigation into effective pruning approaches for encoder-decoder models.

**Cons**:
- Some reviewers express concerns regarding the novelty of the work, as it primarily involves making straightforward adaptations to prior research like CoFi.
Decision: Accept-Findings