To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency
Year: 2023

Abstract:
Sequence-to-sequence language models can be used to produce abstractive summaries which are coherent, relevant, and concise. Still, model sizes can make deployment in latency-sensitive or web-scale implementations difficult. This paper studies the relationship between model size, structured pruning, inference efficiency, and summarization accuracy on widely used summarization datasets. We show that model accuracy is tied to the encoder size while inference efficiency is connected to the decoder. Using asymmetric pruning can lead to nearly 3x improvement in inference latency with 1 point loss in Rouge-2. Moreover, we find both the average degradation and the role of asymmetry to be consistent across model sizes and variations in datasets. We release our code1, training regimes, and associated model 2 for broad usage to encourage usage and experimentation.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 91–109 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The application of sequence-to-sequence language models has become an important tool for natural language processing tasks such as machine translation (Sutskever et al., 2014), audio transcription (Radford et al., 2022), and abstractive summarization (Raffel et al., 2020). Sequence-to-sequence models effectively turn each of these aforementioned tasks into two-step problems: extraction and generation, and heavily condition the generation on the input. Besides ensuring on-topic responses sequence to sequence models have the added benefit of being able to map inputs to targets with varying lengths
∗ Corresponding author: dcampos3@illinois.edu 1https://github.com/spacemanidol/Efficient-Web-Scale-
Absractive-Summarization 2https://huggingface.co/spacemanidol
Accuracy vs. Inference Speed
and modalities in ways encoder or decoder-only systems cannot. When used for abstractive summarization, sequence-to-sequence modeling has two steps, extraction using the encoder and generation using the decoder, which usually involves repeated execution until an end-of-sequence token is emitted. Since the encoder runs once on the input (Sutskever et al., 2014) its cost of execution is proportional to the batch size. The cost of decoder execution can be highly variable based on the generation length (Tay et al., 2021). Despite the broad study of sequence-to-sequence models (Raffel et al., 2020) and how they compress (Li et al., 2022), the role of model symmetry as applied to inference efficiency and model accuracy
91
is lacking. Recent advances in scaling language models have led to a wide study on scaling laws as applied to language model performance (Kaplan et al., 2020), training data size (Hoffmann et al., 2022), machine translation (Henighan et al., 2020), and even reinforcement learning (Neumann and Gros, 2022). We build on this work and study the impact of scaling on abstractive summarization and what role model asymmetry has in it. This asymmetry can manifest in various ways, such as the number of layers and hidden units in the encoder and decoder and the type of attention mechanisms used. In this paper, we explore the role of asymmetry in the number of layers in encoder-decoder language modeling for summarization and its impact on the performance of these models. As shown in Figure 1, the symmetry of pruning drives the impact on accuracy and inference speedups for sequence-to-sequence models. The following research questions drive our work:
• What scaling laws can be observed in abstractive summarization?
• What impact does encoder-decoder asymmetry have on abstractive summarization accuracy?
• What impact does encoder-decoder asymmetry have on abstractive summarization inference efficiency?
• What is asymmetries impact on accuracy and inference efficiency does scale have in encoder-decoder models for abstractive summarization?
It is in answering these questions that we deliver the following contributions:
• We present the first robust study on scaling laws applied to the compression of sequenceto-sequence modeling.
• We demonstrate that the asymmetric inference cost of sequence-to-sequence models leads to asymmetric pruning for optimal inference efficient compression.
• We empirically demonstrate on a wide variety of benchmarks how Asymmetric Compression can lead to a 2.7x inference speedup with no loss in accuracy on the XSUM dataset.

heading: 2 Related Work
text: Transformer Based Language Models such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) provide contextual language representations built on the Transformer architecture (Vaswani et al., 2017) which can be specialized and adapted for specific tasks and domains (Lee et al., 2020). Using these models, it becomes relatively easy to excel at a broad range of natural language processing tasks such as question answering, text classification, and sentiment analysis. Scaling Laws has become an increasingly impor-
tant area of study as models’ size and training data grows. Performance of the transformer-based language model improves with the relation to model size (Radford, 2018) and that larger models outperform smaller models (Brown et al., 2020) on most NLP tasks. Increasing the training corpus size can lead to large improvements in performance, and model sizes can have a optimal training data size (Hoffmann et al., 2022). Li et al. (2020) (Li et al., 2020) explore the relationship between model size and training efficiency finding larger models train faster and are more robust to pruning and quantization (Na et al., 2022). Asymmetrical in sequence-to-sequence models broadly refers to non-uniformity between encoder and decoder model shape or attributes. Training and inference procedures should match as closely as possible (Ranzato et al., 2015) (Mihaylova and Martins, 2019) as improvements in training loss during optimization result in improvements in model performance during Inference. While this may lead to the best model performance, it ignores the variable inference cost of sequence-tosequence models. During Inference, latency is dominated by the asymmetric execution of the language model. The auto-encoding encoder executes once over the entire input sequence, while the auto-regressive decoder executes iteratively until an end-of-sequence token is produced. Kasai et al. demonstrated how the sequence-tosequence language model performance for ma-
2
chine translation is dominated by the encoder depth (Kasai et al., 2020). Tay et al. 2021 extend this work by finding a DeepNarrow which shows that for broad language modeling, it is possible to have 50% fewer parameters and a 40% faster inference with no loss in accuracy (Tay et al., 2021). Efficient Inference for language modeling is a growing area of study that broadly focuses on reducing the inference cost without losses in accuracy. Unstructured Pruning has been broadly studied (Han et al., 2015) (Sanh et al., 2020) (Kurtić et al., 2022) (Zafrir et al., 2021) (Campos et al., 2022) but realizing speedups can be difficult. Structured Pruning removes fundamental structural components in a language model such as individual attention heads (Voita et al., 2019) or entire model layers such as transformer encoders (Sanh et al., 2019). Rosenfeld et al. 2020 demonstrate that unstructured pruning impacts follow scaling laws (Rosenfeld et al., 2020) where larger models can be pruned with greater ease. Compressing Sequence-to-sequence is a growing area of study where approaches from regular, efficient Inference has shown some transfer ability. Shleifer et al. show that it is possible to gain 1.93x speedup on a BART summarization model by applying structural pruning (Shleifer and Rush, 2020) but find compression approaches differ in their success depending on the dataset. Leveraging semi-structured pruning, Lagunas et al. can gain a 1.19 speedup (Lagunas et al., 2021) for minor losses in accuracy. While they find that the encoder is easier to prune than the decoder, they do not use this evidence of asymmetry to speed up performance further. Li et al. investigate how to enable quantization, finding that without specialized distillation during quantization, performance collapses (Li et al., 2022). Leveraging that generation occurs iteratively, and some tokens are easier to generate than other CALM (Schuster et al., 2022) apply early exiting to improve inference speed by 1.4x. While existing work has found interest in asymmetry, it has not been studied directly, nor has relationships in model scale been explored. While there are other approaches such as knowledge distillation (Hinton et al., 2015) (Sanh et al., 2019) (Jiao et al., 2020), quantization (Zafrir et al., 2019), early exiting (Xin et al., 2020) and token pruning (Kim et al., 2021) these are not the focus on our work as understanding the impact of many variables together limits the depth of our exploration. We leave further study of the interplay between summarization and quantization, unstructured pruning, structured pruning, and knowledge distillation for future work.

heading: 3 Scale and Abstractive Summarization

heading: 3.1 Background
text: Sequence-to-sequence language models such as BART (Lewis et al., 2021), T5 (Raffel et al., 2020), and PEGASUS (Zhang et al., 2020) combine transformer encoders and decoders to produce models which can adapt to novel tasks and reach top performance on tasks ranging from information retrieval (Nogueira et al., 2020) to summarization (Raffel et al., 2020). We focus on the instruction-tuned FLAN-T5 models (Wei et al., 2021) as their performance is competitive and they feature wide variations in model size ranging from 60 million to 11 billion parameters and given the cost of training the larger variants, focus on the small, base, and large variants. Details on model size and architecture can be found in table 1. Abstractive summarization is a method of sequence compression where a source document D is transformed into a target document dsum, which is shorter but faithful to the input. Datasets of use are a combination of public and academic benchmarks and a proprietary web search dataset. The CNN/DailyMail (CNNDM) (See et al., 2017) and XSUM (Narayan et al., 2018) datasets are based on the summarization of English new language models. The Query Independent
3
Web Summary (QIWS) is a proprietary corpus of abstractive summaries of web pages that are used to create informative contextual snippets for search engine users. It is important to note the differences in compression factor in each dataset as each impact how decoder-driven inference latency is. Further information on the makeup of each dataset can be found in table 11.
Metrics For each dataset, we evaluate model performance by measuring the ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L), RougeSumL (RSL) 6 (Lin, 2004), and Generation Length (GenL) on the test portion of the dataset. To aid the reproducibility and extension of our work, we experiment using HuggingFace’s Transformers 7, release our training and pruning scripts 8 and model variants for datasets that are publicly available datasets 9.

heading: 3.2 Scaling Laws for Abstract Summarization
text: To study the role of scale in abstractive summarization, we train small, base, and large models of the three datasets mentioned above. We do not study
6Rouge-L is sentence level vs. RougeSum-L is summary level
7https://github.com/huggingface/transformers 8https://github.com/spacemanidol/Efficient-Web-Scale-
Absractive-Summarization 9https://huggingface.co/spacemanidol
the XL (3B) and XXL (11B) as they are expensive and slow to train. For all of our experiments, we train on various hardware but fix the batch size to 64 using gradient accumulation and leverage the hyperparameters in 12. While further hyperparameter optimization and instruction tuning would likely lead to further gains in accuracy, our work is not focused on absolute Gains but on the relative relation of scale.
As shown in 2, 13, 14, and 15, there is a substantial role between scale and performance, but there is a substantial variation across datasets. Datasets with short candidate summaries, such as XSUM, see nearly three times the impact compared to the long summaries of QIWS and CNNDM. During qualitative evaluations, the role of scale can easily be observed as smaller models generate more short keyword summaries while introducing scale makes responses more natural.

heading: 3.3 Inference Benchmark
text: To evaluate the impact of asymmetry on inference, we run experiments on the throughput of each model. Using an A10 GPU and the models from our QIWS datasets, we evaluate performance with a max sequence length of 1024, a max summary of 256, and batch sizes 1, 8, and 16 using native inference in PyTorch. We report the mean and standard deviation of timings on seven runs. In comparing the impact of scale on R-2 vs. the effects on latency across batch sizes in 2, 4, 3 it becomes clear that larger models are more expensive to execute significantly as batch sizes increase. This is because of potential differences in output length within a batch as the batch completes when all sequences have produced an EOS token. To alleviate this issue bottleneck, improved streaming methods for improved batching have been proposed (Yang et al., 2020) but can be challenging to manage.

heading: 4 To Asymmetry and Beyond
text: While prior work has studied how to improve inference and tangentially explored the asymmetry between the encoder and decoder, we study that explicitly and across model scales. We focus our studies on structural pruning as inference gains are easy to realize, and this approach is highly
4

heading: 5 6 27.92 96.17% 33.57 98.18% 36.39 97.38%

heading: 5 5 26.82 92.38% 32.88 96.18% 36.32 97.20%
text: compatible with other methods like quantization and unstructured pruning. We do not study how asymmetry is impacted by unstructured pruning or quantization as these methods are difficult to combine optimized libraries like FasterTransformers10. Following Shleifer et al., we adopt the "Shink and then fine" (STF) tune approach for compression. First, a model is trained until convergence on a fine-tuning summarization task. Then, entire layers are removed from the encoder, decoder, or both, and the model is further fine-tuned until it has re-
10https://github.com/NVIDIA/FasterTransformer
converged. We do not study the use of knowledge distillation to avoid the additional training overhead without guaranteed improvements following Shleifer et al.’s results. Each model we study has a uniform number of encoder and decoder layers, so we prune only the encoders, decoders, and a symmetric combination of the two combinations. We used our three scales of uncompressed models (small, base, large), and we pruned the model in multiples of 1 on the encoder, the decoder, and both. After pruning, models are fine-tuned again and evaluated. This means that for each dataset, we have 16 variants for each model size leading to 48 models per dataset and 144 models overall. Given the wide number of models and the cost of multiple seeds or model-specific optimization, we train each model once and do not optimize the parameters for each model. While this leads to a worse-than-ideal performance, our goal is not to hyper-optimize models but explore where there is high sensitivity. To save space, we use the shorthand lenc and ldec to refer to the number portion of transformer encoder and decoder layers (out of 6), and R refers to the percentage performance recall vs. uncompressed baseline. Detailed results have been moved to the A.3 to save space.
5

heading: 5 6 17.69 100.81% 19.92 100.76% 21.13 99.88%

heading: 5 5 17.07 97.23% 19.72 99.74% 21.23 100.34%

heading: 5 6 11.18 100.82% 15.92 101.47% 19.43 118.88%

heading: 4 6 10.61 95.68% 14.10 89.91% 18.33 112.16%

heading: 3 6 10.11 91.16% 13.84 88.21% 16.90 103.39%

heading: 2 6 8.59 77.52% 12.10 77.12% 14.97 91.61%

heading: 1 6 7.70 69.43% 10.27 65.47% 12.52 76.63%

heading: 5 5 10.73 96.76% 15.72 100.22% 19.18 117.38%

heading: 4 4 10.19 91.96% 14.30 91.15% 17.56 107.43%

heading: 3 3 9.50 85.69% 12.44 79.32% 15.89 97.21%

heading: 2 2 7.31 65.91% 10.67 68.05% 12.15 74.34%

heading: 1 1 4.00 36.09% 7.74 49.35% 8.96 54.86%

heading: 4.1 Scale and Pruning
text: Looking at abridged results in 5, 6, and 7, there is a clear scaling law as smaller models see much larger drops in performance when compressed to the same degree. For example, on the QIWS dataset, compression to 16 of the layers on the encoder and decoder cause an 80% drop in R-2 on a small model but only 40% on the larger model. This scale comparison is 65% to 26% on CNNDM and 64% to 45% on XSUM datasets.
Similar scaling results hold with encoder or decoder pruning, where compressing large models lead to a 5x lower loss in performance than small models. As the model’s size grows, the impact of decoder vs. encoder-only pruning becomes more muted. On the CNNDM dataset, the gap between the decoder only and encoder only pruned to 16 is 10% with the FLAN-T5 small but only 4% with the large variant. When comparing asymmetric and symmetric, the gap is even further pronounced where the small gap is 30% while the large is 20%.
Role of scale and compression on CNNDM
6 96
As shown in Figure 3, the impact of compression becomes more muted as the model size grows. In other words, larger models are more compressible and amenable to asymmetry in this compression. The impact of asymmetry is easiest to understand as it is not surprising that complete pruning of a model leads to higher losses than partial pruning across datasets and model sizes. While this finding is not immediately surprising, evaluating the inference costs becomes important.

heading: 4.2 Inference Benchmarks
text: We evaluate the impact of asymmetry in a similar method to our scale experiments. Using an A10 GPU, we evaluate performance for summarization on a portion of each model’s respective evaluation datasets with a max sequence length of 1024, a max summary length of 256, and batch sizes 1, 8, and 16. We choose these batch sizes to represent streaming workloads (batch size 1), real-time results for the top results from a search query (batch size 8i), and max throughput given the A10’s memory budget (batch size 16)
Looking at the focused set of results for large models across datasets in table 8 on the impact of R-2 vs. inference speedup, we can see a clear relationship between asymmetry and inference efficiency. While detailed inference results can be found in the appendix A.4 on this focused set of results, we can see that pruning only the encoder leads to no more than 30% improvement in inference efficiency at a sizable loss in accuracy. Pruning the
model symmetrically leads to realizable inference improvements of up to 5x at the expense of summarization accuracy. Alternatively, when only the decoder is pruned, it is possible to see most of the inference speedups seen during constant pruning with a substantially lower impact on accuracy. On the CNN/DM dataset, constant pruning leads to 8% better inference but losses nearly four times the performance of nonuniform compression.

heading: 5 Discussion

heading: 5.1 Scale, Inference, and Pruning
text: As shown in table 9, the gains found by pruning are extremely consistent independently with scaling.
7
Pruning only the encoder leads to a 4-6% improvement in latency, and pruning just the decoder leads to 400%, as does uniform compression. This is expected as structural pruning removes a constant portion of the network, which leads to consistent latency gains irrespective of model scale.

heading: 5.2 Scale, Pruning and Generated length
text: Despite expecting a significant trend in the role of scale and pruning in a generation, we do not see any noticeable trends. As shown in figures 6 and 4, there is no discernible trend of the Role of scale and pruning in generation length. There is a minor jump in generation length from FLAN-T5 small to FLAN-T5 base across all datasets but no such jump from FLAN-T5 base to FLAN-T5 large. We believe this is because the smaller models are less fluent and need more tokens to ensure accurate coverage. As models scale, this is no longer needed, and the models converge to a uniform summary length.

heading: 5.3 Asymmetry with large batches
text: Despite the allures of asymmetrical pruning, it is not without fault. As shown in table 10 and Figure 5, the improvements in inference efficiency are heavily influenced by the batch size. When the batch size is minimal, the difference in the type of non-uniformity has a significant impact
8
on inference efficiency. As batches scale, the speedup from encoder only or decoder only becomes much closer and becomes minor when compared to uniform methods. This indicates why further work on improving generative inference methods is highly relevant, as this problem impacts other efficiency-driven processes like CALM (Schuster et al., 2022).

heading: 6 Conclusion and Future Work
text: In this work, we explore the role of symmetry in the pruning of sequence-to-sequence models for abstractive summarization, finding that pruning asymmetrically can lead to inference speedups with low losses in accuracy. Our work also explores the relationship between model scale and the sensitivity to pruning, finding that larger models see lower losses when pruned. This compresses FLAN-T5 models to deliver 3x inference gains with a 1 Rouge-2 point loss. In future work, we seek to study how pseudo labeling, early exiting, and quantization can be combined to improve further the inference efficiency of sequence-to-sequence models.

heading: A Appendix
text: A.1 Training Details In all of our experiments, we leverage the parameters shown in 12 on the datasets shown in 11
A.2 Scale and Abstractive summarization The role of model scale on performance on the QIWS, CNN/DM, and XSUM datasets can be found in tables 14,13, and 15
A.3 Asymmetry in Summarization The role of the model scale, structural pruning, and asymmetry on performance on the QIWS, CNN/DM, and XSUM datasets can be found in tables 22,23,24,16,17,18,19,20, and 21.
A.4 Inference Benchmarks Detailed variations in latency measurements across batch size, scale, structural pruning, and asymmetry on performance on the QIWS, CNN/DM, and XSUM datasets can be found in tables 25,26, 27, 28,29, 30, 33, 31, and 32.
A.5 Responsible NLP Research - Reproducibility Checklist
A.5.1 Scientific Artifacts Datasets. We perform our experimentation on well-established benchmarks using many broad domains and a proprietary web summarization dataset. We do not perform any modification or augmentation on public benchmarks in any dataset.
Models. The model used as a starting point for all of our experiments is the family of flan-t5 models, publicly available via HuggingFace Hub 13. All other models presented in this paper are openlyavailable in the hugging face hub.
A.5.2 Computational Experiments Our experimentation on finetuning our compressed models uses a single 40GB A100. Finetuning time varies across datasets ranging from 1 hour for T5small to 24 hours for T5-Large.
A.5.3 Computational Packages All of our experimentation is done using public libraries and datasets to ensure extensibility and reproducibility. Our investigation is done using HuggingFace’s Transformers 14 and Datasets 15.
13https://huggingface.co/bert-base-uncased 14https://github.com/huggingface/transformers 15https://github.com/huggingface/datasets
11
Table 11: Statistics for the abstractive summarization datasets which we study. Source and Summary refer to the number of words in each, and the compression factor is the ratio between the two on the train portion of the dataset.
Dataset Train Validation Test Source Summary Compression CNNDM 11 287,113 13,368 11,490 691.87 51.57 14.80 XSUM 12 204,045 11,332 11,334 373.86 21.09 18.70 QIWS 10000 1000 1000 1410.12 73.78 19.11
HyperParameter Value
Training Length 3,10 Epochs
Initial learning rate 1e-4 Learning rate schedule constant
Batch size 64
Weight Decay 0.01, 0.05, 0.1
Table 12: Training Hyperparameters for summarization experiments
12

heading: Future Work
text: In future work, we seek to study how pseudo labeling, early exiting, and quantization can be combined to improve further the inference efficiency of sequence-to-sequence models