Compressing Transformer-Based Sequence to Sequence Models With Pre-trained Autoencoders for Text Summarization
Authors: Ala Alam Falaki
Robin Gras
Venue: ICLR 2022 Submitted (2022)
Keywords: Transformer
Automatic Text Summarization
sequence-to-sequence
Compression

Abstract: We proposed a technique to reduce the decoder’s number of parameters in a sequence to sequence (seq2seq) architecture for automatic text summarization. This approach uses a pre-trained AutoEncoder (AE) trained on top of a pre-trained encoder to reduce the encoder’s output dimension and allow to significantly reduce the size of the decoder. The ROUGE score is used to measure the effectiveness of this method by comparing four different latent space dimensionality reductions: 96%, 66%, 50%, 44%. A few well-known frozen pre-trained encoders (BART, BERT, and DistilBERT) have been tested, paired with the respective frozen pre-trained AEs to test the reduced dimension latent space’s ability to train a 3-layer transformer decoder. We also repeated the same experiments on a small transformer model that has been trained for text summarization. This study shows an increase of the R-1 score by 5% while reducing the model size by 44% using the DistilBERT encoder, and competitive scores for all the other models associated to important size reduction.

Under review as a conference paper at ICLR 2022
COMPRESSING TRANSFORMER-BASER SEQUENCE TO SEQUENCE MODELS WITH PRE-TRAINED AUTOEN- CODERS FOR TEXT SUMMARIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
We proposed technique to reduce the decoder's number of parameters a sequence-to-sequene (seq2seq) architecture for automatic text summarization. This approach uses a pre-trained autoencoder (AE) trained on top a pre-trained encoder's 's output reduce its embedding dimension and allow significantly re- duce the summarizer model's decoder size. The ROUGE score is used to measure the effectiveness of this method by comparing four different latent space dimen- sionality reductions: 96%, 66%, 50%, 44%. A few well-known frozen pre-trained encoders (BART, BERT, and DistilBERT) have been tested, paired with the re- spective frozen pre-trained ABs to test the reduced dimension latent space's ' ability train a summarizer model. We also repeated the same experiments on a small transformer model that has been trained for text summarization. This study shows an increase of the R-1 score by 5% while reducing the model size by 44% using the DistiBBERT encoder, and competitive scores for all the other models associ- ated to important size reduction. It is also shown that our approach can used in combination with other network size reduction techniques (c.g. Distillation) further reduce any encoder-decoder model parameters count.
INTRODUCTION
It is safe to say that the combination of Transformer (Vaswani et al..
2017) architecture, and transfer
learning concept dramatically modified the landscape of Natural I Language Processing (NLP). Intro- duction of large-scale pre-trained language models like BERT (Devlin et al. 2018). GPT-2 (Radford cct al. 2019). MegatronLM (Shocybi et al. 2019). BART (Lewis et al. 2019). and GPT-3 (Brown al. 2020) keeps on improving state-of- results by fine-tuning them for downstream tasks such as Sentiment Analysis, Question Answering, and Summarization. However, the upward trend of the network size in mentioned models raises serious environmental (Strubell et al. 2019 and usability issues.
The number of parameters in Transformers-basse based models is constantly rising. As result, the hard- ware requirement for fine-tuning or inference drastically increased in the past couple of years. challenging for both researchers and developers to use these models and build on top of them without sufficient resources. To put it in perspective, the earliest and smallest BERT had 110M parameters (Devlin al. 2018), and the latest and largest Switch Transformer (Fedus et al. 202T) model was introduced with 1.5 trillion parameters which makes it accessible only using high-end servers. This issue is even more consequential for tasks such as automatic text summarization and machine translation that incorporate sequence-to-sequee of encoder (encoding the input sequences) paired with a decoder (generates tokens conditioned on the encoder's representation). It means even more parameters will be added to the model for these tasks which is the min focus this paper.
architecture. This architecture consists
It is important to discover techniques reduce the overall network parameters while maintaining the quality of the generated text. this paper, autoencoders' (AE) (Liu et al.) 2019) property of dimensionality reduction is evaluated in setting with wereence-to--eueeeee architecture and pre- trained encoders. The autoencoder will act as an intermediate model to compress the encoder's 's final representation and decoder will this compressed latent representation to generate summaries

---

Under review as a conference paper at ICLR 2022
with minimal information loss. The idea is to find the ideal trade-off between the compression ratio and model's 's text generation capability.
2 BACKGROUND
Multiple different approaches have already been explored to tackle the problem of neural network models growing size. Quantization (Gupta et al., 2013) is one of the first experiments which apply to any deep learning model by using a pref-pprei (16-bit) floating point to greatly reduce the network size and memory usage. Micikevicius et proposed a mixed precision algorithm in (Mi-) cikevicius et al., 2017) to further close the gap in evaluation results. Recent works experimented the effect of knowleds inowledge distillation (Bucila et al. 2006) method to transfer information from a larger network to smaller one without significant loss in accuracy. There are multiple papers that present different combinations of fine-tuning and distillation on top of BERT. (Chatterjee) [2019] Ture et al.) 2019) However, DistilBERT (Sanh et al., 2019) obtained the best results with training the smaller student model on BERT and then fine-tuning it for downstream tasks that resulted a more general- ized pre-trained model. Their approach led to similar implementations on other classic transformer models such as DistilGPT2 with 33% less parameters (two times faster) which resulted in 21.1 per- plexity score comparing the GPT-2's 16.3, and 35% smaller RoBERTa (Liu al. 2019) model while maintaining 95% of the accuracy on GLUE, named DistilRoBERTa
The pruning (LeCun al.,) 1990) method's influence on transfer learning have recently gained atten- tion from researchers. It refers to determining the parts in the network that have the weaker effect on the model accuracy and removing them without compromising the model's ''s accuracy on downstream tasks. The main ideas are to either focus on finding the less important weights (Gordon et al.. 2020). or components such as number self-attention heads (Michel et al. 2019) and layers (Sajjad et al, 2020). This technique was also used in the Lottery Ticket hypothesis (Frankle & Carbin 2018 Prasanna et al. 2020) to uncover subnetworks performing on par with the full model.
The latest research area focuses on rethinking the self-attention mechanism to eliminate its quadratic memory usage connection with respect to the input sequence length. The goal is to find the best trade-off between performance and memory usage. Big Bird (Zaheer et al. 2020) and Longformer
(Beltagy et al., 2020) papers experiment on different attention patterns to reduce connections and result in fewer computations. Wang al. presented the Linformer (Wang et al.)
2020) network that
projects the self-attention vectors to lower dimensions. Reformer (Kitaev et al.
2020) paper studies
the idea of grouping key and query vectors based on the locality sensitive hashing (LSH) reduce the computations needed to find similar vectors.
is worth noting several studies that combined two or more methods build even smaller models without significantly compromising the accuracy. Several such experiments are present in the lit- erature, namely, a combination of distillation with pruning (Hou et al. 2020). or quantization (Sun) et al. 2020). Also, Tabmbe et al. made the EdgeBERT TTmbe et al. 2020) model by leveraging both pruning and quantization along with other methods. These techniques are not exclusive, and is possible to study them independently without testing all possible combinations. This is why can focus our study only on one reduction method and only few important models.
3 PROPOSED METHOD
Our proposed architecture is a sequence-to-sequenee Transformer (T) model that includes a pre- trained autoencoder (AE) connecting the network's encoder (Tene) to its decoder (Tdec). (Fig, [ The mentioned approach will result in a smaller Taee and reduce the overall number of trainable parameters. The modules are described in the sub-sections.
3.1 TRANSFORMER ENCODER (Tene)
The encoder (T) (eeee) part of the proposed architecture a pre-trained transformer-basee model. We have selected set of models that include some of the best models for text summarization; BERT,
'The results related to BoBRiT model is www.lalkkalttcppss//gtttbbo.cm trans former rartreeeeeeeee wrreeeeeetattte eexamp httpr///eaaaaaeee/eeeeeroooor/rr
2

---

Under review as a conference paper at ICLR 2022
Liner Layer + Soltmax
Block R3
Pre-Trained
AE
Block #2
Frozen Pre- Traiiinnee Encoder
Decoder Block #1
000
Figure 1: The proposed architecture. The red-colored components in the diagram indicates being both pre-trained, and frozen during training the summarizer model. The green-colored units are learned from scratch for summarization task.
DistilBERT, BART's encoder (all base versions), and a custom transformer model. The custom transformer model is used evaluate the effectiveness of our approach on small pre- pre-trained model with only 6 encoder layers and subsequently its scores are used as baseline and not supposed to be competitive with the ones of the other approaches. These models are frozen during the training processes to reduce the number of influential factors.
3.2
AUTOENCODER (AE)
The AE (Fig. D purpose is to reduce the Tene's output size to a smaller latent space using the following equation:
XAAE AE AEdec(AEEnc
2 AFene (XAE)
Where X SXDD is the input, AE indicates the encoder responsible for compressing the input to latent space 7SxC and a decoder A F generating the output X'SxD trying to reconstruct the AE
input SDD during the training process. Variables and D denote the sequence length and in- put's embedding dimension respectively. The values are dependent the chosen pre-trained Tene model's 's configuration and are set to 512 and 768 in this paper. However, C' that represents the compressed latent space size, will vary to find out the optimal latent space size. 6-layer linear AE (3 for encoder and 3 for decoder) was selected after comparing its reconstruction ability to the same architecture with Long Short-Term Memory (LSTT) (Hochreiter Schmidhuber 1997). or Convolutional Neural Network (CNN) (LeCun al. 1998) building blocks.
The final AE architecture with linear layers was independently trained for each selected pre-trained encoder Tenc- It attempts to reconstruct the output of Tene using smaller representation Z. The frozen AEene is then used in our summarizer architectures to pass a compressed representation to the decoder Tdec- (From size D to C') Refer to appendix A.I for more information about the hidden layers' sizes.
3.3 TRANSFORMER DECODER (Taec)
The decoder component of the architecture (Tenc) an original transformer decoder with 3-layers and linear head on top in all the experiments. It is the only piece of the network that is not frozen after the has been trained. embedding dimension ties AF's latent space size (C) that can drastically alter the architecture overall number of trainable parameters.
3.4 DATASETS
combination of the CNN//ailymail (300K samples) (Hermann et al, 2015) and the Newsroom (1.3M samples) (Grusky et al. 2018) datasets used for training all the summarizer models; and the

---

than Under review as a conference paper at ICLR 2022
L
4
Input
Encoder AE
Compressd
Decoder
Output
Reprasentation
AE
Figure 2: The linear autoencoder architecture with encoder (L1, L2, L2, L3), and decoder (L4, L5, L6) layers. Tensors X, XX, and Z are representing the input, output, and the compressed latent repre- sentation respectively. The autoencoder maintain the same sequence length (S) during compression and only reduce the embedding size (D).
pre-defined test set of CNN//ailymail dataset utilized to evaluate them. However, we randomly selected only 60% of these combinations to train each autoencoder models to hold unseen data to evaluate the generalization ability of the summarizer model.
3.5 EXPERIMENTS
We performed several experiments to evaluate the effectiveness of the proposed method. First (AE), we combined different pre-trained encoders with several autoencoder compressed latent space sizes paired with 3-layers decoder component in each instance. Second (AE-S), used the same autoencoders (without the pre-training step) and trained them jointly with the decoder from scratch measure the effect of the pre-training step on the autoencoder. Third (LL), using a small 1-layer learnable linear model to lower the encoder's output dimensionality from D to C. Lastly (PCA), we used the classical dimensionality reduction algorithm, incremental PCA (Ross et al., 2008) trained to project the outputs of the encoder to the 458 first principal components to preserve 90% variances and used them as the decoder input.
All these models are trained with fixed hyperparameters to make them comparable. They are trained using the lcycle (Smith & Topin 2019) training policy that increases the learning rate from 2e-5 to maximum of 5e-4 while decreasing the momentum beta from 0.95 0.85 for faster convergence. We also used the Adam optimizer and the label smoothing CrossEntropy (Pereyra et al., 2017) loss function. The decoder's feedforward layers and the eight attention heads' inner dimensions 1024 and 32, respectively, and their input sizes are dependent on the dimensionality reduction rate.
4 RESULTS
The summaries are generated using three different de- coding strategies. The fastest and easiest method to develop is picking the most probable output at each timestep, known as the Greedy algorithm. A more ex- tensive approach is to use the Beam Search algorithm that develops K paths parallel using the top K tokens with the highest scores. Finally, we used the Weighted Random Sampling algorithm that randomly chose to- ken from the top K probable outputs proportionally to their respective probabilities. In this paper, we used K value equals 5 and for the beam search and weighted random sampling, respectively, based on the previous experiments.
Table 1: The comparison of MSE loss be- tween linear, LSTM, and CNN blocks to train an autoencoder with 64 compres- sion size.
Number Layers
Types LSTM CNN
0,0813
0.0766
0.0776
0.08K3
1.0000
0.784
0.2666
0.7599
0.2750
We experimented with different AE layer types and numbers. Linear, Long Short- Memory (LSTTM), and Convolutional Neural Network (CNN) AE architectures trained with the Mean
4

---

Under review as a conference paper at ICLR 2022
Table 2: Comparing the number of parameters in 3-layer decoder network with 768 input size to the number of parameters of the same decoder after applying the AutoEncoder to reduce the encoder's output dimension.
Decoder Input Size
AE Parameters Count
Trunsformer
BERRY
Decoder For Coant
Meduction
BART
Reduction (%)
Transformer
Network's Total Number Paramelers (by Eneeder Type7
BART
BERT
DisGBBERT
708 (Kkfalli
19.35
95937.00
512
2:316.288
249970746
60922.005
176415199
1342570004
384
L312430
35
165901
94.003.700
1,120756
2984
R06RT85
10:01
44.276066
149
116641.00 594
30
1,01.100
9EAT
2000300
93.5
112,400240
65,80,900
Table 3: The ROUGE score for using a pre- rre-trained autoencoder on top of pre-trained transformer- based encoders with different compression sizes. Tested each network using greedy, weighted ran- dom sampling, and beam search methods.
Inference Methook
Models
Greedy
Kandoma
Beam
R-1
R-2
R33
R-L
R-W
R-1
R-2
R[3
R-L
R-W
R-1
R-2
R-3
R-W
R-L
Transformer
0.346
0.143
0.079
0.312
0.121
0.344
0.156
0001
0.304
0.117
0.259
0.116
0.044
0.261
0.005
+A
0,388
0.157
0.088
0.325
0.129
0.303
0.147
0.079
0.315
0.123
0.238
0.127
0.070
0.250
0.104
(0 512)
(1068)
(1095)
(1119)
(1049))
(106%)
(1059)
(1085)
(111%)
(10958)
(1088)
(1119)
(1099)
(109%)
(10798)
(1098)
AE
0.363
0.154
0.086
0.322
0.127
0.360
0.146
0.078
0.314
0.122
0.278
0.125
0.068
0.274
0.101
(C = 384)
(1948)
[1078]
(108%)
(102563
(1085)
(1049)
(1076)
(109%)
(1075)
(1085)
(1079)
(1095)
(106%)
(1045)
(10955)
AF
0,508
0.114
0.060
0.286
0.109
0.315
0.110
0.054
0,280
0.106
0.280
0.116
0.064
0.271
0,100
(C 128)
(899))
(196)
(7656)
(1085))
(100%)
(10358)
(10816)
+AE
0.019
0.003
0.132
0.147
(C 52)
(45%)
(198))
(75))
(59%)
(42%)
(179)
(45))
(60%)
(576))
(505)
(189)
(68)
(165)
(47%)
BART
0.335
0.142
0.076
0.310
0.121
0.349
0.134
01007
0.301
0.116
0.304
0.128
0070
0.281
0.106
AAA
0.341
0.128
0.066
0.298
0.114
0.337
0.120
0.058
0.289
0.110
0.312
0.126
0.68
0,285
0.107
(C 512)
(90%)
(945))
(546)
(302%)
(97%)
(301%)
0.121
0.291
0.053
0.282
0.27%
0.088
0.274
[C 384)
(938)
(85R)
(80%)
(91%)
(93%)
(838)
(769)
(937))
(915)
(911))
(96%)
(97%)
(9655)
(95))
+E
0.257
0.063
0.023
0.230
0.04
0.260
0.058
0.019
0.252
0.081
0.245
0.071
0.030
0.234
0.080
(C 128)
(7288)
(44%)
(10%)
(77%)
(895)
(349)
(439)
(27%)
(72%)
(909)
(509)
(55%)
(42%)
(825)
(75R)
AE
0.145
0.014
0.001
0,168
0.055
0,174
0.019
0.002
0,179
0.089
0.128
0.015
0.002
0,146
0,045
(C S2)
(408))
(4895)
(49%)
(25))
(59%)
(5063
(429))
(51%)
(42%)
BERT
AE
0.339
0.
0.058
0.298
0.114
0.339
0.116
0051
0.289
0.108
0.291
0.117
0.09
0.275
0.100
(C - 512)
(979))
(925)
(376)
(778)
(978)
(971))
(938)
(875)
(978))
(969)
(102%)
(1009)
(98%)
(3019)
(1015)
+AE
0.332
0.119
0.056
0.294
0.110
0.333
0,112
0.048
0.286
0.106
0.272
0.107
0.053
0,263
0.094
(C 384)
(099)
(898)
(349)
(99)
(94%)
(95%)
(909)
(225)
(9675)
(94)
(969)
(115)
(89)
(97))
(94%)
AAE
0.044
0,025
0.050
0
0.022
0.252
0.088
0.242
0.029
0.237
(C'=
(599)
(37%)
(10%)
(82%)
(37%)
(88%)
(64%)
(87%)
AAE
8.168
0.211
0.001
0.187
0.02
0.197
0.036
0.003
0.194
0.044
0.140
0.020
0.003
0.153
0,447
(C 32'
(468))
(15%)
(49)
(615))
(525)
(563)
(209)
(58)
(6575)
(57%)
(49%)
(17%)
(5%)
(365)
(478)
DistIBERT
0.317
0.124
0.064
0.283
0.100
0.316
0.117
0.057
0.275
0.097
0.302
0.125
0.066
0,280
0.099
AE
0.335
0.123
0.059
0.295
0.112
0.332
0.116
0.052
0.290
0.108
0.287
0.116
0.059
0.275
0.100
512)
(1085)
(996)
(92%)
(108%)
(112%)
(1055)
(995)
(91%)
(1085)
(111%5
(955))
(94%)
(8916)
(98%)
(10115)
+AE
0.334
0.122
0.060
0.297
0.112
0.334
0.115
0.022
0.288
0.108
0.292
0.114
0.058
0.270
[C = 384)
(105%)
(98%)
(93%)
(1045)
(112%)
(305%)
(95%)
(91%)
(104%)
(111%)
(95%)
(95%)
(57%)
(965)
(97))
LAE
0.287
0.083
0.031
0.265
0.044
0.297
0.081
0.02%
0.25)
0.092
0.240
0.082
0,855
0.238
0.081
(C 128)
(909))
(668)
(48%)
(93%)
(94%)
(93%)
(699)
(495)
(945)
(94%)
(79%)
(6693)
(53%)
(85%)
(81%)
+AE
0,161
0.020
0.003
0.089
0,189
0.024
0.003
0.188
0
0.134
0.019
0.003
0.147
0.045
(C
(16%)
Square Error (MSE) loss for seven epochs to measure their performance. A latent space represen- tation of size 64 was selected for this comparison benchmark (Table [.. The linear autoencoder outperforms both LSTM and CNN in all experiments (Refer to Appendix [A.]] Table (6 for the full list of all compression rates) Also, the 6-layer design choice results in a better score in all experi- ments.
The model size comparisons (number of parameters) are presented in Table The first step is to calculate the decoder's size without using the dimensionality reduction method. As previously men- tioned, based on the choice of the pre-trained encoder models, the default encoder's representation dimension is 768 which results in a decoder with either 48M (for BART encoder) or 33M (other op- tions) parameters. We use this value as the reference number to measure the reduction percentage. is important to keep in mind that even though there are parameters being added to the model from the autoencoder that affects the percentage, the number of trainable parameters in all the experiments is equal to the decoder size since the rest of the components (encoder, and autoencoder) are frozen during the tests. The reason why we have decoders with different parameter counts while using the same apper-parameters is because we use each model's pre- pre-trained tokenizers which have different vocabulary size, which affects the decoder's last layer output dimension. BERT, DistilBERT, and Transformer models use the BERT's pre-trained tokenizer, and BART uses its own.
5

---

Under review as a conference paper at ICLR 2022
Table 4: The comparison between using the pre-trained AutoEncoder (AE), training the AutoEn- coder's s encoder jointly with the network from scratch (AE S), using simple linear layer model for the projection (LL), and PCA to do the dimensionality reduction.
Intervice Methods
Models
Groedy
Random
Ream
R-T
R-2
R-3
K-L
K-W
R-1
R-2
R-3
R-
R-W
R-1
R-2
R-3
R-W
R-L
BERT
0.348
0.133
0.066
0.306
0.117
0.347
0.121
0.053
0.297
0.112
0233
0.117
0.000
0.270
0.099
0.123
0,058
0.298
0.339
0.108
0.291
0.100
(C 512)
(97%)
(92%)
(87%)
(97%)
(97%)
(975)
(93%)
(87%)
(97%)
(96%)
(1028))
(100%)
(98%)
(101%)
(101%)
ARR
0.289
0.079
0.027
0.262
0.093
0.300
0.079
0.025
0.258
0.092
0.36
0,084
04133
0.250
0.086
(C
(825)
(99%)
(40%)
(85%)
(79%)
(86%)
(63%)
(43%)
(86%)
(82%)
(92%)
(71%)
(55%)
(92%)
(865))
0.277
0.083
0.33
02600
0,092
03283
0.080
0.000
0.254
0.000
0.334
0.083
0.036
0.238
0,081
(C' 512)
(7998)
(625)
(50%)
(86%)
(78%)
(815)
(64%)
(515)
(85%)
(80%)
(825)
(7085)
(00%)
(8858)
(815)
PCA
0.143
01016
0.002
0.156
0.016
0.158
0.017
0.002
0.139
0.048
0.116
0.013
0.002
0.131
0,038
(128)
(508)
(458)
(35)
(40%))
(38%)
Table 5: The comparison of a few vanilla encoder-decoder model generated summaries the ones generated by the same model with the addition of AE with a latent space size of 384. Both results generated using the greedy decoding method.
Model
Vanilla Model Generated Summary (Greedy) handguns have robbed
+AE Model Geuerated Sumamary (Greedy)
in pinshurgh ares they are helieved to have had military maining and are heing described as armed and extremely
maskedd men armed
two macn armed with handguns robbed three banks in pilt-burgh area so far this year the unknown men, who are soen on surveillance footage pointing their guns bank employees heads. have threatened to kidrapping lhuse
Transformer
dangerous' the
believed have threatened to
kidnspping those their targets shoot police. however. the way that the men handle
targets shoot police however, way the men hundle their weapons has led
two are believed to have had military training and heing described dangerous', men are seen holding their finger strecched along the barrel of his gun, just ofT of the tri gger, a safltyy method used by law es enforcement the men. who wear dark
two peunsylvvni vaniaa bank robber believed to have had milltary training. they helieved
'armed and extremely
been armed and extremely dangerous the
BART
men believed have been armed ammed with handgun, the men are belioved to have been from pinshurgh.
pair of masked men
two ponnsylvara, bank robbers have robbed the pittsburgh srea are seen on surveillance footage have threatened to kidmpping those Usiir targes and shoot police. the two -9" april
two peunsy ivanii bank robburs armed with handguns have robbod in the pittsburgh area so this year they have been seen
this year. the unknown men, who
BERT
jumping over the counter as they take their guns targets and shoce who
them police. the two have threatened kidnopping
surveillance foutage.
men, boih
deseribed
their
two robbets have been seen in a scries of recent bosstnng rubberies the mon are believed to have had militrry training
two masked men armed with handguns have robbed three banks the pirshuugh area so far this training and being deseribed by fhi as armed and extremely dangerous men believed have military training are heing deseribed by the fhi as
and are being descrihed as armed and extremely dungerous" men belioved men are believed to have been arned and extremely dangerous.
they are helieved have had military
DiSGBBRT
boen armed armed.
The proposed method results on the text summarization task are shown in Table where the classic ROUGE (R-1, R-2, R-3, R-I,, and R-W) metric is reported using the greedy, weighted random sampling, and beam search decoding strategies. Autoencoders are trained with four (32, 128, 384, 512) different latent space dimensions (C) to analyze the effect of these dimensionality reductions. The models are also evaluated using the BERTScore (Zhang et al. 2019) metric and the results follow the same direction. (Refer to Appendix[A.4
Our configurations not only reduce the number of parameters in the network by reducing the decoder size, but also shows the ability to increase the model's summarization ability with a higher score compared to the original setup for the Transformer and DistilBERT models. The experiments show that adding an AE with C' = 512 outperforms the same vanilla encoder-decoder network. Both BART and BERT experiments with the same AE size outperformed their vanilla model in several metrics only using the beam search method; however, the rest of the scores with a smaller decoder are still competitive. An even more surprising result that a smaller dedicated Transformer with the proposed method and C = 384 performed better than BART and BERT in all the summarization benchmarks.
It also worth noting that the greedy inference method constantly results in better scores, with the weighted random sampling method following closely. The fact that beam search algorithm leans to- wards shorter sequences (Wu et al., 2016) reduces the ROUGE scores since there are fewer matching N-grams in the generated and target summarics. It does not mean that the sentences structure/quality flawless using greedy/weighted random sampling, poor using beam search, the results just flect what the ROUGE score is measuring: an N-gram overlap between the generated and the target sequences.
Our ablation results show (Table the impact of autoencoder pre- pre-training step on the final score. surpasses both training the A Fooe jointly with the network from scratch (AE S) and using simple
6

---

Under review as a conference paper at ICLR 2022
learnable linear layer for projecting (LL). Lastly, the PCA dimensionality reduction technique does not produce desirable results. The results also show that using an autoencoder with a latent space size of 384 (C 384) generates ROUGE score close to the vanilla model. It reduces the decoder size by 46% from 48M to 24M for the BART model and 44% from 33M to 16M parameters for the other models. The critical point is that the combination of this configuration associated with greedy decoding algorithm shows no noticeable degrading quality in the generated summary. (Table 5)
5 CONCLUSION
In this paper, we presented a method to use pre-trained encoders models in sequence-to--eqquenee setting while training a small decoder from a compress representation of the decoder's output for automatic text summarization task. The proposed architecture is based on an autoencoder pre- pre-trained reduce the encoder's representation size. The resulting compressed latent representation used as inputs for decoder. The main idea is that decreasing the size of the AE latent representation leads to dramatic decrease in the decoder size. We have shown that by reducing the encoder's output dimension from 768 to 384, not only we can reduce the decoder size by 44%, but it will also keep 95% the R-1 score in the worst case (BERT) or increase up to 105% with DistilBERT. Moreover, with almost 80% reduction in the decoder size we can still keep 90% of the ROUGE score with our dedicated transformer and DistilBERT architectures.
Our method can be directly used together with other approaches such distillation, pruning and quantization to further reduce the network size. One of our future research projects will be to investigate which of such combination could lead to the best compromise between the size and the accuracy of the model. might also be interesting to study the effectiveness of our approach on other generative tasks like translation and question answering for further future works.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint FrXXi:2004.051500 2020.
Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Cristian Bucila, R. Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD '06, 2006.
Debajyoti Chatterjee. Making neural machine reading comprehension faster. arXiv preprint 1904.0090000 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961, 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint /803.08600333333 2018
Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning transfer learning. arXiv preprint arXiv:2002.08307, 2020.
Max Grusky, Mor Naaman, and Yoav Artzi Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv: / /283, 2018.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737-1746. PMLR, 2015.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 18:161111701, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint 1iXi:22001.044511 2020.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228-23224 1998.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? arXiv preprint arXiv: 10650, 2019.
8

---

Under review as a conference paper at ICLR 2022
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training, arXiv preprint arXiv: 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 1rXiiv1701.000 2017.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are win- ning. arXiv preprint arXiv:2005.0055 00566, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
David A Ross, Jongwoo Lim, Ruci-Sung Lin, and Ming- Ling-Hsuan Yang. Incremental learning for robust visual tracking. International journal of computer vision. 17(1)1125-141, 2008
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. arXiv preprint (rXiv:2004008844 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training bulli-billion parameter language models using model par- 2019.
allelism. arXiv preprint
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- Doati-D Operations Applications, volume 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 1906.0224, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact task-agnostic bert for lesoueee-limited devices. arXiv preprint arXiv:2004.0298: 2020.
Thierry Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En Yang. Marco Donato, Victor Sanh, Paul Whatmough, Alexander Rush, David Brooks, et al. Edgebert: Sentence-level energy optimizations for lateney-aware multi-task nlp inference. arXiv preprint arXiv:2011. 2020.
Julia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprinl arXiv: 000///00.0000 08962, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 201::2000.04768, 2020.
Yonghui WU, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore; Evaluat- 2019.
ing text generation with bert. arXiv preprint

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 AUTOENCODER ARCHITECTURE
Table the full results of the different autoencoders building blocks and sizes. The linear layer AE outperforms both LSTM and CNN in almost all the compression sizes. While the only exception the LSTM network with the smallest latent space dimension (32) that narrowly achieved better accuracy, none of the experiments had acceptable performance that representation size. Also, shows that the CNN architecture resulted in the worst score by a large margin.
Table shows the details of the 6-layer linear architecture that consists of projections in each encoder and decoder. (Illustrated in Fig. 2 The idea is to keep a gentle decrease in size for large latent spaces, and have enough learning capacity with wider networks in smaller compressed sizes.
Table 6: The MSE loss value of the selected 3 network types (LSTM, linear, CNN) with different number layers.
Type
Compression Rate
Number of Layers
MSE Loss
32
0.0905
0.1176
0.0863
0.0810
0.0849
LSTM
10
0.1043
128
0.0670
256
0.0543
384
0.0462
448
0.0427
512
0.0400
32
0.0930
0.0752
64
6
0.0766
0.0775
Linear
128
0.0637
256
0.0453
384
0.0278
448
0,0239
512
0.0200
0.2666
CNN
0.2759
0.2750
Table 7: The autoencoder models projections for different compression rates.
Third Projection/ /
First Projection (P1)
Second Projection (P2)
Compressed Latent Space Size (C)
640
576
512
608
528
448
576
480
384
640
320
256
512
256
128
512
256
64
512
256
32
10

---

Under review as a conference paper at ICLR 2022
A.2 EXTRA RESULTS FOR BERT MODEL + AE
We did a few more experiments on the BERT model to find the optimal latent space size and ROUGE score combination. As shown in Table g the intermediate C' values are also following the same trend as presented in Table 6
Table 8: The ROUGE score of more experiments on the optimal latent space size options.
Inference Methods
Models
Greedy
Random
Beam
R-1
R-I.
R-W
R.T.
R-W
R.3
R-W
R-1.
+ AE
0.337
0.123
0.059
0.297
0.112
0.337
0.115
0.051
0,289
0.108
0.283
0,113
0.057
0.270
(C (18)
AE
(C 256)
0.323
0.109
0.048
0.285
0.105
0.323
0.101
0.041
0.277
0.101
0,272
0.103
0.048
0.262
0.093
+E
0.250
0.048
0.011
0.226
0.077
0.234
0.047
0.011
0.227
0.077
0.195
0.047
0.013
0.199
0.064
(C 64)
A.3 MORE GENERATED SUMMARIES EXAMPLES
The samples of generated summaries using the weighted random sampling and beam search infer- ence methods are presented in Tables 9 and I0] Both tables are using an autoencoder with latent space size of 384 which showed promising results in our experiments. The generated summaries maximum length is set to 60 tokens in all experiments. All the generated summaries sound good and capture the main information of the original texts.
Table 9: Comparing the generated summaries of the vanilla model and the model with a latent space size of 384 using the weighted random sampling decoding method.
Model
Vanilla Model Generated Summary
(Wrighted Randons Sampling)
(Weighted Random Sampling)
AAB Generated Summarr
masied men armed with handgans have robhed three banks in pittsburgh so this
the mon, who are soon on surveillace looing pointng guis at bank employecss heads, have threatened kidnaping thoss their turget policc. however, way that weapons has led the thi snspect that the thieves are actually fomeer police oflicers themselves... they are also
they are belioved to be
Transfrmer
arned extromely dangerous. they thought luve heen anned with handguns and thought piltsburgh. the suspects are deseribed as white 8 to 5
men handle
from
have been robbed by the thi since april 10, according to surveillance footage. they have been seen bolding his finger strcchhd along the barne of his gun. they have boen soun
pennsylvane bank robbery suspects have been seen in string of recent heists, the sispects are believed to have boen from pittsburgh.
BART
over the counter as they begin their beists, the two robbers have
the suspects are believed to he from pittsburgh because of their atritudes.
during the robberies
unknown men, who are seen have threatened to kidnap those at their tugees and shoot police the two men. both 5 5 -9 and april 10, have also been taken the bank in pittsburgh. pennsylvania. the fbi two suspects may the men, who wear dark sw sweetpanss are believed to he armed and extremely violent. the two men are thought have been ameed and armed. they believed be from pitsshurgh. pennsylvania. who have heen robhed three banks the men are thought to have been wearing the gun and a gun.
surveillance Tootage.
two penneyllva Ivanin bank robbers amned they do series recent robberies they have been described as armed and extremely dangerous" they have heen seen on surveillance footage showing the have been described ummdd and extremely dangerous dangerous.
BERT
two masked men are thought to have rubbed three banks in pitsburgh
they are believed to have been armed and extremely dangetous. they have deseribed as armed extromely dangerous.
DisEBKRT
Table 10: Comparing the generated summaries of the vanilla model and the model with latent space size 384 using the beam search decoding method.
Model
Vanilla Model Generated Summary (Beam Search)
AE Generated Summary (Beam Search) two men armed with handguns robbed three bunks in pittsbuggh area far this year. recently april 10
masked men ammed with handguns have robbed three banks in pittsburph
Transformer
far this year, most recently april the men, who mr seen on surveillance footae pointing their guns at bank employets' heads, have threatened to kadnapping those at their targets and shoot police. the two men
two pernsaylvnnia bank thieves are helieved to have had milltary training and are being descrited by the fbi as 'ammed and extremely dangerous'
BART
actaally former police officers themselves.
BERT
two pennsylvania
robhers are helieved to have had military training and are being described by
two pennsyyvvani bank robbers armed handguns have robbed in the pittsburgh area so far this year, most recently on april 10
pair of masked men ameed with handgans have roithed far this year, most
three banks the pinshhurgh unknown men. who are on surveillanee footage pointing their gans at hank emplayees heads, have threatened to kidnapping and shoot police.
two marked armed with handguns robbed three banks piltssuugg area so for this year. most recently on apeil 10
DisiBBERT
recently on april
11

---

Under review as a conference paper at ICLR 2022
A.4 BERTSCORE RESULTS
The following table (Table demonstrates the evaluation results of the proposed method using the
BERTScore metric. The mentioned metric also confirming the paper's discussions contextual level.
Table 11: Comparing the vanilla and proposed models generated summaries quality using the
BERTScore metric.
Model Inference Method
Vanilla
(C 512)
+AE (C 384)
+AE (C' 128)
+AE 32)
Greedy
0.858
0.861
0.860
0.846
0.801
Transformer
Random
0.857
0.860
0.859
0.847
0.809
Beam
0.852
0.858
0.857
0.853
0.805
Greedy
0.867
0.865
0.863
0.845
0.814
BART
Random
0.869
0.864
0.862
0.819
Beam
0.866
0.866
0.865
0.851
0.821
Greedy
0.858
0.857
0.854
0.854
0.809
BERT
Random
0.857
0.856
0.854
0.843
0,815
Beam
0.841
0.854
0.854
0.846
0.814
Greedy
0.798
0.855
0.842
0.809
DistilBERT
Random
0.836
0.855
0.854
0.844
0.815
Beam
0.802
0.856
0.855
0.846
0.814
12


Reviews:
[{"review_id": "gD93-pF793", "reviewer": "Reviewer_wmem", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper was written clearly enough to understand the basic ideas.\n2. The paper conducted extensive experiments to find the ideal trade-off between the compression ratio and model’s text generation capability.\n\nWeaknesses:\n\n1. The results of the transformer baseline is far from previous literatures, such as [Li et al., 2018; Lewis et al., 2019; Wei et al., 2020] that achieves at least 33.42 R-L score, while this paper report a 31.2 R-L score with the transformer baseline.\n\n2. There is no any comparisons with existing compression methods, including pruning and distillation.\n\n[Li et al., 2018] Improving Neural Abstractive Document Summarization with Explicit Information Selection Modeling.\n[Lewis et al., 2019] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.\n[Wei et al., 2020] Multiscale Collaborative Deep Models for Neural Machine Translation.", "summary_of_the_paper": "This paper proposes to compress the transformer-based summarization models with well pre-trained autoencoders (AE). In this architecture, AE produces an intermediate states that is obtained by compressing the encoder’s final representation and pass it to the decoder.", "summary_of_the_review": " In this paper, the baseline system is far from literatures and is lack of comparisons to existing methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fn2Udw6u9pU", "reviewer": "Reviewer_1LU5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper investigates the use of a pre-trained autoencoder to reduce the output dimensionality of a pre-trained transformer (such as BERT or BART) so that a lower dimensionality decoder can be used.  \n\nThree kinds of AE are investigated: Linear, LSTM and CNN.  The way these are used is not very clear but I assume that the linear AE maps each symbol independently whereas the LSTM and CNN see the whole sequence?  This point should be clarified.  Also, what motivated the choice of a purely linear network.  A standard MLP bottleneck autoencoder seems to me to be a more obvious choice than a CNN.  Why wasnt this used?\n\n\nThe AE is trained using \"60% of these combinations of dataset\" - please explain what motivates the choice of 60% and confirm that no test material is used for pre-training the autoencoder.\n\nThere appears to be a problem with Table 4 - perhaps the labelling in the left hand column?  In any event, I cant find the comparisons between AE, AE S and LL.  It's really not obvious why AE S should be worse than AE.  Please comment on this.\n\nThe results show that you can reduce the decoder size by up to 40% without sacrificing too much in performance.  However, this does nothing to change the size of the pre-trained encoder and it is the problem of the ever-increasing size of the pre-trained encoder which \nis highlighted in the introduction to the paper.\n\nIt seems to me that what you are really doing here is adjusting the size of the decoder to best match the amount of training data for the downstream task.  In your case, text summarisation.  A different task with much more training data might give a different result.  Please comment on this.", "summary_of_the_paper": "This paper investigates the use of a pre-trained autoencoder to reduce the output dimensionality of a pre-trained transformer (such as BERT or BART) so that a lower dimensionality decoder can be used.  The results show that for a summarisation task, the decoder size can be reduced by 40% without significant loss of accuracy as measured by rouge. ", "summary_of_the_review": "This paper describes a simple approach to reducing the dimensionality of the decoder to be used for a pre-trained encoder.  The experimental results may be of interest to practitioners concerned with reducing model sizes.  However, the original contribution is rather small and the restriction to a single downstream task leaves questions over its general applicability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4P0ulAowfQE", "reviewer": "Reviewer_vaDv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. The paper is clearly written. The paper tries to discover the trade-off between the compression ratio and model performance. The paper shows experiment results by comparing different types of autoencoders as well as different compression rates. The paper also presents the generation results and another related experiment in the appendix. \n\nWeakness\n1. How is the compression rate determined? It would be better to show why 32, 128, 384,512 are chosen for the decoder size. The scale of those dimensions is not even linear. It might be more reasonable for readers to see dimensions with 32, 64, 128, 256, 512. The paper needs to include more comparison between other compression methods such as distillation, information bottleneck, pruning. The paper fails to show insight qualitative analysis for the Table. 3 and Table 4. Table 4 seems to be incomplete. In Section 3.4, why not use all training data to train the autoencoder. \n\n2. The rouge scores might not be good  It would be better to incorporate other metrics such as BLEU, BERTscore. ", "summary_of_the_paper": "The paper proposes a new autoencoder-based seq2seq model for text summarization tasks. The paper tries to find the best trade-off between compression ratio and model performance. The paper conducts extensive experiments by evaluating the loss of accuracy with ROUGE.", "summary_of_the_review": "Overall, this paper presents an interesting experiment to discover the trade-off between the compression rate and the performance. The experiment seems a little bit naive without detailed analysis.  I recommend rejection for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2RRapThRckD", "reviewer": "Reviewer_Lsjg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weaknesses\n1. The motivation for using autoencoder is not quite clear. Actually, BART is already pretrained with a denoising loss which should be better than autoencoder loss. How about a more detailed comparison between these two losses? \n2. It's not clear whether the encoder is frozen or not. Is the encoder further optimized when fine-tuning? If not, it's not clear why not optimize encoder. If optimized, then pre-training losses from BART can also be adopted.\n3. Missing some knowledge distillation baselines, such as Noisy Self-Knowledge Distillation for Text Summarization and PRE-TRAINED SUMMARIZATION DISTILLATION\n4. The paper writing can be further improved. Figures 1-3 have been shown in many papers. It would be better to emphasize more on the novel part.\n", "summary_of_the_paper": "The authors propose to add an autoencoder on top of a pre-trained encoder to reduce the encoder’s output dimension and allow to significantly reduce the size of the decoder.", "summary_of_the_review": "Overall, the proposed loss is not novel enough and needs to have a further comparison with other pertaining losses such as the losses from BART. And also missing some strong Seq2Seq baselines with knowledge distillation losses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to incorporate an autoencoder to transformer-based summarization models in order to compress the model while preserving the quality of summarization. The strengths of the paper, as identified by reviewers, are in extensive experiments presented in the paper and in a relatively clear write-up. However, the reviewers identify several weaknesses, including missing state-of-the-art summarization baselines and missing relevant compression/knowledge distillation baselines. Although the author response have addressed some of reviewers' concerns, all the reviewers agree that the draft is not yet ready for publication.
Decision: Reject