Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to add an autoencoder on top of a pre-trained encoder to reduce the encoder’s output dimension and allow to significantly reduce the size of the decoder.
main_review: Weaknesses
1. The motivation for using autoencoder is not quite clear. Actually, BART is already pretrained with a denoising loss which should be better than autoencoder loss. How about a more detailed comparison between these two losses? 
2. It's not clear whether the encoder is frozen or not. Is the encoder further optimized when fine-tuning? If not, it's not clear why not optimize encoder. If optimized, then pre-training losses from BART can also be adopted.
3. Missing some knowledge distillation baselines, such as Noisy Self-Knowledge Distillation for Text Summarization and PRE-TRAINED SUMMARIZATION DISTILLATION
4. The paper writing can be further improved. Figures 1-3 have been shown in many papers. It would be better to emphasize more on the novel part.

summary_of_the_review: Overall, the proposed loss is not novel enough and needs to have a further comparison with other pertaining losses such as the losses from BART. And also missing some strong Seq2Seq baselines with knowledge distillation losses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new autoencoder-based seq2seq model for text summarization tasks. The paper tries to find the best trade-off between compression ratio and model performance. The paper conducts extensive experiments by evaluating the loss of accuracy with ROUGE.
main_review: Strength
1. The paper is clearly written. The paper tries to discover the trade-off between the compression ratio and model performance. The paper shows experiment results by comparing different types of autoencoders as well as different compression rates. The paper also presents the generation results and another related experiment in the appendix. 

Weakness
1. How is the compression rate determined? It would be better to show why 32, 128, 384,512 are chosen for the decoder size. The scale of those dimensions is not even linear. It might be more reasonable for readers to see dimensions with 32, 64, 128, 256, 512. The paper needs to include more comparison between other compression methods such as distillation, information bottleneck, pruning. The paper fails to show insight qualitative analysis for the Table. 3 and Table 4. Table 4 seems to be incomplete. In Section 3.4, why not use all training data to train the autoencoder. 

2. The rouge scores might not be good  It would be better to incorporate other metrics such as BLEU, BERTscore. 
summary_of_the_review: Overall, this paper presents an interesting experiment to discover the trade-off between the compression rate and the performance. The experiment seems a little bit naive without detailed analysis.  I recommend rejection for this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to compress the transformer-based summarization models with well pre-trained autoencoders (AE). In this architecture, AE produces an intermediate states that is obtained by compressing the encoder’s final representation and pass it to the decoder.
main_review: Strengths

1. The paper was written clearly enough to understand the basic ideas.
2. The paper conducted extensive experiments to find the ideal trade-off between the compression ratio and model’s text generation capability.

Weaknesses:

1. The results of the transformer baseline is far from previous literatures, such as [Li et al., 2018; Lewis et al., 2019; Wei et al., 2020] that achieves at least 33.42 R-L score, while this paper report a 31.2 R-L score with the transformer baseline.

2. There is no any comparisons with existing compression methods, including pruning and distillation.

[Li et al., 2018] Improving Neural Abstractive Document Summarization with Explicit Information Selection Modeling.
[Lewis et al., 2019] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.
[Wei et al., 2020] Multiscale Collaborative Deep Models for Neural Machine Translation.
summary_of_the_review:  In this paper, the baseline system is far from literatures and is lack of comparisons to existing methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates the use of a pre-trained autoencoder to reduce the output dimensionality of a pre-trained transformer (such as BERT or BART) so that a lower dimensionality decoder can be used.  The results show that for a summarisation task, the decoder size can be reduced by 40% without significant loss of accuracy as measured by rouge. 
main_review: This paper investigates the use of a pre-trained autoencoder to reduce the output dimensionality of a pre-trained transformer (such as BERT or BART) so that a lower dimensionality decoder can be used.  

Three kinds of AE are investigated: Linear, LSTM and CNN.  The way these are used is not very clear but I assume that the linear AE maps each symbol independently whereas the LSTM and CNN see the whole sequence?  This point should be clarified.  Also, what motivated the choice of a purely linear network.  A standard MLP bottleneck autoencoder seems to me to be a more obvious choice than a CNN.  Why wasnt this used?


The AE is trained using "60% of these combinations of dataset" - please explain what motivates the choice of 60% and confirm that no test material is used for pre-training the autoencoder.

There appears to be a problem with Table 4 - perhaps the labelling in the left hand column?  In any event, I cant find the comparisons between AE, AE S and LL.  It's really not obvious why AE S should be worse than AE.  Please comment on this.

The results show that you can reduce the decoder size by up to 40% without sacrificing too much in performance.  However, this does nothing to change the size of the pre-trained encoder and it is the problem of the ever-increasing size of the pre-trained encoder which 
is highlighted in the introduction to the paper.

It seems to me that what you are really doing here is adjusting the size of the decoder to best match the amount of training data for the downstream task.  In your case, text summarisation.  A different task with much more training data might give a different result.  Please comment on this.
summary_of_the_review: This paper describes a simple approach to reducing the dimensionality of the decoder to be used for a pre-trained encoder.  The experimental results may be of interest to practitioners concerned with reducing model sizes.  However, the original contribution is rather small and the restriction to a single downstream task leaves questions over its general applicability.
review_title: null

Meta-review: The paper proposes to incorporate an autoencoder to transformer-based summarization models in order to compress the model while preserving the quality of summarization. The strengths of the paper, as identified by reviewers, are in extensive experiments presented in the paper and in a relatively clear write-up. However, the reviewers identify several weaknesses, including missing state-of-the-art summarization baselines and missing relevant compression/knowledge distillation baselines. Although the author response have addressed some of reviewers' concerns, all the reviewers agree that the draft is not yet ready for publication.
Decision: Reject