Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new model architecture for abstractive text summarization. The framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency at a coarser time scale, and the bottom token level preserves the detail. The proposed model shows some advantage in both capability in capturing global and local context  and being computationally efficient. 
The model is validated on many datasets and it beats almost all previous model. 

main_review: Strength
Even a little surprisingly, the performance is super good on all datasets. It beats all state-of-the-art models, including Longformer, LSH, BigBird, etc, by a decent margin (usually more than +1 R-1). The model is just initialized from BART. I feel it’s almost magic that fine-tuning a BART parameter based model could be that effective. A lot of prior work shares the same conceptual design principles (hierarchical, coarse-and-fine grained, local-global, etc.), but this model works very well indeed.

Weakness
I can’t really conclude the difference or principle which contributes to the performance gain. The authors developed a new way to assemble these blocks to build a model, and it empirically works well. However, are there any distilled piece we can take from this paper? What’s the killing component in this model? It’s not clear for me. Some ablation study would be helpful. 

The paper is a very straightforward technical report, but it could be more thought-provoking or intellectual. I don’t mean to criticize the paper because it’s easy to follow. I sometimes prefer papers with more insights or thoughts, although I totally understand they are usually not accompanied with empirical success. 

Minor: 
I am not a huge fan of the usage of “inference” in this paper. All these “inference” modules are trained and fine-tuned. It is a little confusing when I read the first page. 
The caption and presentation of Figure 1 could be improved. The caption contains too many terms which do not show in the Figure. Please consider unify the model name between the figure and caption. 
summary_of_the_review: It's a model architecture paper for neural abstractive text summarization. It works super well on many widely used datasets. Despite the empirical success, it lacks more principled insights and justification about what works behind it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Modeling long documents is a challenging problem. This paper works on this problem with top-down and bottom-up structures. The hierarchical structure proposed in this paper adopts local and top-down correction to make the model learn local and long-range dependency. The experiment on long document summarization benchmarks shows the effectiveness of their proposed method.
main_review: 
Strength:
- The hierarchical structure proposed in this paper is reasonable. By modeling the input document in a bottom-up and top-down manner, the model can model long documents in coarse and fine granularity levels.
- The performance on benchmark datasets looks good.

Weakness:
- In their paper, the author mentioned that the local attention window size is 1024, which is even larger than 512. I think such a large window size might not be local. However, in Table 1, most datasets are not that long and even shorter than 1000. Did the authors consider different window sizes? (except the window size in Table 3.). Also, how to slide the window across the inputs? Is there any overlap between local windows?
- The combination of top-down and bottom-up is new. But what's the performance without cross-attention? Also, it is not clear how the copy operation is implemented. 
- The author mentioned that they use BART as the initialization. Did they consider initialization with vanilla BERT or just random? I think this might better show the effectiveness of the proposed method by removing the BART contribution.
- In the experiment part, what is the parameter for those baseline methods (BigBird, Longformer, etc.) Instead of only comparing with GPT-3 on model size, they should also report the model size for other models.
- The writing of this paper needs to be improved. Especially for the experiment part, what's the define and implementation of OracleAdaPool?  What is RL denote?
summary_of_the_review: The top-down and bottom-up framework proposed in this paper achieves good performance on long document summarization tasks. However, despite the results, the novelty of this design is somehow limited since the bottom-up and top-down idea is not new. The experiment part of this paper is also not clear, some details are missing, and some issues need to resolve (please see the weakness part).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Aiming at the task of long text summarization, this paper proposes a top-down and bottom-up reasoning method to improve the traditional bottom-up converter encoder structure, which has higher memory and computational efficiency. The model captures remote dependencies on a coarser time scale at the top, and the token level at the bottom retains details. It uses local self-attention to improve computational efficiency. At the same time, it has achieved good results on a large number of long text summary data sets. However, the method proposed in this article has been extensively studied in many fields, and there is no detailed analysis of the problems of long texts, and the theoretical innovation is insufficient.
main_review: Strengths 
1、This paper conducted a variety of experiments on long document data sets of different levels, and proved that the top-down structure can effectively improve the performance of long document summaries.
2、Compared with the model of gpt-3, the model proposed can summarize a complete book, and use 0.27% parameters and less training data to achieve competitive performance.
3、The experimental result has proved that the proposed model can effectively improve the performance of long text summaries, which has been greatly improved comparing with other baseline models.
Weaknesses 
1、The method proposed in this paper is not innovative enough. Similar methods have been extensively studied in images and speech, and the model structure does not fully reflect the problem of information redundancy faced by long document summaries.
2、The detailed description of the model in this article is not clear enough, for example, there is no detailed description of the overall operation process of the local attention and the top-down model.
3、In this paper, the experimental verification for calculation efficiency and memory usage of model is not sufficient. It only compares the parameters of the model, and does not conduct quantitative analysis through experiments.
summary_of_the_review: 1、It is recommended to explain the specific calculation process of the model in detail, especially for the top-down inference process;
2、It is recommended to further verify the computational efficiency and memory usage of this method through experiments.
review_title: null

Meta-review: This paper deals with the task of long text summarization. Inspired by earlier work on top-down and bottom-up architectures, this work focuses on improving the traditional bottom-up converter encoder structure, and the fine resolution representations. 

Pros:
- Their model can model longer documents in coarse and fine granularity levels.
- The performance on benchmark datasets looks pretty good compared to strong baselines
- Computationally efficient. 

Cons: The reviewers have raised several concerns including:
- the experimental verification for calculation efficiency and memory usage of model is not sufficient.
- the novelty of this design is somehow limited since the bottom-up and top-down idea is not new. 
- several details about the figures and especially the experiments were missing.

The authors have addressed several of the suggestions, added new experiments results addressing the issues raised by the reviewers. During the rebuttal period, the authors further conducted empirical investigations showing that the top-down update for token representations, especially with good top-level representations, leads to good summarization because of enriched token-level representations by the top-down. Despite positive results, some reviewers raised concerns that with only using BART as a backbone, it is surprising to achieve this great performance boost with the top-down/bottom-up models on long document summarization when they compared to the state-of-the-art transformer models (BigBird, Longformer and T5) that have been shown to encode longer sequences and beat several summarization models.
Decision: Reject