Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an explainable abstractive summarization framework by exploring the two research questions: RQ1 and RQ2. Experiment results on CNN/DailyMail dataset are given to demonstrate the effectiveness of the proposed SP framework.
strength_and_weaknesses: It is nice to propose a novel framework to explore the interpretability of abstractive summarization on a public benchmark dataset. The proposed framework is novel. But some explanations are lacking in this paper,
1. why neural modular tree could be employed in this paper? Could you give some explanations?
2. In Appendix A, the top-k sentences are selected by computing ROUGE-1 with a summary. Why is "ROUGE-1" used as the evaluation metric?
clarity,_quality,_novelty_and_reproducibility: This paper attempts to explore the interpretability of abstractive summarization with neural modular tree. An efficient best-first method is presented based on human-written abstractive summary to answer whether SPs could effectively present the summarization process; then a summarization program generation model is proposed to generate SPs from a source document. The proposed model could help to simulate abstractive summarization behavior, and also improve the interpretability of the summarization model.
summary_of_the_review: This paper proposes a summarization program, which is an interpretable modular framework to explain the abstractive summarization. SP-SEARCH could identify SPs for human-written summaries, then a summarization program generation model is generated with SPs. Experiment results are given to demonstrate the effectiveness of the proposed approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a system/paradigm called “summarization programs”. The goal is to build an interpretable search-then-assemble program for abstractive summarization. Best-first search is deployed to find the the neural module operations, including compression, fusion, and paraphrasing. There are comprehensive analysis and results about the proposed model and its ablated variants. 
Although the system does not beat end-to-end SOTA models, it provides a data point for building modular text generation systems. The idea behind is novel and exciting, and the author explains the design choices well. I would like to accept this paper.

strength_and_weaknesses: Accept:
The paper introduces the notion of “program” to decompose the generation of summaries. It’s an exciting direction that blurs the boundary of extractive and abstractive, end-to-end and modular summarization systems. 

The paper carefully chooses components and explains the reason behind the choices. 

Reject:
There are some reasons to “reject” this paper, but they are mostly inevitable on the way building a complicated modular systems. 

The search algorithm and process is very complicated, and I intuitively can imagine how fragile it is. 
The experimental results (non-oracle) are not competitive. 

The definition and derivation of the three meta operations are worth further consideration. For example, compression and paraphrasing are highly overlapped with each other. The granularity of operation can also be more fine-grained (EDUs, clauses, etc.).

clarity,_quality,_novelty_and_reproducibility: The analysis of search could be more visualized and comprehensive. Some interesting baselines to see: an always-right-branched search tree, or a binary fusion tree like fusion(fusion(0,1), fusion(2,3)). I am not suggesting that you need to do these during the rebuttal. The concern I have in mind is: what’s the added value of the search program/function, and how hard to get it work. If there is no easy recipe to tune {top-k, filtering, PQ, height, etc.} on a new dataset, it’s hard to build real systems upon this paper. 




Two missing references:

Learning to Fuse Sentences with Transformers for Summarization
Lebanoff et al.

Single Document Summarization as Tree Induction
Liu et al. 
summary_of_the_review: It's a strong paper with novel ideas and solid experiments. Although the system does not beat end-to-end SOTA models, it provides a data point for building modular text generation systems. The idea behind is novel and exciting, and the author explains the design choices well. I would like to accept this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new framework, summarization programs (SPs), for interpretable summarization. It starts with source document sentences as leaves, applies a series of transformations of fusion, compression, and paraphrasing, and finally leads to the root nodes for each summary sentence. Such design makes the generation process clear to humans and makes it easier to catch any hallucination created along the way. The authors come up with a best-first search algorithm, SP-SEARCH, to take document sentences and gold summaries as input, and output SPs for supervised training. They also attempt to train models to generate the SPs, and conduct simulation studies to gauge the interpretability of the summarization. Nevertheless, their results still lag behind those of the SOTA models.
strength_and_weaknesses: Strength:
‒ SP is an interesting and novel idea 

‒ The design makes it much easier to trace the generation process, and to manually detect if any hallucination has occurred; the method helps to limit the issue of hallucinations down to mostly a single module (fusion)

Weaknesses:

On experiments for RQ2

‒ The results for generating SPs are much weaker than the SOTA both in terms of fluency and factuality: as compared to the model for the base architecture (BART), the best SP model experience a 6.53 drop in Rouge-1 and a 2.27 drop in QuestEval. Even though the authors show the Oracle scores to be much higher, it is not guaranteed if such potential can be realized since the Oracles make use of gold in the first place.

‒ From what I understand, for the RQ2 experiments, authors only finetuned BART for generating SPs (or used an existing model for the sentence extraction stage). There is little novelty and the approach seems unnatural. It is possible that the target output (SPs) differs too much from the pre-trained text. The three modules (fusion, compression, and paraphrase) are decoupled from the SP-generating module, and it is possible that the SP-generating module is totally unaware what are the exact implications of its generated modular instructions. 

‒ The authors used three separate large neural models for paragraph, compression, and fusion; from the perspective of parameter sizes, it may be unfair to compare the performance of the SP Models to the BART or PEGASUS models. 

On experiments for RQ1

‒ It is not explicitly stated in the paper, but from what I understand, the BART-Oracles (Beam/Sample) are first fine-tuned on the training set and then used for generation? In this case, it is not surprising that it should have worse results than SP-SEARCH, which refers to the gold summaries for practically every step of its search process.  
clarity,_quality,_novelty_and_reproducibility: ‒ Overall the description is clear and easy to follow, with a good illustration of SPs 

‒ Authors explain evaluating Rouge-L during SP-SEARCH leads to better performance, but it is unclear why Rouge-1 is used to first select the top-k document sentences

‒ It is unclear, for the first research question, whether the BART-Oracle is fine-tuned on the training data, whether the score for beam search is an average of the top-10 summaries or the best of all  10 summaries.  

‒ Although the idea is quite new, there is a lack of novelty in terms of the models used.

‒ The paper is likely to be reproducible as the experimental settings are reported in detail. 

summary_of_the_review: Overall this paper proposes an interesting idea, and the writing is easy to follow. Most details on the experiments are reported clearly, and there is extensive analysis of the results as well. However, there is limited novelty in the model, and the proposed approach is too simple and cannot give good results as compared to the SOTA models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to model summaries in the form of "summarization program", a sequence of 3 actions (compression, fusion, paraphrase) over sentences. Such an approach has the advantage of being more interpretable, as the final summary comes together with a "proof" on how it was obtained. 
This open very interesting avenues for interactive summarization, as well as inspection of the intermediate steps performed by the system.

Two types of experiments are reported in this paper:
1. An exploratory one, which aims to evaluate what could be achieved with such an approach. This is done in by optimizing for the oracle summary.
2. A predictive one, which evaluates the summaries predicted on unseen example (the more traditional ML approach)
strength_and_weaknesses: Strengths

1. This is a very original paper, in an undervalued research line - that of interpretable summarization. The proposal is compelling and its advantages well argued for
2. The empirical results showing the potential of proposed model are promising. The best obtainable summary is +20 Rouge-L points above what can reasonably be expected to be obtained with state-of-the-art methods

Weakness
1. The second set of experiments are underwhelming. Not only do they perform rather far from current pure neural methods, they do so even when using those as starting point: "Extract-and-Build" (RL=21.8) starts from the extractive summary obtained through MatchSum (RL=28.7). A very ruthless reading of Table 3 would be that the proposed technique starts from something good and turns it into something worse, although not as bad as random selection.
The authors discuss why this could be the case, but the results in the paper do not indicate that this gap could be overcome.
2. Those underwhelming results come as a total surprise to the reader, as they are not forshadowed in the abstract or introduction. This disappointment could easily be avoided, at the cost of turning away those readers that only come for the SOTA results (which would in any case not be interested after finishing the paper)
3. The simulation protocol is very interesting to study the intepretability of SP programms, but it doesn't seem to me that too much should be read into it. In short, in the study humans have to predict the program before and after seeing the program explained by the proposed system. The Rouge score of the summaries obtained by the human-chosen program _after_ the training phase is slightly better. However, it should be noted that the 3 humans who performed this were authors of the paper. So, on one side they were clearly not neutral, and on the other hand they probably started with a very good understanding from the form of the program (therefore the difference might be smaller than with non-experts).
4. Experiments are only reported on one dataset, and the last years of summarization papers have but the bar higher than that.

clarity,_quality,_novelty_and_reproducibility: Clarity: extremely clear. The paper proposes a non-trivial pipeline, but is detailed without loosing the big picture
Quality: high
Novelty: very original
Reproducible: I would rate this low, accompanying source code. There are lots of small design choices which have been taken - and while mentioned - I would expect those to be hard to reproduce

Minor comments (not sure where to put those):
- authors mention that the paraphrase module also performs compression. This does not seem surprising, as it starts from a summarization model (Pegasus)
- on “When none of the top-k outputs is well-formed (<1% of the samples), we generate the corresponding extractive summary” -> 
what is the corresponding extractive summary for “Joint Summarisation Program Generation Model”, which does not generate an intermediate extractive summary?

summary_of_the_review: An original proposal, with lots of potential for interpretability work. The search-space seems wide enough to also be possible to use the method to obtain SOTA results - as measured by Rouge scores. However, this potential is far from being realized in thie paper.

My recommendation is still for acceptance, because of its originality and because it takes "the path less trodden".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea of neural modular trees (adopted from neural modular nets) for interpretable summarization is interesting and novel. It shows a new way for abstractive summarization that can influence future work. I also found the SP-SEARCH module interesting and significant. I am in favour of accepting this paper. 
justification_for_why_not_higher_score: all high 
justification_for_why_not_lower_score: NA
Decision: Accept: poster