Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work proposes an architecture to generate summaries for multi-participant postings as in chat conversations. Previous work tackled this problem as a pipeline of disentangling threads from mixed chat posts and then generating a summary for each post. The presented work proposes a hierarchical attention model to solve this task in an end-to-end fashion. 

The proposed architecture is divided into :

- a hierarchical encoder: encodes each sentence to a vector using bi-lstm and average pooling which is then fed to a post-to-post encoder which generates a set of representations corresponding to those posts.

- a hierarchical decoder: number of disentangled posts are generated using a thread-to-thread decoder which given the max pooled representations from the post-to-post encoder generated a number of initial representations for each thread summary using hierarchical attention combining attention over paragraphs and over words at each decoding time step. 

Experiments: 
Authors demonstrate the effectiveness of their approach by comparing against a set of baselines traditional seq2seq, as well as cluster-> seq2seq and the model from Shang et al. (2018). Evaluation datasets consisted of several datasets for chat summarization synthesized to mimic interleaved conversation with different difficulties. 

Through an ablation study, authors also show the effectiveness of the hierarchical encoding and hierarchical decoding as well as the post level and high-level attention. 

overall:
Although this paper is well written and well-motivated. The work is a bit incremental and builds upon previous ideas from hierarchical attention literature to apply for interleaved text generation. The provided baselines are quite weak compared to the SOTA summarization methods at the moment, although none of them is directly modelled for interleaved text summarization through multi summaries. 

typos: 
* Nallapati et al. Nallapati et al. (2016)
* See et al. See et al. (2017)


Question: 
Is there an intuition behind penalizing a model that generates correct summaries for each thread with different order? Do references summaries follow the same order as in the threads in the conversation?

In the paper you dedicated several paragraphs to reimplement basic models such as seq2seq or (See et al.) pointer generator network and make sure they work correctly. There are existing implementations and pre-trained models available such as in OpenNMT. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a technique for generating summaries of interleaved texts. Unlike previous work that first perform unsupervised clustering to extract ordered threads, the authors instead propose a hierarchical model that directly process interleaved threads. In particular, the authors propose using a hierarchical encoder that encodes words -> post -> threads as well as a hierarchical decoder that decodes threads -> words. On synthetic interleaved datasets composed from PubMed and StackExchange, the proposed method outperform seq2seq with attention as well as a pipeline system that applies unsupervised clustering followed by seq2seq. The authors present results for well-chosen ablations as well as baselines. 

My concerns stem from the synthetic nature of these tasks. In particular:

- is it difficult to disentangle the threads? Suppose we train a simple, supervised model instead of using unsupervised clustering and run the same pipeline experiment, how well would the model perform compared to hier2hier?
- due to the nature of these two tasks, is disentanglement even helpful for summarization? If not, heir2hier might be picking up on helpful signals that are not accessible to baselines. I don't have a good idea of what the tasks look like from reading the paper. I would ask that the authors put some examples of input output pairs in the paper. The algorithm in the appendix for interleaving is not very helpful.
- I would like to see some upperbound of running seq2seq on ground-truth disentangled threads.

Some comments on the writing:
- Figure 4 and Table 2 are way too small
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: <Strengths> 
+ This paper addresses the problem of summarizing interleaved posts, which involves two subtasks of identifying the clusters of threads and summarizing each thread into a sentence. 
+ This is an understudied area of research in summarization research, only few previous works have been done; for example, (Shang et al. 2018). The proposed approach is end-to-end  trainable, whereas (Shang et al. 2018)’s method tackles the problem in two separate stages. 

<Weakness>
1. The technical novelty may need to be further justified.
- The proposed model (in Fig.2) consists of the encoder, the decode and hierarchical attention module. 
- The encoder and the decoders are standard ones based on (Nallapati et al. 2017) and (Li et al. 2015), respectively. It is hard to find any technical novelty here. 
- This paper argues that the proposed hierarchical attention is novel with no clear ground. It simply mentions the inspiration from (See et al. 2017), but does not explicitly what aspects are novel as Hierarchical attention has been popularly used in general seq2seq models for many NLP applications. 
- Table 1 is rather cursory and incomplete as it compares with only three different models.

2. Experiments have much room for improvement. 
- The two datasets are simply mixture of text from existing datasets and rather artificial and far from actual interleaved posts, as shown in Table 2. 
- In page 6, three different corpora are introduced with varying interleave parameters. However, the parameters a,b,m, and n are rather small (<= 5), so it is hard to say the configurations are challenging.
- The SOTA model (Shang et al. 2018) is only compared on the PubMed Easy Corpus in Table 3. Is there any special reason to exclude it from the comparison with more challenging corpora? 
- In most results of experiments, the paper focus on showing that the proposed hierarchical attention model is better than basic seq2seq, which may not be strong results to show the effectiveness of the proposed model.

<Minor comments>
- This draft does not seem to follow the ICLR format.
- Fig.2-4 are hard to see in details. They should be enlarged and re-organized. 
- References should be added to Table 1. 
- This paper exceeds the limit of 8 pages. I cannot find any good strong ground to have more additional pages than the commended length. 

<Conclusion>
My rate is borderline with slightly leaning toward reject, mainly due to issues of technical novelty and evaluation. I will decide my rate finally, based on the authors’ response to the concerns above. 

review_title: Official Blind Review #1786

Meta-review: This paper proposes an end-to-end approach for abstractive summarization of on-line discussions. The approach is contrary to the previous work that first disentangles discussions, and the summarizes them, and aims to tackle transfer of disentanglement errors in the pipeline. The proposed method is a hierarchical encoder - hierarchical decoder architecture. Experimental results on two corpora demonstrate the benefits of the proposed approach. The reviewers are concerned about the synthetic nature of the datasets, limited novelty given the previous work, lack of clear explanation of whether disentanglement is actually needed for summarization, and simpler baselines in comparison to the state-of-the-art. Hence, I recommend rejecting the paper.
Decision: Reject