Generating Multi-Sentence Abstractive Summaries of Interleaved Texts
Authors: Sanjeev Kumar Karn
Francine Chen
Yan-Ying Chen
Ulli Waltinger
Hinrich Schütze
Venue:  (2020)

Abstract: In multi-participant postings, as in online chat conversations, several conversations or topic threads may take place concurrently. This leads to difficulties for readers reviewing the postings in not only following discussions but also in quickly identifying their essence. A two-step process, disentanglement of interleaved posts followed by summarization of each thread, addresses the issue, but disentanglement errors are propagated to the summarization step, degrading the overall performance. To address this, we propose an end-to-end trainable encoder-decoder network for summarizing interleaved posts. The interleaved posts are encoded hierarchically, i.e., word-to-word (words in a post) followed by post-to-post (posts in a channel). The decoder also generates summaries hierarchically, thread-to-thread (generate thread representations) followed by word-to-word (i.e., generate summary words). Additionally, we propose a hierarchical attention mechanism for interleaved text. Overall, our end-to-end trainable hierarchical framework enhances performance over a sequence to sequence framework by 8-10% on multiple synthetic interleaved texts datasets.

Under review as a conference paper at ICLR 2020
GENERATING MULTI-SENTENCE ABSTRACTIVE SUM- MARIES OF INTERLEAVED TEXTS
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-partie multi-particciant postings, as in online chat conversations, several conversa- tions or topic threads may take place concurrently. This leads difficulties for readers reviewing the postings in not only following discussions but also in quickly identifying their essence. A two-step process, disentanglement of inter- leaved posts followed by summarization of each thread, addresses the issue, but disentanglement errors are propagated to the summarization step, degrading the overall performance. To address this, we propose an end-to-end trainable encoder- decoder network for summarizing interleaved posts. The interleaved posts are en- coded hierarchically, i.e., word-to-word (words in post) followed by post-to-post to-post (posts a channel). The decoder also generates summaries hierarchically, thread- to- to-thread (generate thread representations) followed by word-to- word (i.e., gener- ate summary words). Additionally, we propose a hierarchical attention mechanism for interleaved text. Overall, our end-to-end trainable hierarchical framework en- hances performance over a sequence to sequence framework by 8- 1-10% on multiple synthetic interleaved texts datasets.
INTRODUCTION
Interleaved texts are becoming more common with new ways of working and new forms of com- munication, e.g., multi-author entries for activity reports, meeting minutes and social media conver- sations, such Slack. Quickly getting sense of following the content of different threads in interleaved texts, where posts belonging to different threads occur one sequence, often difficult. An example of two threads with multiple posts interleaved to form sequence is:
Postl -Thread Postl-Threaid Thread2 - Post3-Threade
Post2-Threadd - Bost-TTreadd - Post2-
This intermingling leads to difficulties in not only following discussions but also in retrieving their essence. conversation disentanglement, interleaved posts are grouped by the thread; e.g., the previous example could be rearranged as:
Postl-Thread + Post2-Thread - Post3-Thread Post1-Threaad - Sarea-2 - Post3-Thread2
In analyzing interleaved texts, Shang al. (018)) went step further and proposed summarization of the interleaved texts. They designed an unsupervised two-step system and evaluated the system on meeting texts. In the first step, a conversation disentanglement component disentangles the texts thread-wise. Then, in the second step, multi-sentence compression component compresses the thread-wise posts to single sentence summaries. However, this system has major disadvantage, in that the disentanglement obtained through either supervised Jiang et al. (2018) unsupervised Wang & Oard (2009) methods propagate its errors to the downstream summarization task, and thus, degrades the overall performance.
We aim to tackle this issue error propagation through an end-to-end trainable encoder-decoder system that takes variable length input, e.g., interleaved texts, processes it and generates variable length output, e.g., a multi- summary; see Figure [or an illustration. An end-to-end system eliminates the disentanglement component, and thus, the error propagation.

---

Under review as a conference paper at ICLR 2020
John bought car Battmore yesterday.
John bought
betrayed bought
you mean he's not going buy my car? (
that's why missed
game.
the poker game yesterdey.
Did visit my Aunt Minnie too?
Did
That was my money he used.
John dian' visit and paint aunt Minnie house Ballinore.
put up He was supposed today, he probably didntt paint the
thing
put up
supposed today probobly didin
house after
Fouse after
7 interleaved posts are implicitly disentangled into 3 threads, and single sentence sum- maries are generated for each thread. Posts outlined with colors corresponding the color of each summary.
Figure I:
The encoder first performs word-to-word encoding to embed each post, followed by post-to-post encoding embed the overall content of the posts and represent the discourse structure of the interleaved texts. The decoder has thread-to-threat to-tthreed decoder to generate a representation for each thread, and the thread representation is given to a word word decoder generate summary sentence. We also propose to use hierarchical attention similar to Nallapati et al. (2016), but instead of computing post-level attention every word, attentions are only computed when decoding new sentences. Further, the attention networks are trained end-to-end.
Despite the availability a multitude real-world interleaved texts, a major challenge to train encoder-decoder models is the lack of labels (summaries). As labeling is difficult and expensive Barker et al. (2016); Aker et al. (2016): Verberne et al. (2018), we synthesize two separate corpora. Each is derived by mixing texts and associated summaries from corpus of documents, where the mixed text has a structure reflective of multi-party conversation with inter-lcaved threads and the summary highlights the information in the threads. We find abstracts and titles of randomized controlled trial (RCT) articles, a PubMed corpus, and also for Stack Exchange posts and titles.
To summarize, contributions are threefold:
We propose combine hierarchical encoder and decoder to obtain multi-sentence sum- maries of interleaved texts.
We propose novel hierarchical attention mechanism that is integrated with the hierarchical encoder-decoder architecture and which equips the decoder to disentangle the interleaving further.
We use two synthetic datasets to verify the ideas and show our end-to-end trainable archi- tecture addresses not only the issue of error- proorproopagation observed in competitive methods but also improves the performance.
RELATED WORK
Quite often multi multi-party conversations, e.g. news comments, social media conversation and activity report, have tens of posts discussing several different matters pertaining to a subject. Ma et al. (2012): Aker et al. (2016); Shang al. (2018) designed methodologies to summarize posts order to provide an overview on the discussed matters. They broadly follow the same approach: cluster the posts and then extract a summary from each cluster.
There are two kinds of summarization: abstractive and extractive. In abstractive summarization. the model utilizes a corpus level vocabulary and generates novel sentences as the summary, while extrac- tive models extract rearrange the source words as the summary. Abstractive models based ral sequence- requence-to-sqqeenee (seq2seq) (Rush et al. (2015)) proved to generate summaries with higher ROUGE scores than the feature-based abstractive models. Integration of attention into seq2seq
2

---

Under review as a conference paper at ICLR 2020
(Bahdanau al. (2014)) led to further advancement of abstractive summarization (Nalappti et al. Chopra et al. (2016)).
There are many possible patterns of organization of the information in texts, c.g., chronological pattern. News articles have an inverted pyramid pattern, i.c., core information in the lead sentences and the extra information in later sentences. A seq2seq model is appropriate for summarization of a news article as it encodes and decodes sequentially. However, in interleaved texts, related information maybe separated; thus seq2seq model may be competent but sufficient.
Li et al. 2013) proposed an encoder-decoder (auto-encoder) model that utilizes a hierarchy of net- works: word-to-word followed by sentencc-to-sentenee Their model is better at capturing the derlying structure than vanilla sequential mucoder-decoder model (seq2seq). Krause et al. (2017); Jing et al. (2018) showed multi-sentence captioning of an image through hierarchical Recurrent Neural Network (RNN), topic-to-topic followed by word-to-word, is better than seq2seq.
These works suggest a hierarchical encoder, with word-to- word encoding followed by post-to-post, will better recognize the dispersed information in interleaved texts. Similarly, hierarchical decoder, thread-to-thread followed by word-to-word, will intrinsically disentangle the posts, and therefore, generate more appropriate summaries.
Nallapati et al. (2016) devised hierarchical attention mechanism for seq2seq model, where two levels of attention distributions over the source, i.e., sentence and word, are computed at every step of the word decoding. Based on the sentence attentions, the word attentions are rescaled, Hsu et al. 2018 slightly simplified this mechanism and computed the sentence attention only at the first step. Our hierarchical attention more intuitive and computes new sentence attentions for every new summary sentence, and unlike Hsu et al. (2018), is trained end-to-end.
MODEL
word-to-word emooder
posl-toppual enooder
thwead-t-ttraed decoder
word to word decoder
bought beltrarn yesardes
balkayel and bought
Und be rial Minnwioo?
Juler vout sd sual Minaac
Figure 2: Hierarchical encoder-decoder architecture.
PROBLEM STATEMENT
We aim to design a system that when given a sequence of posts, C'= (PI...., produces a sequence of summaries, = (Si..... S(r) For simplicity and clarity, unless otherwise noted, we will use lowercase italics for variables, uppercase italics for sequences, lowercase bold for vectors and uppercase bold for matrices.
Figure 2 illustrates the proposed hierarchical encoder-decoder framework. In the framework, first, low-level, word-to-word encoder converts a sequence of words in a post, Py, to a sequence of repre- sentations, H, (h w2w h w2w JIP,
Subsequently, a top-level, post-to-post encoder converts those (2.2...... p2pp These encoded representations are then passed to the decoder, which utilizes a top-level, thread- to-thread, decoder disentangle them into sequence of thread representations (h/2t. ..... h )-
representations, (H..... H. to a sequence of top-level post representations (h)²p ,...,
Finally, a low-level, word-to-wond tord, decoder takes a feed-forward mapped thread representation h and generates sequence of summary words (Wa....., (sss;))-
The maximum number of posts in a sequence of interleaved texts is denoted by n and threads by We limit the number of words in posts and summaries to fixed lengths by either truncating or
3

---

Under review as a conference paper at ICLR 2020
D
D
Rno
h
Figure 3: A step in the thread-to-thread decoder. Figure 4: Hierarchical attention mechanism for hierarchical encoder.
padding, and denote them by p and q respectively. The hidden states of the encoder and decoder have dimensionality L.
ENCODER
The hierarchical encoder used in Figure 2 is based on Nallapati et al. (2017). where word-to- word and post-to-post encoders are bi-directional LSTMs. The word-to-word Bi-LSTM encoder (Ew2w) runs over word embeddings of any post, Pj. and generates a set of hidden representations ((h Ewaw ..... h )); see left middle section in Figure 2 The average pooled value of the word- to-word representations of post P, ( Eh=
jk h Fuuw) are input to the post-to-post Bi-LSTM encoder
(Ezz)). which then generates a set of representations ((h Frep ..... "ah,,....,hhh ( corresponding to those posts. Overall, output word-to-word representations, W, has dimension n X p and post-to-pon post representations, P, has dimension n x 2/ for given channel, C.
DECODER
Our hicrarchical decoder (see Figure 2 based on on et al. (2015), where both thread-to- tto-tttootdddd and word-to- sord-to-word decoders are uni-directional LSTMs.
The initial state ho Dizs the thread-to-thread LSTM decoder (fat)) is set with a feedforward- mapped representation of an average pooled post representations (c' = _ Eh=ob h(²²). At each
step i of the decoder, sequence of attention weights, (B..,... ... (8......,,,,xxx. B,nx)), corresponding to the of encoded word representations, (he2e ..... h are computed utilizing the previous state, h Pe We will elaborate the attention computation in the next section.
A weighted representation of the words (crossed blue circle) is then computed: Σ/= ig. and used as one input to LSTM Pra., Additionally, we use the last hidden state h of the word- to-word decoder LSTM (Dwaw. of the previously generated summary sentence as the second input compute the next state of thread-to-thread decoder, i.c., h Peze The motivation to provide information about the previous sentence.
The current state h Pi2s is passed through a single layer feedforward network and a distribution over STOP=1 and CONTINUE=0 is computed:
STOP = (g (h(x2))
p
where g is feedforward network. In Figure the process is depicted by red circle. The thread- to-thread decoder keeps decoding until p, STOT is greater than 0.5.
Additionally, the current state h Pi2i is passed through another single layer feedforward network
followed by tanh activation to compute the thread representation s; = tanh(k h Peze
Given thread representation s,, the word-to-word decoder generates summary for the thread. Our word-too-ord decoder is based on Bahdanau et al.) (2014). It is a unidirectional attentional LSTM (fDwz ) see the right-hand side of Figure We refer to Bahdanau et al. (2014) for further details.
4

---

Under review as a conference paper at ICLR 2020
HIERARCHICAL ATTENTION
Our novel dynamic hierarchical attention is based on two popular concepts: attention scores should integrate in top-down manner (Nallapati al. (2016) and two distributions same words must be modeled together (See et al. (2017)) )), not just in isolation.
Our hierarchical attention works 3 levels, the post level (corresponding posts), i.c., y, and higher level (corresponding to souree tokens), i.e., B, are computed while obtaining a thread repre- sentation, s. The lower level attention (also corresponding source tokens), i.c., a, computed while generating word, of summary, S.; see yellow rectangles in Figure
At step i of thread decoding. we also compute elements of post-level attention, i.e., Yr as below.
Ki = a (hPrr, j (1,....,,
At the same step, also compute elements of high- attention, i.c, Bg as below.
Bik (atttn (h ak))
(L.....,7 p}
(3)
a (W P)) where == and E {1, (1....,,1) n), l (1.....p)
(4)
where ag is computed as in Eq. W in Eq. sion p 21, add aligns a post representation its constituting word representations and does element-wiss wise addition, and attn feedforward network that aligns the current thread decoder
4 is a word-to-word encoder representations of dimen-
state Pige with all n number of ag vectors.
Then, y rescale high-level attention, 33 below.
Bi Bik * Tiy where == and E {I.....p) p}} j E (1......m]
(5)
At step of word-to- wordd decoding of summary thread i, compute elements of low level atten- tion, i.e., Ctg. as below.
exp(enrk)
Otitk
k=1 exp(exx) where eark attn ( (h Puzu ax)
(6)
and as same as in Eq. and altn is feedforward network that aligns the current word decoder state h Pap-- with all p number of ag vectors.
Finally, we use rescaled high-level word attentions, 3, for rescaling low level attention, a as below:
Be x exp(exx)
Σ R.p k= 3x x exp(ex*)
Table I compares our hierarchical attention with previous work on hierarchical attention in the text summarization domain.
Table 1: Forms of hierarchical attention. End2End=No training labels, x woods=for every word in
target, x surmearies=for every summary in the target, D=ssentence(poss)/lveel,,, B=high-level
Model
End2End Decoding 7 computations yes no hier
Nallapati et al. (2066) Cheng & Lapata 2016 Tan ct al. 2017)
seq
words
no
Isu et al. 2018
seq Xsumaariss yes
Ours
yes
TRAINING OBJECTIVE
We train our hierarchical encoder-decoder network similarly to an attentive seq2seq model Bah- danau et al. 2204), but with additional weighted of sigmoid cross-entropy loss stopping distribution; see Eq. [ Given a summary, Y, = (wso,..., m%g), our word-to-word decoder generates
target y = (ıho, ... Mig), with words from same vocabulary U. We train model end-to-end by minimizing the objective given in Eq.
m g
m
log pe (Wg. W) + STOP log(p, STOP)
5

---

Under review as a conference paper at ICLR 2020
The left rows contain interleaving of 3 articles with to 5 sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols.
Table 2:
this study was conducted evaluate the influence c... to assess the effect of a program of supervised fitness...
V caffeine in sport. influence of endurance exercise the urinary caffeine concentration.
an 8-week randomized, controlled trial
nine endurance- trained athletes participated in randomised...
supervised fitness walking in putients with osteoarthritis of the knee. randomized, controlled trial..
* examined the effects of intensity of training ratings...
* subjects recruited sedentary controls domly...
the group trained velocity and greater than... data were obtained on 47 51 intervention patients and 45...
effect of training intensity on ratings perceived exertion.
DATASET
Obtaining labeled training data for conversation summarization is challenging. The available ones are either extractive /Verberne et al. (2018) or too small (Barker et al. to train a neural model. To get around this issue and thoroughly verify the proposed architecture, we synthesized a dataset by utilizing corpus of conventional texts for which summaries are available. We create two corpora of interleaved texts: one from the abstracts and titles of articles from the PubMed corpusand one from the questions and titles of Stack Exchange questions. A random inter- leaving of sentences from a few PubMed abstracts or Stack Exchange questions roughly resembles interleaved texts, and correspondingly interleaving of titles resembles its multi-sentence summary.
et al. (2012)
The algorithm that we devised for creating synthetic interleaved texts is defined in detail in the Appendix. The number of abstracts include in the interleaved texts is given as a range (from a to 6) and the number of sentences per abstract to include is given as second range (from m to n). We vary the parameters below and create three different corpora for experiments: Easy (a=2, b=2, m=5 and n=5), Medium (a=2, b=3, m=2 and n=5) and Hard (a=2, b=5, m=2 and n=5). Table shows an example of a data instance in the Hard Interleaved RCT corpus.
EXPERIMENTS
we report ROUGE-1, 1, ROUGE-2, and ROUGE-L as the quantitative evaluation of the models. The Pubmed Medium and Hard corpora train, test and validation have approximately 290k, 6k and 1.5k instances, respectively. The Stack-Exchange Medium corpus train has 180k and Hard has 210k instances, while the both corpora have test and validation of approximately 5k and 4k instances, respectively. The remaining
are described detail in the Appendix.
BASELINES
We first trained and tested it on the Easy corpus. We also ran Shang al. (2018)'s unsupervised two-step system on the test set of the Easy Interleaved Pubmed corpus. Additionally, we also utilized Shang et al. (2018) 's clustering component to first cluster the interleaved texts of the corpus, and then the disentangled corpus is used to train the seq2seq model. We refer to the latter cluster >segssg The performance comparison of [Shang et al. (2018) and the two seq2seq models are shown in Ta- ble Clearly, seq2seq performs better than Shang al. (2018), the reason being seq2seq model trained on a sufficiently large dataset is better at summarization than the unsupervised sentence com-
Table 3: Rouge F1-Scores for seq2seq models on the Pubmed Easy Corpus.
Table 4: Recalls sampled Pubmed Hard- Corpus. Int=Interleaved, Dis=Disentangled.
Model
Rouge-1
Rouge-2
Rouge-L
Model
type Rouge-1
Rouge-2
Rouge-L
Shang et al. 2018
30.37
10.77
20.04
seq2seq
Int
32.40
11.09
22.97
seq2sey
44.38
19.47
35.20
hier2hier
Int
33.51
12.50
24.27
Ssster->seq2se
42,93
18.76
30.68
seq2seq
Dis
37.01
14.44
26.91
hier2hier
Dis
37.68
15.11
28.25

---

Under review as a conference paper at ICLR 2020
pression (extractive) method. The lower performance of cluster->seg2seer in comparison to seq2seq shows that a disentanglement component unnecessary in easy scenario.Furthhermocs Furthermore, we utilize the ground-truth labels and disentangle the interleaved posts of a sampled (150k) Hard Corpus, and re-run the seq2seq model. The results (see Table show seq2seq (row 3) takes advantage of cluster labels but hier2hier (row 4) does more than it.
SEQ2SEQ vs. HIER2HIER MODELS
We then compare the proposed hierarchical approach against the seq-to-seq approach in summariz- ing the interleaved texts by experimenting on the Medium and Hard corpora.
Table shows the experimental results several corpora; clearly, increase in the complexity of interleaving scantly impacts the performances. Though, in case of Pubmed, seq2seq depreciates by 1-2 points while hier2hier remains nearly same with increase of complexity. In Stack Exchange corpora, as Hard corpora has more training data (+30k) seq2seq improves slightly. However, a noticeable improvement is observed changing the decoder to hierarchical, i.e., 1.5-3 Rouge points in Pubmed and 2-4.5 points in Stack Exchange depending on the interleaving complexity.
Table 5: Rouge Recall-Scores of models on the Medium and Hard Corpus.
Corpus Difficulty
Pubmed Rouge-1
Stack Exchange
Model
Rouge-2
Rouge-L
Rouge-1
Rouge-2
Rouge-L
seq2seq
30.67
11.71
23.80
18.78
03.52
14.73
Medium
hier2hier
32.78
12.36
25.33
24.34
05.07
18.63
seq2seq
29.07
10.96
21.76
20.21
04.03
14.93
Hard
hier2hier
33.36
12.69
24.72
24.96
05.56
17.95
To understand the impact of hierarchy the hier2hier model, perform ablation study and use the Hard Pubmed corpus for the experiments, and Table (6 shows the results. Clearly, adding hierarchical decoding already provides boost in the performance. Hierarchical encoding also adds some improvements to the performance; however, the enhancement attained training and infer- speed by the hierarchical encoding is much more valuable. We will discuss in depth later.
Further, we also compiled corpora to mimic real world conversation interleaving, wherein the se- quence of summaries for interleaved posts may follow the sequential occurrence of posts, 17-18 in Algorithm. T An example, a post mention an action in the very beginning of conversation, but does only elaborates at the end. Therefore, summary corresponding to the action should be at the end. So, we again create Medium and Hard real world mimicking corpora of the Stack Exchange dataset as it is more similar real world conversations, and perform the same set of experiments. As seen in Table 7 the results show that both the seq2seq and hier2hier models performance are slightly lower as is tougher task; see Table addition, the hier2hier model still consistently better than the seq2seq model.
Importantly, hier2hier converges much earlier than the seq and also reaches lower training loss (Figure 5 Appendix). hier2hier takes x1.5 days for training on Tesla V100 GPU, while seq2seq takes 244.5 days. Therefore, the hier2hier model not only achieves greater accuracy but also reduces training and inference time.
Table 6: Rouge Recall-Scores of models on the Pubmed Hard Corpus.
Table 7: Rouge Recall-Scores of models on the Stack Exchange Medium and Hard Corpus.
Medium Corpus
Pubmed Hard Corpus
Model
Rouge-1 I Rouge-2 Rouge-I.
Model seq2seq
Rouge-1
Rouge-2 Rouge-L L
19.67
03.88
15.37
seq2scq
29.07
10.96
21.76
seq2hier
32.92
11.87
24.43
hier2hier
23.97
05.63
18.75
hier2seq
31.86
11.9
23.57
Hard Corpus
hier2hier
33.36
12.69
24.72
seq2seq
19.62
03.71
14.90
hier2hier
24.14
05.00
17.25
7

---

Under review as a conference paper at ICLR 2020
HIERARCHICAL ATTENTION
The contribution of the post-level and high-level attentions a hierarchical decoder is two-fold: computing the thread representation and rescaling the word-level attentions. To understand the impact of hierarchical attention on the hier2hier model, we perform an ablation study of post-level attentions (r) and high-level attentions (3), using the Hard corpus for the experiments.
Table 8: Rouge Recall-Scores of ablated models the Hard Pubmed Corpus.
Model
Rouge-1 | Rouge-2
Rouge-L
hier2hier +7 +
33.36
12.69
24.72
hier2hier - (3)
32.65
12.21
24.23
hier2hier( +- (3)
31.28
10.20
23.49
29.83
09.80
22.17
hier2hier( (3)
30.58
10.00
22.96
seq2seq
29.07
10.96
21.76
Table [| shows the performance comparison. Clearly, 7 attention improves the performance (0.5-1) of hierarchical decoding but not lot. The high-level attention, i.e., / is very important as without the model performances is noticeably reduced (Rouge values decrease from 2-3). Additionally, we also include Li, Luong and Jurafsky, 2015 post-level attention technique in the comparison, as it is the closest hierarchical attention to ours (see Table []]. Though, (Li, Luong and Jurafsky) utilize softmax obtained 3 to compute thread representation, and in contrast to they do not reuse 33 for re-scaling a, and thereby, closest to hier2hier( ++y - -3). Evidently, re-utilization gives hier2hier( - (3) edge over (Li, Luong and Jurafsky, 2015). Lastly, removing both the 7 and () makes the hier2hier similar to seq2seq. except a few more parameters, i.e., two additional LSTM, and the performance is also very similar.
DISCUSSION
Table shows output of our hierarchical abstractive system, in which interleaved texts are in the top, and ground-truth and generated summarics in the bottom. Table 9 also shows the top two post indexes attended to by the post-level attention (7) while generating those summaries, and they coincide with relevant posts. Similarly, the top 10 indexes (words) the high-level attention (3)
Table 9: Interleaved sentences of 3 articles, and corresponding ground-truth and hier2hier generated summaries. The top 2 sentences that were attended (ry) for the generation are on the left. Addition- ally, top words (B) attended for the generation colored accordingly.
Interlcaved Texts
this study was conducted evaluate the influence of excessive sweating during long- distaces running on the urinary concentration of caffeine... to assess the effect of program of supervised fitness walking and patient education functional status, pain, and...
0
1
total of 102 patients with documented diagnosis of primary osteoarthritis of both knees participated....
5
6
we examined the effects of intensity of training on ratings of perceived exertion (...
GroundTrrthh/Geeeraate
caffeine in sport. influence of endurance excrcise the urinary caffeine concentration. effect of excessive [UNK] during [UNK] running on the urinary concentration caffeine.
0,2
supervised fitness walking in patients with osteoarthritis the knee randomized, controlled trial..
1,4
effect of physical fitness walking functional status pain, , and pain
the effect of training intensity ratings of perceived exertion.
6.8
effects of intensity of training on perceived [UNK] [UNK] athletes.
8

---

Under review as a conference paper at ICLR 2020
directly visualized in the table through the color coding matching the generation. The system not only manages to disentangle the interleaved texts but also to generate appropriate abstractive summaries. Mcanwhile, 33 provides explainability of the output.
As interlcaving in the table includes abstracts from the same category, c.g. Physical Activities, the interleaving is complex and approximates the real-world conversations. Despite that, the end-to- end hierarchical system tackles the task to a large extent. The next, future stage in this research is transfer learning of the hierarchical system to more types of conversations. We find models trained for 50k iteration on Pubmed Hard corpora (a=6, b=10, 10, m=2 and n=3) and then fine-tuned on the popular meeting AMI corpus (McCowan al. (2005) already reaches competitive results (Rouge- 1=39.81, Rouge-2-11.355 against SOA system on AMI (Shang et al.] (2018) Rouge-1=37.8m and Rouge-2=07. 84). Additionally, we aim to modify hier2hier to include some of the recent additions of seq2seq models, e.g., SSee et al. (2017) pointer mechanism.
CONCLUSION
We presented an end-to-end trainable hierarchical encoder-decoder architecture which implicitly disentangles interleaved texts and generates multi-sentence abstractive summary covering the text threads. Furthermore, the architecture addresses the error propagation issue that occurs the two- step architectures. Our proposed novel hierarchical attention further boosts both disentanglement and summary generation.
REFERENCES
Ahmet Aker, Monica Paramita, Emina Kurtic, Adam Funk, Emma Barker, Mark Hepple, and Rob Gaizauskas. Automatic label generation for news comment clusters. In Proceedings of the 9th International Natural Language Generation Conference, pp. 61-69, 2016.
Xavier Anguera, Simon Bozonnet, Nicholas Evans, Corinne Fredouille, Gerald Friedland, and Oriol Vinyals. Speaker diarization: review of recent research. IEEE Transactions on Audio, Speech, and Language Processing, 20(2):356-377 2012.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CORR, abs/1409. 0473, 2014. URL /tttp://axxiv.orrg
abs/ 1409. 0473
Emma Barker, Monica Lestari Paramita, Ahmet Aker, Emina Kurtic, Mark Hepple, and Robert Gaizauskas. The sensei annotated corpus: Human summaries of reader comment conversations in on-line news. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pp. 42-52, 2016.
Jianpeng Cheng and Mirella Lapata. Neural summarization by extracting sentences and words. arXiv preprint arXiv: 1603.07232733 2016.
Sumit Chopra, Michael Auli, and Alexander M. Rush. Abstractive sentence summarization with attentive recurrent neural networks. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), NAACI. HIT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp. 93 98. The Association for Computational Linguistics, 2016. URL Stthtt::
1012. pdf
Wan-Ting Hsu, Chich-Kai Lin, Ming-Yine Lee, Kerui Min, Jing Tang, and Min Sun. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings the 56th Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 132 141. Association for Computational Linguistics, 2018. URL http:://////cwee antho hwtpolloyyyP11000000
Jyun-X Yu Jiang, Francine Chen, Yan- Ying Chen, and Wei Wang. Learning to disentangle inter- leaved conversational threads with siamese hierarchical network and similarity ranking. In

---

Under review as a conference paper at ICLR 2020
Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies. Volume (Long Papers), pp. 1812-
1822. Association for Computational Linguistics, 2018. doi: 10. 10.18653// 1164. URL
http://ast //ccl web. org/ antho 1 ogy /N1 8-1164]
Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports.
In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 2577-2586 Association for Computational Linguistics, 2018. URI.
http: / aacl web. weebboogg/nttin ogy/F18-1240
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. toss/1422.6980 2014. URL http: /ettp://arxiv.eo www.oorgggaa..... abs/ 1412. 6980
CoRR,
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Patterm Recognition (CVPR), 2017.
Jiwei LI, Thang Luong, and Dan Jurafsky. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1106- 1115. Association for Computational Linguistics, 2015. doi: 10.3115/ v1/P15-1107. URL http: /ttp:://clwweee www.wb..w...e................... org/ http:/nnhoooooooo hogthholoy/P15:. 1107.
Zongyang Ma, Aixin Sun, Quan Yuan, and Gao Cong. Topic-driven reader comments summariza- tion. In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 265-274. ACM, 2012.
I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos. M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. The ami meeting corpus. In L.P.J. Noldus, F Grieco, L.W.S. Loijens, and P.H. Zimmerman (eds.). Proceedings of Measuring Behavior 2005, 5th International Con- ference on Methods and Techniques in Behavioral Research, pp. 137-140. Noldus Information Technology, 8 2005. ISBN 90-74821-71-5.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang- Abstractive text summarization using sequence-to-sequene rnns and beyond. In Yoav Goldberg and Stefan Riezler (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12. 2016, pp. 280-290. ACL, 2016. URLLpttp://aclwee org htt///nhhooooo/oyaaeo 6 fd.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI Conference on Artificial In- telligence, 2017. URL https::/aaaae http:///aaa...rroc////aaa///a//a phtp//AA///AAA11//aaaa//// wiew/14636. 14080
Alexander M. Rush, Sumit Chopra, and Jason Weston. neural attention model for abstractive sen- tence summarization. In Lluís Mårquez, Chris Callison- Burch, Jian Su, Daniele Pighin, and Yuval Marton (eds.). Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 379-389. The Asso- ciation for Computational Linguistics, 2015. URL http::t/p/// h wweb.weebb... org wwtp://oooot.oooggggooooooooocee D15/ D15 1044.14400 100-0044pp.... pdd
Abigail See, Peter J. Liu, and Christopher D. Manning. Get the point:
Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1073-1083, Association for Compu-
tational Linguistics, 2017. doi: 10.18653/1//117-1099, URL http: www. www.bb...bbbo...
aathology/E17-1093
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazir- giannis, and -annPie Lorré Unsupervised abstractive meeting summarization with multi- sentence compression and budgeted submodular maximization. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp-
10

---

Under review as a conference paper at ICLR 2020
664-674. Association for Computational Linguisties, 2018. URL http://aclweb.orrg arthology/P18-1022
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. From neural sentence summarization to headline gen- eration: A coarse-to-fine approach. In IJCAI, pp. 4109-4115, 2017.
Suzan Verberne, Emiel Krahmer, Iris Hendrickx, Sander Wubben, and Antal van Den Bosch. Creat- ing reference data set for the summarization of discussion forum threads. Language Resources and Evaluation, pp. 1-23, 2018.
Lidan Wang and Douglas Oard. Context-based message expansion for disentanglement in- terleaved text conversations. In Proceedings human language technologies: The 2009 anmual conference of the North American chapter of the association for computational linguistics, pp- 200-208. Association for Computational Linguistics, 2009.

---

Under review as a conference paper at ICLR 2020
APPENDIX
INTERLEAVE ALGORITHM
In Algorithm. T INTERLEAVE takes of concatenated abstracts and titles, C (1,:7,.... Ic Tia)), minimum, and maximum, b, number of abstracts to interleave, and min- imum, m, and maximum, n, number sentences in source, and then returns interleaved texts and summaries. WINDOW takes sequence of texts, X, and returns window iter- ator of size | + 1, where and window size and sliding step respectively. window reuses elements of X, and therefore, enlarges the corpus size. Notations U/ refers to uniform sampling, H array indexing, and REVERSE reversing an array.
of concatenated
Algorithm Interleaving Algorithm
1: procedure INTERLEAVE(C, a, b, m, m,
Č,Z < WINDOW(C, w b,1 1), Array))
3: while C 0 do
4:
C','', ,'' < C.NEXT(), ArrayO. ArrayO. 0
5:
6:
for do
> Selection
7:
A, Cjj
8:
7' (TA)
9 U(m,
A' ADD(A(L:))
11:
S SU (ixa}
12:
A, + ArrayO, Array()
13:
for to [S] do
b Interleaving
14:
k < U(S)
15:
Sik
+ DEVERSE(A'(R)).oOr)
17:
A.ADD(D
18:
T'K]]
19:
i & T then:
20:
T.ADD(J)
21:
ZAAD)
22:
return Z
PARAMETERS
For the word-to-word encoder, the steps limited 20, while the steps in the word-to-word de- coder are limited to 14. The steps in the post-to-post to-post encoder and haread-to-threal decoder depend on the corpus type, e.g., Medium has steps in post-to-post and steps in thread- to-ud-tt-trrad. In seq2seq experiments, the source is flattened, and therefore, the number of steps in the source is limited to 300. We initialized weights, including word embeddings, with a random normal dis- tribution with mean and standard deviation 0. The embedding vectors are of dimension 80. The vocabulary size is limited to 5000 and 15000 for Pubmed and Stack Exchange corpora respectively. All hidden states of the encoder and decoder in the models set to dimension 100. We pad short sequences with a special token, (PAD). We use Adam Kingma & Ba (2014) with an initial learning rate of .0001 and batch size of 64 for training. Texts lowercased and numbers replaced by the special symbol %.
DATA POINTS AND RESULTS
TRAINING: SEQ2SEQ VS HIER2HIER
12

---

Under review as a conference paper at ICLR 2020
Table 10: The left rows contain interleaving of 3 articles with 2 to 5 sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols.
botulinum loxin is effective for treatment...
prospective randomised controlled trial comparing trgoon-spprrng versus Tigone-includiee intradetrusor injection of ababotulinumtxxina for refractory idiopathic detrusor overactivity.
the trigone is generally spared because of the theoretical...
evaluate efficacy and safety of gegone-including ....
wechadone-maintainee injection drug gender-relates differences the incidence of bleeding....
we studied patients with stemi receiving tihrinolyiss..
physicians may be reluctant to treat hev in idus because
rationale and design of randomized controlled trial of directly observed hepatitis treatment delivered in methadone clinics.
oulcomes included moderate or severe bleeding defined
optimal hev management approaches for idus remain
* moderate or severe bleeding was 1.9-fold higher... conducting randomized controlled trial in a net- work...
* comparison of incidence of bleeding and mortality of with st-clevation myocardial infarction treated fibrinolysis.
women
* hiceding remained higher in women even after adjustment
Table 11: Interleaved sentences of 3 articles, and corresponding ground-truth and hier2hier gen- erated summaries. The top 2 sentences that were attended (r) for the generation are the left. Additionally, top words (3) attended for the generation are colored accordingly.
Interleaved Texts
0 botulinum toxin is effective treatment of idiopathic detrusor overactivity ([UNK))
the [UNK] is generally [UNK] because of the theoretical risk of [UNK] reflux ( ([UNK) although studies assessing...
1
most [UNK] injection drug users (idus) have been infected with hepatitis virus (hev )
3
but...
[UNK] differences incidence bleeding and relation subsequent mortality patients with st-segment elevation myocardial infarction...
4
8
optimal hev management approaches for idus remain unknown
GoounddTruth/Geneaane
prospcetive randomised controllcd trial comparing trigone-sparing versus trigonc- including in- tradetrusor injection abobotulinumtoxina for refractory idiopathic detrusor overactivity. efficacy of [UNK] [UNK] in patients with idiopathic detrusor overactivity rationale, design
0,1
rationale and design randomized controlled of directly observed hepatitis e treatment delivered in methadone clinics. validation of a point-of-care hepatitis injection drug injection drug, hev medication, and
3.4
comparison of incidence of bleeding and mortality of men versus women st-elevation myocardial infarction treated with fibrinolysis. subgroup analysis of patients with st-elevation myocardial infarction with st-elevation myocardial infarction.
4.8
13

---

Under review as a conference paper at ICLR 2020
The left rows contain interleaving of 4 articles with 2 to sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols.
Table 12:
the effects of short-course antiretrovirals given to...
v hiv-1 persists in breast milk cells despite antiretroviral treatment prevent mother-to-child transmission.
good adherence is essential for successful antiretroviral...
women in kenya received short-course zidovudine ( zdv )...
breast milk samples were collected two to three times weekly....
the present primary analysis of antiretroviral therapy with... this was an ohservational analysis of an open multicenter...
* patterns of individual and population adherence antiretroviral therapy and risk factors poor adherence in the first year of the dart trial in uganda and zimbabwe.
patients with hiv-I ma at least 5000 copies/ml were...
* 4-weckly clinic visits, art drugs were provided and...
the primary objective was demonstrate non-inferiority...
viral load response assessed subset of patients...
efficacy and safety of once-daily darunavir/ritonavir versus lopinavir/ritonaiir in treatment-naive hiv- I -infectted patients week 48.
we explored the link between serum Appa-fetoprotehi levels...
drug possession ratio percentage of drugs taken between...
low retoprorttoooiin level 5.0 ng/ml randomized...
an...
six hundred and eighty-nine patients
48 weeks, 84 % of drv/r and 78 % of 1pv/r patients...
serum dipha-fetprrotein predicts virologic response hepatitis treatment in hiv coinfected patients.
hiv-I dna was quantified by real-time per
scrum Aipha-fetoprost in measurement should be integrated...
Table 13: Interleaved sentences of 4 articles, and corresponding ground-truth and hier2hier gen- erated summaries. The top 2 sentences that were attended (r) for the generation are the left. Additionally, top words (3) attended for the generation are colored accordingly.
Interleaved Texts
0 the effects of short-course antiretrovirals given to reduce mother-to-child transmission [UNK] on temporal patterns of [UNK] hiv-1 ma
good adherence is essential for successful antiretroviral therapy (art) ) provision, but sim- ple measures have rarely been validated...
2 women in kenya received short-course zidovudine (zdv) ), singleedose nevirapine ( ([UNK));
combination [ÜNK] or short-course...
3 breast milk samples were collected two to three times weekly for 4-6 weeks the present primary analysis of antiretroviral therapy with [UNK] examined in naive subjects (UNK)) compares the efficacy and...
4
10 we explored the link between serum [UNK] levels and virologic response in [UNK] [UNK]
c virus coinfected patients....
GroundTruthh/eeeraatoon
hiv-1 persists in breast milk cells despite antiretroviral treatment to prevent mother-to-chile transmission.
impact of hiv-1 persists on hiy-1 rna in human immunodeficiency virus-infected individ- uals with hiv-1
0,2
patterns of individual and population level adherence to antiretroviral therapy and risk factors for poor adherence in the first year of dart trial uganda and zimbabwe. impact of a antiretroviral treatment algorithm on adherence to antiretroviral therapy [UNK),
1.3
efficacy and safety of once-daily darunavin/ritonavir versus lopinavir/ritonavir in treatment- naive hiv- l-ifeeced patients at week a randomized trial of [UNK] versus [UNK] treatment-naive hiy-1-infected patients with
4,2
hiv-1 infection
serum alpha-fetoprote predicts virologic response hepatitis treatment in hiv coinfected patients.
predicting virologic response in [UNK] coinfected patients coinfected with hiv-1: [UNNK] randomized
10,12
14

---

Under review as a conference paper at ICLR 2020
1.400e+4
1.200e+4
1.000e+4
8.00e+3
0.000
150.0k
300.0k
450.0k
Figure 5: A comparison of running average training loss between seq2seq (pink) and hier2hier (gray) for Stack Exchange Hard corpus.
15


Reviews:
[{"review_id": "HklrJnbpFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a technique for generating summaries of interleaved texts. Unlike previous work that first perform unsupervised clustering to extract ordered threads, the authors instead propose a hierarchical model that directly process interleaved threads. In particular, the authors propose using a hierarchical encoder that encodes words -> post -> threads as well as a hierarchical decoder that decodes threads -> words. On synthetic interleaved datasets composed from PubMed and StackExchange, the proposed method outperform seq2seq with attention as well as a pipeline system that applies unsupervised clustering followed by seq2seq. The authors present results for well-chosen ablations as well as baselines. \n\nMy concerns stem from the synthetic nature of these tasks. In particular:\n\n- is it difficult to disentangle the threads? Suppose we train a simple, supervised model instead of using unsupervised clustering and run the same pipeline experiment, how well would the model perform compared to hier2hier?\n- due to the nature of these two tasks, is disentanglement even helpful for summarization? If not, heir2hier might be picking up on helpful signals that are not accessible to baselines. I don't have a good idea of what the tasks look like from reading the paper. I would ask that the authors put some examples of input output pairs in the paper. The algorithm in the appendix for interleaving is not very helpful.\n- I would like to see some upperbound of running seq2seq on ground-truth disentangled threads.\n\nSome comments on the writing:\n- Figure 4 and Table 2 are way too small", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bye6JV5cYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Strengths> \n+ This paper addresses the problem of summarizing interleaved posts, which involves two subtasks of identifying the clusters of threads and summarizing each thread into a sentence. \n+ This is an understudied area of research in summarization research, only few previous works have been done; for example, (Shang et al. 2018). The proposed approach is end-to-end  trainable, whereas (Shang et al. 2018)’s method tackles the problem in two separate stages. \n\n<Weakness>\n1. The technical novelty may need to be further justified.\n- The proposed model (in Fig.2) consists of the encoder, the decode and hierarchical attention module. \n- The encoder and the decoders are standard ones based on (Nallapati et al. 2017) and (Li et al. 2015), respectively. It is hard to find any technical novelty here. \n- This paper argues that the proposed hierarchical attention is novel with no clear ground. It simply mentions the inspiration from (See et al. 2017), but does not explicitly what aspects are novel as Hierarchical attention has been popularly used in general seq2seq models for many NLP applications. \n- Table 1 is rather cursory and incomplete as it compares with only three different models.\n\n2. Experiments have much room for improvement. \n- The two datasets are simply mixture of text from existing datasets and rather artificial and far from actual interleaved posts, as shown in Table 2. \n- In page 6, three different corpora are introduced with varying interleave parameters. However, the parameters a,b,m, and n are rather small (<= 5), so it is hard to say the configurations are challenging.\n- The SOTA model (Shang et al. 2018) is only compared on the PubMed Easy Corpus in Table 3. Is there any special reason to exclude it from the comparison with more challenging corpora? \n- In most results of experiments, the paper focus on showing that the proposed hierarchical attention model is better than basic seq2seq, which may not be strong results to show the effectiveness of the proposed model.\n\n<Minor comments>\n- This draft does not seem to follow the ICLR format.\n- Fig.2-4 are hard to see in details. They should be enlarged and re-organized. \n- References should be added to Table 1. \n- This paper exceeds the limit of 8 pages. I cannot find any good strong ground to have more additional pages than the commended length. \n\n<Conclusion>\nMy rate is borderline with slightly leaning toward reject, mainly due to issues of technical novelty and evaluation. I will decide my rate finally, based on the authors’ response to the concerns above. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BklB71JS5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an architecture to generate summaries for multi-participant postings as in chat conversations. Previous work tackled this problem as a pipeline of disentangling threads from mixed chat posts and then generating a summary for each post. The presented work proposes a hierarchical attention model to solve this task in an end-to-end fashion. \n\nThe proposed architecture is divided into :\n\n- a hierarchical encoder: encodes each sentence to a vector using bi-lstm and average pooling which is then fed to a post-to-post encoder which generates a set of representations corresponding to those posts.\n\n- a hierarchical decoder: number of disentangled posts are generated using a thread-to-thread decoder which given the max pooled representations from the post-to-post encoder generated a number of initial representations for each thread summary using hierarchical attention combining attention over paragraphs and over words at each decoding time step. \n\nExperiments: \nAuthors demonstrate the effectiveness of their approach by comparing against a set of baselines traditional seq2seq, as well as cluster-> seq2seq and the model from Shang et al. (2018). Evaluation datasets consisted of several datasets for chat summarization synthesized to mimic interleaved conversation with different difficulties. \n\nThrough an ablation study, authors also show the effectiveness of the hierarchical encoding and hierarchical decoding as well as the post level and high-level attention. \n\noverall:\nAlthough this paper is well written and well-motivated. The work is a bit incremental and builds upon previous ideas from hierarchical attention literature to apply for interleaved text generation. The provided baselines are quite weak compared to the SOTA summarization methods at the moment, although none of them is directly modelled for interleaved text summarization through multi summaries. \n\ntypos: \n* Nallapati et al. Nallapati et al. (2016)\n* See et al. See et al. (2017)\n\n\nQuestion: \nIs there an intuition behind penalizing a model that generates correct summaries for each thread with different order? Do references summaries follow the same order as in the threads in the conversation?\n\nIn the paper you dedicated several paragraphs to reimplement basic models such as seq2seq or (See et al.) pointer generator network and make sure they work correctly. There are existing implementations and pre-trained models available such as in OpenNMT. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an end-to-end approach for abstractive summarization of on-line discussions. The approach is contrary to the previous work that first disentangles discussions, and the summarizes them, and aims to tackle transfer of disentanglement errors in the pipeline. The proposed method is a hierarchical encoder - hierarchical decoder architecture. Experimental results on two corpora demonstrate the benefits of the proposed approach. The reviewers are concerned about the synthetic nature of the datasets, limited novelty given the previous work, lack of clear explanation of whether disentanglement is actually needed for summarization, and simpler baselines in comparison to the state-of-the-art. Hence, I recommend rejecting the paper.
Decision: Reject