Title: GENERATING MULTI-SENTENCE ABSTRACTIVE SUM-MARIES OF INTERLEAVED TEXTS

Abstract: In multi-participant postings, as in online chat conversations, several conversations or topic threads may take place concurrently. This leads to difficulties for readers reviewing the postings in not only following discussions but also in quickly identifying their essence. A two-step process, disentanglement of interleaved posts followed by summarization of each thread, addresses the issue, but disentanglement errors are propagated to the summarization step, degrading the overall performance. To address this, we propose an end-to-end trainable encoderdecoder network for summarizing interleaved posts. The interleaved posts are encoded hierarchically, i.e., word-to-word (words in a post) followed by post-to-post (posts in a channel). The decoder also generates summaries hierarchically, threadto-thread (generate thread representations) followed by word-to-word (i.e., generate summary words). Additionally, we propose a hierarchical attention mechanism for interleaved text. Overall, our end-to-end trainable hierarchical framework enhances performance over a sequence to sequence framework by 8-10% on multiple synthetic interleaved texts datasets.

INTRODUCTION:

Interleaved texts are becoming more common with new ways of working and new forms of communication, e.g., multi-author entries for activity reports, meeting minutes and social media conversations, such as Slack. Quickly getting a sense of or following the content of different threads in interleaved texts, where posts belonging to different threads occur in one sequence, is often difficult. An example of two threads with multiple posts interleaved to form a sequence is:

Post1-Thread1 → Post1-Thread2 → Post2-Thread1 → Post3-Thread1 → Post2- Thread2 → Post3-Thread2

This intermingling leads to difficulties in not only following discussions but also in retrieving their essence. In conversation disentanglement, interleaved posts are grouped by the thread; e.g., the previous example could be rearranged as:

Post1-Thread1 → Post2-Thread1 → Post3-Thread1 Post1-Thread2 → Post2-Thread2 → Post3-Thread2

In analyzing interleaved texts, Shang et al. (2018) went a step further and proposed summarization of the interleaved texts. They designed an unsupervised two-step system and evaluated the system on meeting texts. In the first step, a conversation disentanglement component disentangles the texts thread-wise. Then, in the second step, a multi-sentence compression component compresses the thread-wise posts to single sentence summaries. However, this system has a major disadvantage, in that the disentanglement obtained through either supervised Jiang et al. (2018) or unsupervised Wang & Oard (2009) methods propagate its errors to the downstream summarization task, and thus, degrades the overall performance.

We aim to tackle this issue of error propagation through an end-to-end trainable encoder-decoder system that takes a variable length input, e.g., interleaved texts, processes it and generates a variable length output, e.g., a multi-sentence summary; see Figure 1 for an illustration. An end-to-end system eliminates the disentanglement component, and thus, the error propagation. The encoder first performs word-to-word encoding to embed each post, followed by post-to-post encoding to embed the overall content of the posts and represent the discourse structure of the interleaved texts. The decoder has a thread-to-thread decoder to generate a representation for each thread, and the thread representation is given to a word-to-word decoder to generate a summary sentence. We also propose to use hierarchical attention similar to Nallapati et al. (2016) , but instead of computing post-level attention at every word, attentions are only computed when decoding new sentences. Further, the attention networks are trained end-to-end.

Despite the availability of a multitude of real-world interleaved texts, a major challenge to train encoder-decoder models is the lack of labels (summaries). As labeling is difficult and expensive Barker et al. (2016) ; Aker et al. (2016) ; Verberne et al. (2018) , we synthesize two separate corpora. Each is derived by mixing texts and associated summaries from a corpus of documents, where the mixed text has a structure reflective of a multi-party conversation with inter-leaved threads and the summary highlights the information in the threads. We find abstracts and titles of randomized controlled trial (RCT) articles, a PubMed corpus, and also for Stack Exchange posts and titles.

To summarize, our contributions are threefold:

• We propose to combine a hierarchical encoder and decoder to obtain multi-sentence summaries of interleaved texts.

• We propose a novel hierarchical attention mechanism that is integrated with the hierarchical encoder-decoder architecture and which equips the decoder to disentangle the interleaving further.

• We use two synthetic datasets to verify the ideas and show our end-to-end trainable architecture addresses not only the issue of error-propagation observed in competitive methods but also improves the performance.

RELATED WORK:

Quite often multi-party conversations, e.g. news comments, social media conversation and activity report, have tens of posts discussing several different matters pertaining to a subject. Ma et al. (2012) ; Aker et al. (2016) ; Shang et al. (2018) designed methodologies to summarize posts in order to provide an overview on the discussed matters. They broadly follow the same approach: cluster the posts and then extract a summary from each cluster.

There are two kinds of summarization: abstractive and extractive. In abstractive summarization, the model utilizes a corpus level vocabulary and generates novel sentences as the summary, while extractive models extract or rearrange the source words as the summary. Abstractive models based on neural sequence-to-sequence (seq2seq) (Rush et al. (2015) ) proved to generate summaries with higher ROUGE scores than the feature-based abstractive models. Integration of attention into seq2seq (Bahdanau et al. (2014) ) led to further advancement of abstractive summarization (Nallapati et al. (2016) ; Chopra et al. (2016) ).

There are many possible patterns of organization of the information in texts, e.g., chronological pattern. News articles have an inverted pyramid pattern, i.e., core information in the lead sentences and the extra information in later sentences. A seq2seq model is appropriate for summarization of a news article as it encodes and decodes sequentially. However, in interleaved texts, related information maybe separated; thus a seq2seq model may be competent but not sufficient. Li et al. (2015) proposed an encoder-decoder (auto-encoder) model that utilizes a hierarchy of networks: word-to-word followed by sentence-to-sentence. Their model is better at capturing the underlying structure than a vanilla sequential encoder-decoder model (seq2seq). Krause et al. (2017) ; Jing et al. (2018) showed multi-sentence captioning of an image through hierarchical Recurrent Neural Network (RNN), topic-to-topic followed by word-to-word, is better than seq2seq.

These works suggest a hierarchical encoder, with word-to-word encoding followed by post-to-post, will better recognize the dispersed information in interleaved texts. Similarly, a hierarchical decoder, thread-to-thread followed by word-to-word, will intrinsically disentangle the posts, and therefore, generate more appropriate summaries.

Nallapati et al. ( 2016) devised a hierarchical attention mechanism for a seq2seq model, where two levels of attention distributions over the source, i.e., sentence and word, are computed at every step of the word decoding. Based on the sentence attentions, the word attentions are rescaled. Hsu et al. (2018) slightly simplified this mechanism and computed the sentence attention only at the first step.

Our hierarchical attention is more intuitive and computes new sentence attentions for every new summary sentence, and unlike Hsu et al. (2018) , is trained end-to-end .

MODEL:

Figure 2 : Hierarchical encoder-decoder architecture.

PROBLEM STATEMENT:

We aim to design a system that when given a sequence of posts, C = P 1 , . . . , P |C | , produces a sequence of summaries, T = S 1 , . . . , S |T | . For simplicity and clarity, unless otherwise noted, we will use lowercase italics for variables, uppercase italics for sequences, lowercase bold for vectors and uppercase bold for matrices.

Figure 2 illustrates the proposed hierarchical encoder-decoder framework. In the framework, first, a low-level, word-to-word encoder converts a sequence of words in a post, P j , to a sequence of representations, H j = h w2w j0 , . . . , h w2w j|Pj | . Subsequently, a top-level, post-to-post encoder converts those representations, H 0 , . . . , H |C | , to a sequence of top-level post representations h p2p 1 , . . . , h p2p |C | . These encoded representations are then passed to the decoder, which utilizes a top-level, threadto-thread, decoder to disentangle them into a sequence of thread representations h t2t 1 , . . . , h t2t |T | . Finally, a low-level, word-to-word, decoder takes a feed-forward mapped thread representation h t2t i and generates a sequence of summary words w i1 , . . . , w i|Si| .

The maximum number of posts in a sequence of interleaved texts is denoted by n and threads by m. We limit the number of words in posts and summaries to fixed lengths by either truncating or ) corresponding to those posts. Overall, output word-to-word representations, W, has dimension n × p × 2l and post-to-post representations, P, has dimension n × 2l for a given channel, C .

DECODER:

Our hierarchical decoder (see Figure 2 ) is based on Li et al. (2015) , where both thread-to-thread and word-to-word decoders are uni-directional LSTMs.

The initial state h Dt2t 0 of the thread-to-thread LSTM decoder (f Dt2t ) is set with a feedforwardmapped representation of an average pooled post representations (c = 1 n n k=0 h p2p k ). At each step i of the decoder, a sequence of attention weights, βi,0 , . . . , βi,n×p , corresponding to the set of encoded word representations, h w2w 0 , . . . , h w2w n×p are computed utilizing the previous state, h Dt2t i-1 . We will elaborate the attention computation in the next section.

A weighted representation of the words (crossed blue circle) is then computed: n j=1 βij W ij , and used as one input to LSTM f Dt2t . Additionally, we use the last hidden state h Dw2w i-1• of the wordto-word decoder LSTM (D w2w ) of the previously generated summary sentence as the second input to compute the next state of thread-to-thread decoder, i.e., h Dt2t i . The motivation is to provide information about the previous sentence.

The current state h Dt2t i is passed through a single layer feedforward network and a distribution over STOP=1 and CONTINUE=0 is computed:

p ST OP i = σ(g h Dt2t i ) (1)

where g is a feedforward network. In Figure 3 , the process is depicted by a red circle. The threadto-thread decoder keeps decoding until p ST OP i is greater than 0.5.

Additionally, the current state h Dt2t i is passed through another single layer feedforward network k followed by tanh activation to compute the thread representation

s i = tanh(k h Dt2t i ).

Given a thread representation s i , the word-to-word decoder generates a summary for the thread. Our word-to-word decoder is based on Bahdanau et al. (2014) . It is a unidirectional attentional LSTM (f Dw2w ); see the right-hand side of Figure 2 . We refer to Bahdanau et al. (2014) for further details.

HIERARCHICAL ATTENTION:

Our novel dynamic hierarchical attention is based on two popular concepts: attention scores should integrate in top-down manner (Nallapati et al. (2016) ) and two distributions over same words must be modeled together (See et al. (2017) ), not just in isolation.

Our hierarchical attention works at 3 levels, the post level (corresponding to posts), i.e., γ, and higher level (corresponding to source tokens), i.e., β, are computed while obtaining a thread representation, s. The lower level attention (also corresponding to source tokens), i.e., α, is computed while generating a word, y, of a summary, S .; see yellow rectangles in Figure 4 .

At step i of thread decoding, we also compute elements of post-level attention, i.e., γ i• as below.

EQUATION

At the same step, we also compute elements of high-level attention, i.e, β i• as below.

β ik = σ(attn β (h Dt2t i-1 , a k )) k ∈ {1, . . . , n × p} (3) a k = add(W jl , P j )

where k := j * l and j ∈ {1, . . . , n}, l ∈ {1, . . . , p} (4) where a k is computed as in Eq. 4. W in Eq. 4 is a word-to-word encoder representations of dimension n × p × 2l, add aligns a post representation to its constituting word representations and does element-wise addition, and attn β is a feedforward network that aligns the current thread decoder state h Dt2t i-1 with all n * p number of a k vectors. Then, we use γ to rescale high-level attention, β as below.

βik = β ik * γ ij where k := j * l and l ∈ {1, . . . , p} j ∈ {1, . . . , n} (5) At step t of word-to-word decoding of summary thread i, we compute elements of low level attention, i.e., α it• as below.

α itk = exp(e itk ) n•p k=1 exp(e itk )

where

EQUATION

, and a k is same as in Eq. 4 and attn α is a feedforward network that aligns the current word decoder state h Dw2w t-1 with all n * p number of a k vectors.

Finally, we use rescaled high-level word attentions, β, for rescaling low level attention, α as below: 2014), but with an additional weighted sum of sigmoid cross-entropy loss on stopping distribution; see Eq. 1. Given a summary, Y i = w i0 , . . . , w iq , our word-to-word decoder generates a target Ŷ = y i0 , . . . , y iq , with words from a same vocabulary U . We train our model end-to-end by minimizing the objective given in Eq. 8.

EQUATION

EQUATION

Table 2 : The left rows contain interleaving of 3 articles with 2 to 5 sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols.

this study was conducted to evaluate the influence of e. . . caffeine in sport . influence of endurance exercise on the urinary caffeine concentration .

! to assess the effect of a program of supervised fitness. . . ! an 8-week randomized , controlled trial .. . . nine endurance-trained athletes participated in a randomised. . . . . . ! supervised fitness walking in patients with osteoarthritis of the knee . a randomized , controlled trial .

, we examined the effects of intensity of training on ratings. . . , subjects were recruited as sedentary controls or were randomly. . . , the at lt group trained at velocity lt and the greater than. . . , the effect of training intensity on ratings of perceived exertion . data were obtained on 47 of 51 intervention patients and 45. . .

DATASET:

Obtaining labeled training data for conversation summarization is challenging. The available ones are either extractive (Verberne et al. (2018) ) or too small (Barker et al. (2016) ; Anguera et al. (2012) ) to train a neural model. To get around this issue and thoroughly verify the proposed architecture, we synthesized a dataset by utilizing a corpus of conventional texts for which summaries are available. We create two corpora of interleaved texts: one from the abstracts and titles of articles from the PubMed corpusand one from the questions and titles of Stack Exchange questions. A random interleaving of sentences from a few PubMed abstracts or Stack Exchange questions roughly resembles interleaved texts, and correspondingly interleaving of titles resembles its multi-sentence summary.

The algorithm that we devised for creating synthetic interleaved texts is defined in detail in the Appendix. The number of abstracts to include in the interleaved texts is given as a range (from a to b) and the number of sentences per abstract to include is given as a second range (from m to n). We vary the parameters as below and create three different corpora for experiments: Easy (a=2, b=2, m=5 and n=5), Medium (a=2, b=3, m=2 and n=5) and Hard (a=2, b=5, m=2 and n=5). Table 2 shows an example of a data instance in the Hard Interleaved RCT corpus.

EXPERIMENTS:

we report ROUGE-1, ROUGE-2, and ROUGE-L as the quantitative evaluation of the models. The Pubmed Medium and Hard corpora train, test and validation have approximately 290k, 6k and 1.5k instances, respectively. The Stack-Exchange Medium corpus train has 180k and Hard has 210k instances, while the both corpora have test and validation of approximately 5k and 4k instances, respectively. The remaining hyper-parameters are described in detail in the Appendix.

BASELINES:

We first trained and tested it on the Easy corpus. We also ran Shang et al. (2018) 's unsupervised two-step system on the test set of the Easy Interleaved Pubmed corpus. Additionally, we also utilized Shang et al. (2018) 's clustering component to first cluster the interleaved texts of the corpus, and then the disentangled corpus is used to train the seq2seq model. We refer to the latter as cluster→seq2seq. The performance comparison of Shang et al. (2018) and the two seq2seq models are shown in Table 3. Clearly, seq2seq performs better than Shang et al. (2018) , the reason being a seq2seq model trained on a sufficiently large dataset is better at summarization than the unsupervised sentence com- pression (extractive) method. The lower performance of cluster→seq2seq in comparison to seq2seq shows that a disentanglement component is unnecessary in easy scenario.Furthermore, we utilize the ground-truth labels and disentangle the interleaved posts of a sampled (150k) Hard Corpus, and re-run the seq2seq model. The results (see Table 4 ) show seq2seq (row 3) takes advantage of cluster labels but hier2hier (row 4) does more than it.

SEQ2SEQ VS. HIER2HIER MODELS:

We then compare the proposed hierarchical approach against the seq-to-seq approach in summarizing the interleaved texts by experimenting on the Medium and Hard corpora.

Table 5 shows the experimental results on several corpora; clearly, an increase in the complexity of interleaving scantly impacts the performances. Though, in case of Pubmed, seq2seq depreciates by 1-2 points while hier2hier remains nearly same with increase of complexity. In Stack Exchange corpora, as Hard corpora has more training data (+30k) seq2seq improves slightly. However, a noticeable improvement is observed on changing the decoder to hierarchical, i.e., 1.5-3 Rouge points in Pubmed and 2-4.5 points in Stack Exchange depending on the interleaving complexity. To understand the impact of hierarchy on the hier2hier model, we perform an ablation study and use the Hard Pubmed corpus for the experiments, and Table 6 shows the results. Clearly, adding hierarchical decoding already provides a boost in the performance. Hierarchical encoding also adds some improvements to the performance; however, the enhancement attained in training and inference speed by the hierarchical encoding is much more valuable. We will discuss it in depth later.

Further, we also compiled corpora to mimic real world conversation interleaving, wherein the sequence of summaries for interleaved posts may not follow the sequential occurrence of posts, see 17-18 in Algorithm. 1). An example, a post mention an action in the very beginning of conversation, but does only elaborates it at the end. Therefore, summary corresponding to the action should be at the end. So, we again create Medium and Hard real world mimicking corpora of the Stack Exchange dataset as it is more similar to real world conversations, and perform the same set of experiments.

As seen in Table 7 , the results show that both the seq2seq and hier2hier models performance are slightly lower as it is tougher task; see Table 5 . In addition, the hier2hier model is still consistently better than the seq2seq model.

Importantly, hier2hier converges much earlier than the seq2seq and also reaches a lower training loss (Figure 5 in Appendix). hier2hier takes ≈1.5 days for training on a Tesla V100 GPU, while seq2seq takes ≈4.5 days. Therefore, the hier2hier model not only achieves greater accuracy but also reduces training and inference time. 

The contribution of the post-level and high-level attentions in a hierarchical decoder is two-fold: computing the thread representation and rescaling the word-level attentions. To understand the impact of hierarchical attention on the hier2hier model, we perform an ablation study of post-level attentions (γ) and high-level attentions (β), using the Hard corpus for the experiments. Table 8 shows the performance comparison. Clearly, γ attention improves the performance (0.5-1) of hierarchical decoding but not a lot. The high-level attention, i.e., β is very important as without it the model performances is noticeably reduced (Rouge values decrease from 2-3). Additionally, we also include Li, Luong and Jurafsky, 2015 post-level attention technique in the comparison, as it is the closest hierarchical attention to ours (see Table 1 ). Though, (Li, Luong and Jurafsky) utilize a softmax obtained β to compute thread representation, and in contrast to us, they do not reuse β for re-scaling α, and thereby, closest to hier2hier(+γ -β). Evidently, re-utilization gives hier2hier(+γ -β) an edge over (Li, Luong and Jurafsky, 2015) . Lastly, removing both the γ and β) makes the hier2hier similar to seq2seq, except a few more parameters, i.e., two additional LSTM, and the performance is also very similar.

DISCUSSION:

Table 9 shows an output of our hierarchical abstractive system, in which interleaved texts are in the top, and ground-truth and generated summaries in the bottom. Table 9 also shows the top two post indexes attended to by the post-level attention (γ) while generating those summaries, and they coincide with relevant posts. Similarly, the top 10 indexes (words) of the high-level attention (β)

Table 9 : Interleaved sentences of 3 articles, and corresponding ground-truth and hier2hier generated summaries. The top 2 sentences that were attended (γ) for the generation are on the left. Additionally, top words (β) attended for the generation are colored accordingly.

Interleaved Texts 0 this study was conducted to evaluate the influence of excessive sweating during long-distance running on the urinary concentration of caffeine. . . is directly visualized in the table through the color coding matching the generation. The system not only manages to disentangle the interleaved texts but also to generate appropriate abstractive summaries. Meanwhile, β provides explainability of the output.

As interleaving in the table includes abstracts from the same category, e.g. Physical Activities, the interleaving is complex and approximates the real-world conversations. Despite that, the end-toend hierarchical system tackles the task to a large extent. The next, future stage in this research is transfer learning of the hierarchical system to more types of conversations. We find models trained for 50k iteration on Pubmed Hard corpora (a=6, b=10, m=2 and n=3) and then fine-tuned on the popular meeting AMI corpus (McCowan et al. (2005) ) already reaches competitive results (Rouge-1=39.81, Rouge-2=11.35) against SOA system on AMI (Shang et al. (2018) Rouge-1=37.86 and Rouge-2=07.84). Additionally, we aim to modify hier2hier to include some of the recent additions of seq2seq models, e.g., See et al. (2017) pointer mechanism.

CONCLUSION:

We presented an end-to-end trainable hierarchical encoder-decoder architecture which implicitly disentangles interleaved texts and generates a multi-sentence abstractive summary covering the text threads. Furthermore, the architecture addresses the error propagation issue that occurs in the twostep architectures. Our proposed novel hierarchical attention further boosts both disentanglement and summary generation.

Table 10 : The left rows contain interleaving of 3 articles with 2 to 5 sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols. botulinum toxin a is effective for treatment. . . prospective randomised controlled trial comparing trigone-sparing versus trigone-including intradetrusor injection of abobotulinumtoxina for refractory idiopathic detrusor overactivity. the trigone is generally spared because of the theoretical. . . evaluate efficacy and safety of trigone-including .. . . ! most methadone-maintained injection drug users . . . , gender-related differences in the incidence of bleeding. . . , we studied patients with stemi receiving fibrinolysis. . . ! physicians may be reluctant to treat hcv in idus because . . . ! rationale and design of a randomized controlled trial of directly observed hepatitis c treatment delivered in methadone clinics.

, outcomes included moderate or severe bleeding defined . . . ! optimal hcv management approaches for idus remain . . . , moderate or severe bleeding was 1.9-fold higher . . . ! we are conducting a randomized controlled trial in a network. . .

, comparison of incidence of bleeding and mortality of men versus women with st-elevation myocardial infarction treated with fibrinolysis.

, bleeding remained higher in women even after adjustment . . . rationale and design of a randomized controlled trial of directly observed hepatitis c treatment delivered in methadone clinics. 3,4 validation of a point-of-care hepatitis injection drug injection drug , hcv medication , and comparison of incidence of bleeding and mortality of men versus women with st-elevation myocardial infarction treated with fibrinolysis .

4,8:

subgroup analysis of patients with st-elevation myocardial infarction with st-elevation myocardial infarction .

Table 12 : The left rows contain interleaving of 4 articles with 2 to 5 sentences and the right rows contain their interleaved titles. Associated sentences and titles are depicted by similar symbols.

the effects of short-course antiretrovirals given to. . . hiv-1 persists in breast milk cells despite antiretroviral treatment to prevent mother-to-child transmission.

, good adherence is essential for successful antiretroviral. . . women in kenya received short-course zidovudine ( zdv ). . . breast milk samples were collected two to three times weekly.. . . ! the present primary analysis of antiretroviral therapy with. . . , patterns of individual and population-level adherence to antiretroviral therapy and risk factors for poor adherence in the first year of the dart trial in uganda and zimbabwe.

, this was an observational analysis of an open multicenter. . . ! patients with hiv-1 rna at least 5000 copies/ml were. . . , at 4-weekly clinic visits , art drugs were provided and . . . ! the primary objective was to demonstrate non-inferiority. . . , viral load response was assessed in a subset of patients. . . ! efficacy and safety of once-daily darunavir/ritonavir versus lopinavir/ritonavir in treatment-naive hiv-1-infected patients at week 48.

p we explored the link between serum alpha-fetoprotein levels. . . , drug possession ratio ( percentage of drugs taken between. . . p a low alpha-fetoprotein level ( < 5.0 ng/ml ) was an. . . ! six hundred and eighty-nine patients were randomized. . . ! at 48 weeks , 84 % of drv/r and 78 % of lpv/r patients. . . p serum alpha-fetoprotein predicts virologic response to hepatitis c treatment in hiv coinfected patients.

hiv-1 dna was quantified by real-time pcr .. . . p serum alpha-fetoprote in measurement should be integrated . . . Table 13 : Interleaved sentences of 4 articles, and corresponding ground-truth and hier2hier generated summaries. The top 2 sentences that were attended (γ) for the generation are on the left. Additionally, top words (β) attended for the generation are colored accordingly. 

:



Figures and Tables:

Figure 1: 7 interleaved posts are implicitly disentangled into 3 threads, and single sentence summaries are generated for each thread. Posts are outlined with colors corresponding the color of each summary.

Figure 3: A step in the thread-to-thread decoder.Figure 4: Hierarchical attention mechanism for a hierarchical encoder.

Figure 3: A step in the thread-to-thread decoder.Figure 4: Hierarchical attention mechanism for a hierarchical encoder.

to assess the effect of a program of supervised fitness walking and patient education on functional status , pain , and. . . 102 patients with a documented diagnosis of primary osteoarthritis of one or both knees participated. . . 6 we examined the effects of intensity of training on ratings of perceived exertion (. . . . . . . . . GroundTruth/Generation caffeine in sport . influence of endurance exercise on the urinary caffeine concentration . 0,2 effect of excessive [UNK] during [UNK] running on the urinary concentration of caffeine . supervised fitness walking in patients with osteoarthritis of the knee . a randomized , controlled trial . 1,4 effect of a physical fitness walking on functional status , pain , and pain the effect of training intensity on ratings of perceived exertion . 6,8 effects of intensity of training on perceived [UNK] in [UNK] athletes .

of short-course antiretrovirals given to reduce mother-to-child transmission ( [UNK] ) on temporal patterns of [UNK] hiv-1 rna 1 good adherence is essential for successful antiretroviral therapy ( art ) provision , but simple measures have rarely been validated. . . 2 women in kenya received short-course zidovudine ( zdv ) , single-dose nevirapine ( [UNK] ) , combination [UNK] or short-course. . . 3 breast milk samples were collected two to three times weekly for 4-6 weeks .. . . 4 the present primary analysis of antiretroviral therapy with [UNK] examined in naive subjects ( [UNK] ) compares the efficacy and. . . . . . . . . 10 we explored the link between serum [UNK] levels and virologic response in [UNK] [UNK] c virus coinfected patients .. . . . . . . . . GroundTruth/Generation hiv-1 persists in breast milk cells despite antiretroviral treatment to prevent mother-to-child transmission . 0,2 impact of hiv-1 persists on hiv-1 rna in human immunodeficiency virus-infected individuals with hiv-1 patterns of individual and population-level adherence to antiretroviral therapy and risk factors for poor adherence in the first year of the dart trial in uganda and zimbabwe . 1,3 impact of a antiretroviral treatment algorithm on adherence to antiretroviral therapy in [UNK] , efficacy and safety of once-daily darunavir/ritonavir versus lopinavir/ritonavir in treatmentnaive hiv-1-infected patients at week 48 . 4,2 a randomized trial of [UNK] versus [UNK] in treatment-naive hiv-1-infected patients with hiv-1 infection serum alpha-fetoprotein predicts virologic response to hepatitis c treatment in hiv coinfected patients . 10,12 predicting virologic response in [UNK] coinfected patients coinfected with hiv-1 : a [UNK] randomized

Figure 5: A comparison of running average training loss between seq2seq (pink) and hier2hier (gray) for Stack Exchange Hard corpus.



Table content: <table/>

Forms of hierarchical attention. End2End=No training labels, ×words=for every word in the target, ×summaries=for every summary in the target, γ=sentence(post)-level, β=high-level

Table content: <table><tr><td>Model</td></tr></table>

Rouge F1-Scores for seq2seq models on the Pubmed Easy Corpus.

Table content: <table><tr><td/><td/><td/><td/><td colspan="4">Table 4: Recalls on a sampled Pubmed Hard-</td></tr><tr><td/><td/><td/><td/><td colspan="4">Corpus. Int=Interleaved, Dis=Disentangled.</td></tr><tr><td>Model</td><td colspan="3">Rouge-1 Rouge-2 Rouge-L</td><td>Model</td><td colspan="3">type Rouge-1 Rouge-2 Rouge-L</td></tr><tr><td>Shang et al. (2018)</td><td>30.37</td><td>10.77</td><td>20.04</td><td>seq2seq</td><td>Int 32.40</td><td>11.09</td><td>22.97</td></tr><tr><td>seq2seq</td><td>44.38</td><td>19.47</td><td>35.20</td><td>hier2hier</td><td>Int 33.51</td><td>12.50</td><td>24.27</td></tr><tr><td>cluster→seq2seq</td><td>42.93</td><td>18.76</td><td>30.68</td><td>seq2seq</td><td>Dis 37.01</td><td>14.44</td><td>26.91</td></tr><tr><td/><td/><td/><td/><td>hier2hier</td><td>Dis 37.68</td><td>15.11</td><td>28.25</td></tr></table>

Rouge Recall-Scores of models on the Medium and Hard Corpus.

Table content: <table><tr><td>Corpus Difficulty</td><td>Model</td><td colspan="6">Pubmed Rouge-1 Rouge-2 Rouge-L Rouge-1 Rouge-2 Rouge-L Stack Exchange</td></tr><tr><td>Medium</td><td>seq2seq hier2hier</td><td>30.67 32.78</td><td>11.71 12.36</td><td>23.80 25.33</td><td>18.78 24.34</td><td>03.52 05.07</td><td>14.73 18.63</td></tr><tr><td>Hard</td><td>seq2seq hier2hier</td><td>29.07 33.36</td><td>10.96 12.69</td><td>21.76 24.72</td><td>20.21 24.96</td><td>04.03 05.56</td><td>14.93 17.95</td></tr></table>

Rouge Recall-Scores of models on the Pubmed Hard Corpus.

Table content: <table><tr><td/><td colspan="3">Pubmed Hard Corpus</td></tr><tr><td>Model</td><td colspan="3">Rouge-1 Rouge-2 Rouge-L</td></tr><tr><td>seq2seq</td><td>29.07</td><td>10.96</td><td>21.76</td></tr><tr><td>seq2hier</td><td>32.92</td><td>11.87</td><td>24.43</td></tr><tr><td>hier2seq</td><td>31.86</td><td>11.9</td><td>23.57</td></tr><tr><td>hier2hier</td><td>33.36</td><td>12.69</td><td>24.72</td></tr></table>

Rouge Recall-Scores of models on the Stack Exchange Medium and Hard Corpus.

Table content: <table><tr><td/><td colspan="3">Medium Corpus</td></tr><tr><td>Model</td><td colspan="3">Rouge-1 Rouge-2 Rouge-L</td></tr><tr><td>seq2seq</td><td>19.67</td><td>03.88</td><td>15.37</td></tr><tr><td>hier2hier</td><td>23.97</td><td>05.63</td><td>18.75</td></tr><tr><td/><td colspan="3">Hard Corpus</td></tr><tr><td>seq2seq</td><td>19.62</td><td>03.71</td><td>14.90</td></tr><tr><td>hier2hier</td><td>24.14</td><td>05.00</td><td>17.25</td></tr></table>

Rouge Recall-Scores of ablated models on the Hard Pubmed Corpus.

Table content: <table><tr><td>Model</td><td colspan="3">Rouge-1 Rouge-2 Rouge-L</td></tr><tr><td>hier2hier(+γ + β)</td><td>33.36</td><td>12.69</td><td>24.72</td></tr><tr><td>hier2hier(-γ + β)</td><td>32.65</td><td>12.21</td><td>24.23</td></tr><tr><td>hier2hier(+γ -β)</td><td>31.28</td><td>10.20</td><td>23.49</td></tr><tr><td>hier2hier(Li,Luong&amp;Jurafsky)</td><td>29.83</td><td>09.80</td><td>22.17</td></tr><tr><td>hier2hier(-γ -β)</td><td>30.58</td><td>10.00</td><td>22.96</td></tr><tr><td>seq2seq</td><td>29.07</td><td>10.96</td><td>21.76</td></tr></table>

Interleaved sentences of 3 articles, and corresponding ground-truth and hier2hier generated summaries. The top 2 sentences that were attended (γ) for the generation are on the left. Additionally, top words (β) attended for the generation are colored accordingly.

Table content: <table><tr><td>Interleaved Texts</td></tr></table>

Peer review (iclr2020):
This paper proposes a technique for generating summaries of interleaved texts. Unlike previous work that first perform unsupervised clustering to extract ordered threads, the authors instead propose a hierarchical model that directly process interleaved threads. In particular, the authors propose using a hierarchical encoder that encodes words -> post -> threads as well as a hierarchical decoder that decodes threads -> words. On synthetic interleaved datasets composed from PubMed and StackExchange, the proposed method outperform seq2seq with attention as well as a pipeline system that applies unsupervised clustering followed by seq2seq. The authors present results for well-chosen ablations as well as baselines. My concerns stem from the synthetic nature of these tasks. In particular: - is it difficult to disentangle the threads? Suppose we train a simple, supervised model instead of using unsupervised clustering and run the same pipeline experiment, how well would the model perform compared to hier2hier? - due to the nature of these two tasks, is disentanglement even helpful for summarization? If not, heir2hier might be picking up on helpful signals that are not accessible to baselines. I don*t have a good idea of what the tasks look like from reading the paper. I would ask that the authors put some examples of input output pairs in the paper. The algorithm in the appendix for interleaving is not very helpful. - I would like to see some upperbound of running seq2seq on ground-truth disentangled threads. Some comments on the writing: - Figure 4 and Table 2 are way too small