Computational Models for abstractive Text Summarization
Leonid Keselman & Ludwig SchubertState of the art in text summarization aresequence-to-sequence models with attention:

Our sum-of-glove model beat our first-sentence baseline onsome datasets, while being worse on others.
NIPS

DUC’04

NIPS

3.652
0.060

3.735
0.074

0.052
00000

3.233
0.165

decode to text sequence

extractive

abstractive

We focus on building an abstractive model that is able to train
faster and scale to larger inputs than traditional sequence-tosequence architectures. We implement a variety of recurrent
decoders, paired with efficient feed-forward and convolutional
encoders.

multi-layer RNN decoder

3.687
00.0870.015

0.802
00.482000with beam search and attention

0.034
01.01700.19600.016* 000

Encoder Architectures
These models are simple & allow for arbitrary length inputs.

0.761
0.095

0.631
00.020000GLOVE embeddingof all tokens

GLOVE embeddingof all tokens

sum

single-layer RNN decoder
no beam search, no attention

conv…

max-poolsingle-layer RNN decoderno beam search, no attention

Decoder Architectures
single-layer RNN decodersummary

Which datasets do you use?
To observe differences in summarization behavior, we trained
and tested our models on four datasets:
DUC 2004—432 news articles with 4 model summaries each
NewsIR ’16—1M online articles; filtered to media type “News”
NIPS—all existing NIPS publications; used abstracts and titles
SQuAD—we flipped this question answering dataset to get arough equivalent of multiple summaries per context paragraph.

DUC’04

4.018
0.237
1.810
01.06402.502
0Example Summary

decoderinternal representation

trained onSQuAD
NewsIR’16

These models aim to increase convergence and ameliorate issues
with generated text, such as repetition.
step-by-step generationand subtraction

single-layer RNN decoder

All datasets were split 80-10-10 for training, evaluation and testing respectively.
Leonid Keselman, LEONIDK & Ludwig Schubert, SCHUBERLGenerated: sword of orion sailor missing
GT: three yachts missing two dead one sailor missing

Transfer task: sentiment classification onSentimentsummaries
Classification
One question we were trying toanswer was whether summarization
could help with other NLP tasks; inour case the answer was “no”—thesimple models did not improve thequality of an LSTM-based sentimentmodel in either case.

trained on
fullsummaries

fulloutput in best order

CS 224n Poster Session | 2017-03-21 | Final Project

first sentence

trained on
SQuAD
NewsIR’16

82.4%

60.1%

summaries

encode hidden state

multi-layerbidirectional encoding

Textsum (SOA architecture; stopped after 180k steps)

tested on

select subset of words

trainable embedding offirst two sentences

tested on

Existing summarization techniques can be classified into two
categories; extractive, and abstractive:

source document

Sum-of-Glove + RNN (F-scores Rouge-1 & -2)

attention over input during decoding

What did you focus on?

encoderBaseline

NIPS

Text summarization is the task of taking a document andcreating a shorter version of it while preserving its meaning.

SQuAD

Results & Discussion

NewsIR’16

Model Architectures

DUC’04

Task & Datasets

75.5%

65.5%

Conclusion & Future Work
Our models produced summaries that were often on topic butwith grammatical issues. On the decoder site we saw promising
results from training on identity datasets.a
We could make additional progress both by using larger
training data sets and training these systems as autoencoders,
where input is expected to match output.
[ludwig, leonidk]@cs.stanford.edu