Text Summarization on the Books of Harry Potter | by Greg Rafferty | Towards Data Science

I’m Greg Rafferty, a data scientist in the Bay Area. You can check out the code for this project on my github. Feel free to contact me with any questions!
In this series of posts, I’m looking at a few handy NLP techniques, through the lens of Harry Potter. Previous posts in this series on basic NLP looked at Topic Modeling with Latent Dirichlet Allocation and Regular Expressions, and my next post will look at sentiment analysis.
Hermione interrupted them. “Aren’t you two ever going to read Hogwarts, A History?”
How many times throughout the Harry Potter series does Hermione bug Harry and Ron to read the enormous tome Hogwarts, A History? Hint: it’s a lot. How many nights do the three of them spend in the library, reading through every book they can find to figure out who Nicolas Flamel is, or how to survive underwater, or preparing for their O.W.L.s? The mistake they’re all making is to try to read everything themselves.
Remember when you were in school and stumbled upon the CliffsNotes summary of that book you never read but were supposed to write an essay about? That’s basically what text summarization does: provide the CliffsNotes version for any large document. Now, CliffsNotes have been written by rather well-educated people who are familiar with the book they’re summarizing. But this is the twenty-first century, aren’t computers supposed to be putting humans out of work? I looked into a few text summarization algorithms to see if we’re ready to put poor old Clifton out of a job.
There are two types of text summarization algorithms: extractive and abstractive. All extractive summarization algorithms attempt to score the phrases or sentences in a document and return only the most highly informative blocks of text. Abstractive text summarization actually creates new text which doesn’t exist in that form in the document. Abstractive summarization is what you might do when explaining a book you read to your friend, and it is much more difficult for a computer to do than extractive summarization. Computers just aren’t that great at the act of creation. To date, there aren’t any abstractive summarization techniques which work suitably well on long documents. The best performing ones merely create a sentence based upon a single paragraph, or cut the length of a sentence in half while maintaining as much information as possible. Often, grammar suffers horribly. They’re usually based upon neural network models. This post will focus on the much more simple extractive text summarization techniques.
Most of the algorithms I’ll present are packaged together in the sumy package for Python but I also use one summarizer from the Gensim package and one other technique I wrote myself using LDA topic keywords to enrich the sumy EdmundsonSummarizer. All examples output five-sentence summaries of the first chapter of Harry Potter and the Sorcerer’s Stone. See my Jupyter notebook for complete code. And a word of caution: don’t judge the results too harshly. They’re not great... (text summarization does seem to work better on drier works of non-fiction)
LexRank is an unsupervised approach that gets its inspiration from the same ideas behind Google’s PageRank algorithm. The authors say it is “based on the concept of eigenvector centrality in a graph representation of sentences”, using “a connectivity matrix based on on intra-sentence cosine similarity.” Ok, so in a nutshell, it finds the relative importance of all words in a document and selects the sentences which contain the most of those high-scoring words.
"The Potters, that's right, that's what I heard —" "— yes, their son, Harry —" Mr. Dursley stopped dead.Twelve times he clicked the Put-Outer, until the only lights left on the whole street were two tiny pinpricks in the distance, which were the eyes of the cat watching him.Dumbledore slipped the Put-Outer back inside his cloak and set off down the street toward number four, where he sat down on the wall next to the cat."But I c-c-can't stand it — Lily an' James dead — an' poor little Harry off ter live with Muggles —" "Yes, yes, it's all very sad, but get a grip on yourself, Hagrid, or we'll be found," Professor McGonagall whispered, patting Hagrid gingerly on the arm as Dumbledore stepped over the low garden wall and walked to the front door.Dumbledore turned and walked back down the street.
One of the first text summarization algorithms was published in 1958 by Hans Peter Luhn, working at IBM research. Luhn’s algorithm is a naive approach based on TF-IDF and looking at the “window size” of non-important words between words of high importance. It also assigns higher weights to sentences occurring near the beginning of a document.
It was now reading the sign that said Privet Drive — no, looking at the sign; cats couldn't read maps or signs.He didn't see the owls swooping past in broad daylight, though people down in the street did; they pointed and gazed open-mouthed as owl after owl sped overhead.No one knows why, or how, but they're saying that when he couldn't kill Harry Potter, Voldemort's power somehow broke — and that's why he's gone.""But I c-c-can't stand it — Lily an' James dead — an' poor little Harry off ter live with Muggles —" "Yes, yes, it's all very sad, but get a grip on yourself, Hagrid, or we'll be found," Professor McGonagall whispered, patting Hagrid gingerly on the arm as Dumbledore stepped over the low garden wall and walked to the front door.G'night, Professor McGonagall — Professor Dumbledore, sir."
Latent Semantic Analysis is a relatively new algorithm which combines term frequency with singular value decomposition.
He dashed back across the road, hurried up to his office, snapped at his secretary not to disturb him, seized his telephone, and had almost finished dialing his home number when he changed his mind.It seemed that Professor McGonagall had reached the point she was most anxious to discuss, the real reason she had been waiting on a cold, hard wall all day, for neither as a cat nor as a woman had she fixed Dumbledore with such a piercing stare as she did now.He looked simply too big to be allowed, and so wild — long tangles of bushy black hair and beard hid most of his face, he had hands the size of trash can lids, and his feet in their leather boots were like baby dolphins.For a full minute the three of them stood and looked at the little bundle; Hagrid's shoulders shook, Professor McGonagall blinked furiously, and the twinkling light that usually shone from Dumbledore's eyes seemed to have gone out.A breeze ruffled the neat hedges of Privet Drive, which lay silent and tidy under the inky sky, the very last place you would expect astonishing things to happen.
TextRank is another text summarizer based on the ideas of PageRank, and was also developed at the same time as LexRank, though by different groups of people. TextRank is a bit more simplistic than LexRank; although both algorithms are very similar, LexRank applies a heuristic post-processing step to remove sentences with highly duplicitous.
Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much.They were the last people you'd expect to be involved in anything strange or mysterious, because they just didn't hold with such nonsense.Mr. Dursley was the director of a firm called Grunnings, which made drills.He was a big, beefy man with hardly any neck, although he did have a very large mustache.Mrs. Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbors.
In 1969, Harold Edmundson developed the summarizer bearing his name. Edmundson’s algorithm was, along with Luhn’s, one of the seminal text summarization techniques. What sets the Edmundson summarizer apart from the others is that it takes into account “bonus words”, words stated by the user as of high importance; “stigma words”, words of low importance or even negative importance; and “stop words”, which are the same as used elsewhere in NLP processing. Edmundson suggested using the words in a document’s title as bonus words. Using the chapter title as bonus words, this is what Edmundson outputs:
The Dursleys shuddered to think what the neighbors would say if the Potters arrived in the street.When Dudley had been put to bed, he went into the living room in time to catch the last report on the evening news: "And finally, bird-watchers everywhere have reported that the nation's owls have been behaving very unusually today.Twelve times he clicked the Put-Outer, until the only lights left on the whole street were two tiny pinpricks in the distance, which were the eyes of the cat watching him.Dumbledore slipped the Put-Outer back inside his cloak and set off down the street toward number four, where he sat down on the wall next to the cat.He couldn't know that at this very moment, people meeting in secret all over the country were holding up their glasses and saying in hushed voices: "To Harry Potter — the boy who lived!"
Another addition I made was to use LDA to extract topic keywords, and then add those topic keywords back in as additional bonus words. With this modification, the Edmundson results are as follows:
At half past eight, Mr. Dursley picked up his briefcase, pecked Mrs. Dursley on the cheek, and tried to kiss Dudley good-bye but missed, because Dudley was now having a tantrum and throwing his cereal at the walls.When Dudley had been put to bed, he went into the living room in time to catch the last report on the evening news: "And finally, bird-watchers everywhere have reported that the nation's owls have been behaving very unusually today.Twelve times he clicked the Put-Outer, until the only lights left on the whole street were two tiny pinpricks in the distance, which were the eyes of the cat watching him.One small hand closed on the letter beside him and he slept on, not knowing he was special, not knowing he was famous, not knowing he would be woken in a few hours' time by Mrs. Dursley's scream as she opened the front door to put out the milk bottles, nor that he would spend the next few weeks being prodded and pinched by his cousin Dudley.He couldn't know that at this very moment, people meeting in secret all over the country were holding up their glasses and saying in hushed voices: "To Harry Potter — the boy who lived!"
The SumBasic algorithm was developed in 2005 and uses only the word probability approach to determine sentence importance. Sorry, but it’s pretty bad on this document.
Mr. Dursley wondered."Harry.The cat was still there."It certainly seems so," said Dumbledore."Yes," said Professor McGonagall.
Wow. That’s terrible. Ahem, moving on...
The KLSum algorithm is a greedy method which adds sentences to the summary as long as the KL Divergence (a measure of entropy) is decreasing.
It was on the corner of the street that he noticed the first sign of something peculiar — a cat reading a map.It grew steadily louder as they looked up and down the street for some sign of a headlight; it swelled to a roar as they both looked up at the sky — and a huge motorcycle fell out of the air and landed on the road in front of them.He looked simply too big to be allowed, and so wild — long tangles of bushy black hair and beard hid most of his face, he had hands the size of trash can lids, and his feet in their leather boots were like baby dolphins."But I c-c-can't stand it — Lily an' James dead — an' poor little Harry off ter live with Muggles —" "Yes, yes, it's all very sad, but get a grip on yourself, Hagrid, or we'll be found," Professor McGonagall whispered, patting Hagrid gingerly on the arm as Dumbledore stepped over the low garden wall and walked to the front door.He clicked it once, and twelve balls of light sped back to their street lamps so that Privet Drive glowed suddenly orange and he could make out a tabby cat slinking around the corner at the other end of the street.
The Reduction algorithm is another graph-based model which values sentences according to the sum of the weights of their edges to other sentences in the document. This weight is computed in the same way as it is in the TexRank model.
Mrs. Potter was Mrs. Dursley's sister, but they hadn't met for several years; in fact, Mrs. Dursley pretended she didn't have a sister, because her sister and her good-for-nothing husband were as unDursleyish as it was possible to be.It seemed that Professor McGonagall had reached the point she was most anxious to discuss, the real reason she had been waiting on a cold, hard wall all day, for neither as a cat nor as a woman had she fixed Dumbledore with such a piercing stare as she did now.Dumbledore took Harry in his arms and turned toward the Dursleys' house."But I c-c-can't stand it — Lily an' James dead — an' poor little Harry off ter live with Muggles —" "Yes, yes, it's all very sad, but get a grip on yourself, Hagrid, or we'll be found," Professor McGonagall whispered, patting Hagrid gingerly on the arm as Dumbledore stepped over the low garden wall and walked to the front door.G'night, Professor McGonagall — Professor Dumbledore, sir."
The Gensim package for Python includes a summarizer which is a modification of the TextRank algorithm. Gensim’s approach modifies the sentence similarity function.
It was now reading the sign that said Privet Drive — no, looking at the sign; cats couldn't read maps or signs.Dumbledore slipped the Put-Outer back inside his cloak and set off down the street toward number four, where he sat down on the wall next to the cat.They're a kind of Muggle sweet I'm rather fond of." "No, thank you," said Professor McGonagall coldly, as though she didn't think this was the moment for lemon drops.All this 'You-Know-Who' nonsense — for eleven years I have been trying to persuade people to call him by his proper name: Voldemort." Professor McGonagall flinched, but Dumbledore, who was unsticking two lemon drops, seemed not to notice.Professor McGonagall shot a sharp look at Dumbledore and said, "The owls are nothing next to the rumors that are flying around.It seemed that Professor McGonagall had reached the point she was most anxious to discuss, the real reason she had been waiting on a cold, hard wall all day, for neither as a cat nor as a woman had she fixed Dumbledore with such a piercing stare as she did now.
So which algorithm do you think provides the best summary? There are a lot of parameters to tweak before making a final judgement. Some might perform better on shorter summaries, some on longer summaries. The style of writing could make a difference (I mentioned before that I’ve had better luck summarizing non-fiction than fiction). But if you’re Harry just entering Hogwarts’ library with the intent to answer some question and are intimidated by “the sheer size of the library; tens of thousands of books; thousands of shelves; hundreds of narrow rows”, then it couldn’t hurt to have a couple summaries available with a few mouse clicks. If only wizards used such silly muggle creations...