Leveraging NLP and ML to Consume News Better—and then Making an App out of It
By: Adithya L Narayanan
Tags: NLP, Sentiment Analysis, Twitter, Data, News

Leveraging NLP and ML to Consume News Better—and then Making an App out of It

His aunt and uncle exchanged looks of outrage.

“Listening to the news! Again?”

“Well, it changes every day, you see,” said Harry.

I like reading the news — a lot. I endlessly scroll through Twitter and Google News daily in the quest to read better news. And when I think of it, it is for no particular reason.

One fine day, my digital wellbeing tracker read that I spent over two hours on Twitter. I know what I was doing: hunting for news.

When push came to shove and I realized my need to cut down on spending time on Twitter, I found a workaround to still find my daily dose of news, without losing all of the fun that Twitter brings along with it. I built a news aggregator app!

Beyond just news.

Reading news and watching videos are not standalone events. Context is essential when consuming any content and as far as news is concerned, the opinions/opinionated text it carries and the reaction it evokes are what majorly form such context.

Public Pulse

News is entangled with opinion at all times—be it that of the journalist or of the people consuming it. Understanding the intricacies of these opinions helps in understanding news itself. For instance, a news piece or an opinion piece with a large amount of negative responses to it can mean that it wasn’t well-received. This may warrant additional digging into what it speaks of.

With or without this exact motivation, a lot of us often read about public perception of several things of interest. This, I believe, is a big reason why comments/replies/threads exist, and often publicly, in several popular social media (be it Twitter, Facebook, YouTube, Reddit). I tend to do this especially when I read news articles—to gauge what people think of happenings—but this only adds to the endless scrolling.

To solve this, I decided to leverage NLP to spoon-feed me with the gist. For every news tweet the app shortlists, it collects all the replies to said tweet and calculates the polarity/sentiment of these using NLTK Vader and spaCy-textblob. By judging whether positive or negative replies dominate the thread, we determine the emotion of the reaction to the news article—we judge the polarity of the discourse surrounding it.

Should news be opinionated?

“Anchors having an opinion isn’t a new phenomenon. Murrow had one and that was the end of McCarthy. Cronkite had one, and that was the end of Vietnam.”, Charlie Skinner, The Newsroom.

There are diverse schools of thought on whether a news source should stick to just providing facts or if it should shape the multilogue that accompanies news. I believe in the latter, but I will leave that debate for a different date.

Regardless of which side of this debate you are on, it is always good to know how much of subjectivity news text carries.

While attaining the subjectivity of the news text itself poses some challenges in terms of attaining news text beyond paywalls or without expensive APIs, with some immediacy, we can determine the subjectivity of the tweet from the verified handle of the news source that carries of the article. With a reasonable assumption that the tweet’s text is a fair yet abridged imitation of the content of the news article itself, a subjectivity measure of the tweet can be a good substitute. The app obtains the subjectivity of these news tweets with spaCy-textblob as well.