Digital Tools for Looking at Texts
By: Scalable Analysis
Tags: Report, NLP, Knowledge Graph, Insights, Data Science

There has been a profusion of tools to enable scholars, readers and analysts to look at texts. I’m not going to go through these tools per se, but rather to demonstrate how it might be done.

I’m an analyst — and that means, I look at reports, and try to glean from them what they are trying to say. In well-constructed reports, that is easy, but as they become long, then things get really difficult. Thankfully there are a whole bunch of tools that are now becoming available that allow for the extraction of insights.

The tools are possible owing to the advancements in natural language processing — the use of computational techniques and models to analyse natural language — language as it is used around us — in the documents, in voice, in chats. The explosion of content generated, and especially Wikipedia — has made various advancements possible. Thanks to Wikipedia, which contains topics arranged in a structured manner, and thanks to the effort put into translation by Google and others, it is now increasingly possible for machines to read texts in a structured way. Such tools are of great help to analysts — they can also aid in helping analysts see what the main points in a document are, and how they might be related to each other. Across numerous texts (think of new articles), analysts could paint a picture of the issue evolving over time.

Such objectives are the intention behind my explorations in NLP. As an analyst looking at emerging trends, I look at news and articles, and I wanted to be able to note the changes over time. The collections process aside, I also had to stitch together various tools. The aim was to be able to build a knowledge graph based on a long text. Why a knowledge graph? A knowledge graph would allow for the analyst to look at the main entities involved, and how they might be involved on an issue. Graphing techniques provide a quick view into the relationships that are involved. Although the findings for this particular exercise are underwhelming, they do nonetheless show enough promise for further refinements.

So here it is. The documents I chose were from recent McKinsey Global Institute publications on climate risks and the Bio Revolution. That these articles come from MGI should raise eyebrows — I would not ordinarily associate MGI or McK as a whole on these topics; I had initially associated McK with the usual efficient capitalist model — how to exploit tools and techniques for maximum profit. But I guess it’s a sign of the times when even McK (albeit the think-tank part) has to pay attention to these issues. But that aside…

The underlying engine powering the whole exercise is Spacy — the natural language processing library for Python. Other tools would be the good ‘ol NLTK (natural language tool kit, an earlier library that provided foundations for other work). I will be using NetworkX, the graphing library, and pyLDAvis, the visualization package for topic models (I’m using package and libraries interchangeably).

This article will focus on the results from the Climate Risks paper, since I had already looked at the Bio Revolutions one before (to demonstrate summary, fuller paper to follow someday…).

Before doing all that, one had to first take the pdf and turn it into a txt file. Sounds easy, but actually non-trivial. After some false-starts, I landed on pdftotext, that actually worked (module import issues, etc.).

Most of the code was taken from this piece on Towards Data Science (thanks Chris Thornton, for the code). The code for this exercise is on the Git — it’s the full notebook, so it’s a big large. I had also tried this (shout out to Prateek Joshi), but the Thornton code seemed more helpful — but it’s a matter of preference).

After converting the pdf to text using pdftotext (and making sure the result is a string, not a pdftotext object), run it through Spacy, and extract the entities. I used the Entity Pair code from Thornton, and I identify the various geopolitical entities and organisations.

I also looked at the topic models:

Now, as I mentioned, the knowledge graphs were… underwhelming, and this was what I meant — this is for a chart for how organisations might be related in the document.

It turns out that just a simple table can be quite useful:

With this table, I can tell at a glance what’s attributed to the term, ‘climate’ that I can interpret and further understand. Or with this:

The table for the network graph you saw previously. I also try a different way of representing the tabular data, using Circos plots.

Table for GeoPolitical Entities (“GPE”)

Circos Plot for the GPE table

Nonetheless, I feel that these are really promising directions — and I hope with further practice, tweaks and refinements, I can get to the point when the knowledge graphs are more usable.