Analysis of Duterte’s Speeches using NLP | by Daniel Del Rio | Towards Data Science

Given the present situation of the COVID-19 across different countries, the urgency for their governments to step up and address the pandemic has elevated even further. The story isn’t any different here in the Philippines, as the frequency of President Rodrigo Duterte’s press conferences addressed to the Filipino people has risen. This inspired me to work on a personal project that will analyze his speeches throughout the years using Natural Language Processing (NLP).
Of course, before I can analyze his speeches, I need to actually collect them first. This is Part 1 of the Duterte Speech Analysis, which will focus on scraping Duterte’s speeches. The succeeding articles of this series will then be centered on analyzing them using NLP. For the code I made, check out my GitHub repository — though I admit my code’s kind of messy here — and leave comments or suggestions if you have any. I’ll also eventually update my README with instructions on how to run the Python programs.
Luckily for me, the country’s Presidential Communications Operations Office (PCOO) publishes his speeches in their website, at least up until a certain point (the page shows that it has 31 pages but if you navigate beyond a certain page number, it returns a “page can’t be found” error). Hence, the speeches I gathered were only from 2020 until 2018.
For the scraping of the speeches, I primarily used theBeautifulSoup4 and requests Python libraries, while also using pandas for structuring the dataset, python-dotenv for the environment variables, and a bunch of libraries for extracting texts from PDFs.
Initially, when I tried using the requests library to make a GET request to the PCOO website, I kept getting a connection error for some reason. Upon further research, I found out that it’s because some websites, including PCOO’s, generally dislike robots crawling all over them. You can see this by going to their robots.txt, showing the following settings aimed for disallowing robots.
User-Agent: *Disallow:
Hence, I had to perform several extra steps as a measure to overcome the settings against robots and as a precaution in order to prevent my IP address from being blacklisted. Firstly, I had to disguise my robot program as a human by attaching a user agent to my every request, which I stored in my .env file (hence why I installed python-dotenv). Secondly, it would be way too obvious if one IP address made GET requests to a website at a rapid-fire pace, so I spread out each request by 15 seconds using time.sleep to make it look more human. Admittedly, this increased the scraping time by a lot, but later I will explain how I was able to vastly speed up the process.
The above snippet shows part of the main program found in the 01_scraper.py file, which scrapes the speeches per page of the website. Each page contains 20 speeches, each of which I refer to as row in my program. Each speech takes you to the speech detail page (example) containing the audio, full transcript, and possibly the video of the press conference. The full transcript redirects you to a PDF of the transcript of the speech (example).
I made a function called extract_page in my scraper_functions.py that processes each row using the above procedure and returns a dictionary containing the title, date of speech, and the extracted text.
The next question then is, how do I extract the text from the PDFs?
To reduce scraping time in case I wanted to try more PDF libraries, I saved the PDFs in my repository. This way, I didn’t have to scrape through the PCOO website again. After that, I found four Python libraries for extracting text from PDFs:
PyPDF2slate3ktikaPDFPlumber
PyPDF2
slate3k
tika
PDFPlumber
Initially, I used PyPDF2, but for some speeches it was only able to extract empty new lines, so it would end up with something like \n\n\n\n\n\n\n\n. As a result, I made a function for to check whether the extracted text returned a string with no words using regex. If the function returned True, then I use slate3k to extract for that speech instead. Still, the library sometimes had faulty results, such as totally missing parts of the speech and weirdly spacing out each letter of the text for some speeches (l i k e t h i s). Nevertheless, I compiled all the results using PyPDF2 and slate3k into one csv.
Next, I tried out tika because I was too frustrated with the previous results while I was trying to analyze, and it worked beautifully. So beautifully in fact that I didn’t get to try out PDFPlumber (so if you’ve used it before, do let me know whether it’s as functional!). No errors were caught while extracting everything, so it was pretty much a smooth process using that library. I’ve decided that I’ll use the tika results instead of the other three packages for my analysis.
I mentioned above that there were two stages of scraping for each speech: the speech detail page and the PDF transcript page, amounting to more than 30 seconds of scraping per speech. Given that there are 20 speeches per page, it takes us more than 10 minutes (plus an additional 15 seconds for scraping the page itself) to extract all the speeches in one page. There were about 21 valid pages, so overall it should theoretically take 210 minutes or about 3.5 hours to extract all of them. However, in reality it took me about a week to finish the scraping because of the numerous errors that popped up while using PyPDF2 and slate3k.
This led me to explore multiprocessing to speed up the whole process using tika instead as my PDF extractor, and it worked wonders.
Since I wasn’t able to save the PDFs locally during my first trial run, I had to restart the whole process of scraping. But with multiprocessing, I could have multiple workers process the same set of data (in this case, the 20 speeches per page) concurrently instead of having to process each speech sequentially.
Using the Pool object representing the pool of worker processes, I was able to map the data (the 20 rows of data per page) to the worker processes, and the workers ran the extract_page function concurrently. After all the workers are done, the list of 20 dictionaries returned by the function was appended to the working pandas DataFrame. I also included a text file for logging each speech as I processed them (which I had also done for the first trial run).
I compared the total time it took to scrape one whole page for the first and second trial runs using my scrape_logs.txt, and the difference was apparent. For the first trial run, it took 13–20 minutes to complete one whole page, while it took only 4–5 minutes for my second trial run involving multiprocessing. This is also thanks to the smooth sailing of the tika library, where I encountered no errors in extracting the text (though I did encounter this weird recursion limit error related to the multiprocessing, which I remedied by adding the sys.recursionlimit(10000)). Overall, the multiprocessing trial run lasted only about 1.5 hrs!
I was also satisfied with the PDF extraction itself, seeing that there were only negligible errors in the texts obtained using tika. Hence, I opted not to proceed with the PDFPlumber anymore.
Overall, I successfully scraped the PCOO website in order to collect Duterte’s speeches and to compile them into one csv file, further speeding up the whole process by incorporating multiprocessing.
For Part 2 (and possibly Part 3), I will attempt to analyze this data that I gathered using NLP tools such as NLTK and TF-IDF/N-gram. Again, please feel free to go through my Github repository and leave comments or suggestions below if you have any! I will very much appreciate it.