Strategic Analysis of Trump Rallies with NLP and Time Series | by Tan Pengshi Alvin | Towards Data Science

The United States Presidential election will be decided on 3 November 2020, with highly reverberating effects on America and the world. While some people see Trump as a good president, others see him as a populist. Regardless of one’s political inclination, for good or worse, the consequences of the 2020 election could potentially be earth-shattering.
Nonetheless in this article, I will present to you a strategic analysis of Trump rallies, playing as a de facto data scientist on the side of the Biden-Harris campaign. I will attempt to topic-model the rallies with Natural Language Processing (NLP), and then I will identify topic trends in the rallies with time series, before offering a strategic analysis. Firstly, I obtained transcripts of Trump rallies from https://factba.se/transcripts.
In total, I obtained data of about 100 Trump rallies transcripts from his first rally (Melbourne, Florida) in 2017, up till the first rally (Sanford, Florida) after Trump recovered from Covid-19 on 12 October 2020. Thereafter, I cast the data into a Pandas data frame with the Venue, Date and Transcripts as columns. Below is an excerpt from Trump’s rally in Sanford, Florida, and immediately we see the need for text processing:
‘Hello, everybody. Hello, Orlando. Hello, Sanford. It’s great to be with you. Thank you. It’s great to be back. [Audience chants “USA”] That’s a lot of people. You know our competitor, sleepy Joe, he had a rally today and practically nobody showed up. I don’t know what’s going on. Sleepy Joe, but it’s great to be back in my home state, Florida to make my official return to the campaign trail. I am so energized by your prayers and humbled by your support. We’ve had such incredible support and here we are. It’s you know, here we are. But we’re going to finish, we’re going to make this country greater than ever before. [Audience chants “We love you”] Thank you. Thank you. Thank you very much.
With regular expressions, I removed the audience inputs (in square brackets) from the transcripts. I further used NLTK’s RegexpTokenizer to tokenize words with 3 or more letters and removing words with any punctuation, before joining them again into a processed/fragmented transcript.
With a ‘cleaned’ set of transcripts, I proceed to begin modeling. On exploratory data analysis, I used WordCloud to generate the top 50 words that Trump often used in his first 2 rallies and the last 2 rallies that I obtained:
For WordCloud images, the larger the word implies the more frequently the word being appearing in the transcript. With the above visualization, we may see why Trump is so influential to his supporters, with frequent words such as ‘going’, ‘great’, ‘people’ and ‘country’, arousing emotions and a constant call to action. Also we see ‘obamacare’ appearing, demonstrating Trump’s fixation in his early years of presidency.
In the last two rallies obtained, we see that not a lot has changed in Trump’s usage of words. Words such as ‘going’, ‘people’ and ‘great’ continue to be top words, while words such as ‘biden’ and ‘china’ appear as Trump begins to attack political rivals closer to the election.
Coming to topic modeling, we usually use Scikit-Learn’s Count Vectorizer or TFIDF Vectorizer to quantize all words in the transcripts into features. Thereafter, we reduce all features into components (linear combination of the features) through Latent Semantic Analysis (LSA), Non-negative Matrix Factorization (NMF) or Latent Dirichlet Allocation (LDA). The main topics can thus be identified by filtering the top words of the components.
After some trial-and-error, I have identified Count Vectorizer (total of 11353 unique words) with NMF to be the most definitive for Trump’s rallies. And exercising careful analysis and discernment, I identified the 3 topics (using 90 top words) from the NMF components as such:
Topic 1 : Touting Achievements
know, said, right, people, like, great, got, think, going, want, good, lot, years, president, guy, time, thing, look, won, sir, trump, love, way, big, billion, remember, come, let, money, million, went, deal, better, tell, little, year, new, mean, guys, world, things, came, democrats, thank, bad, win, state, best, country, beautiful, hell, times, ago, believe, actually, michigan, true, took, somebody, party, crazy, obama, american, incredible, fake, getting, job, place, happened, day, greatest, man, long, china, away, saying, number, news, america, election, night, nice, wouldn, states, care, history, wanted, everybody, texas, maybe
Topic 2 : Plans and Appealing for Support
going, great, want, people, country, thank, vote, know, america, american, right, like, years, president, got, democrats, way, good, jobs, time, lot, need, think, said, come, new, state, job, let, love, incredible, care, states, americans, tell, united, believe, history, tax, borders, look, election, democrat, big, working, republicans, military, wall, coming, happen, world, percent, law, republican, border, taxes, won, billion, protect, trump, work, things, man, governor, happening, thing, pay, win, deal, ice, beautiful, day, remember, party, veterans, help, folks, voted, passed, everybody, greatest, strong, florida, called, bad, women, illegal, hard, unemployment, today
Topic 3: Attacking Political Adversaries
going, know, said, people, got, biden, great, like, right, want, think, years, joe, good, way, thank, china, remember, america, lot, country, let, thing, look, tell, win, guy, time, job, big, world, left, president, american, mean, police, law, bad, won, states, happen, called, things, deal, history, better, love, state, hell, ago, new, took, end, little, wants, half, ballots, happened, vote, open, year, everybody, come, wall, jobs, went, crazy, news, second, terrible, governor, day, sleepy, bernie, order, pennsylvania, peace, north, military, energy, seen, watch, saw, suburbs, gave, closed, war, fake, trump, saying
Taking the new components as features of the model (instead of the quantized words from Count Vectorizer), NMF is able to attribute topic relevance scores to each of the rallies. Subsequently, I proceed to plot the time series of topic relevance across the rallies:
From the time series, we can see that while the topic on ‘Touting Achievements’ does not show a clear trend, the topic on ‘Plans and Appealing for Support’ seem to be on the downward trend closer to election. The topic on ‘Attacking Political Adversaries’ shows an obvious upward trend towards a plateau closer to election. As such, it seems that there is a shift in tone in Trump’s rallies towards the election — Trump seems less optimistic on his plans and is attacking his opponent more. This also can be clearly observed with a Cosine Similarity Heatmap:
Looking at the first column of the heat map (in comparison with the rally on 12 October), there seems to be a sudden shift in tone of Trump on the rally on 2 March 2020. Could the sudden onset of Covid-19 in the United States made Trump insecure, and therefore prompted him to attack his opponents more?
Also, we observe another shift in tone on the rally on 18 September 2020 — Trump appears to spike in jubilance on his achievements according to the time series. Coincidentally, it was also the day that Supreme Court Justice Ginsburg (Democrat) passed away. Could that also contributed to Trump’s shift in tone?
To further analyze Trump rallies, let us explore if we could break them down into clusters, based on their topic relevance. It turns out that we could, after plotting Sum Inertia against No. of Clusters and Silhouette Coefficient against No. of Clusters:
Although we cannot clearly identify an elbow (Elbow Method) in the Sum Inertia against Clusters plot, the Silhouette Coefficient plot clearly showed that the optimum number of clusters is two! Silhouette Coefficient is typically scored between 1 and -1, and quantifies the closeness of a point to the cluster centroid and away from other clusters’ centroids.
Moving on, we produced a 3-Dimensional plot of Trump rallies with Plotly, where we see 2 types of Trump rallies where he is mainly attacking political adversaries or mainly appealing to voters:
Going back to our time series plots, we noticed something interesting — there seems to be a hint of periodicity of topic relevance in Trump rallies. Also the topics on ‘Plans and Appealing for Support’ and ‘Attacking Political Adversaries’ seem to be have a trend component. As such, the time series are non-stationary, and we could potentially use the model Seasonal Autoregressive Integrated Moving Average (SARIMA) for predictive analytics.
Following the path of exploration, I conducted a simple forward validation by splitting the data set into Training, Validation and Test set with a ratio of 90:5:5
After performing a simple grid search on the optimum hyperparameters with SARIMA, I obtained following Mean Absolute Error (MAE) scores:
Using the optimum hyperparameters, I re-trained the Training and Validation set together with the SARIMA model, and then evaluate on the Test set. The following error scores I found were still quite reasonable:
The time series prediction model could be used to discover potential peaks and troughs in future Trump rally topics. While the Biden-Harris campaign might have its own set of strategy against the Trump campaign, one could also take reference from an ancient Chinese strategy manual, the Sun Tze Art of War, which advocates doing the inverse of that of the opponent.
“When strong, avoid them. If of high morale, depress them. Seem humble to fill them with conceit. If at ease, exhaust them. If united, separate them. Attack their weaknesses. Emerge to their surprise.” ― Sun Tzu
For instance, Biden could attack Trump more on periods when he is predicted to be less aggressive. Conversely, Biden could appeal to the crowd and talks about his plans when Trump is predicted to be more aggressive. As such, a model which could predict Trump temperament and tone in his campaigning would be very useful for the Biden-Harris camp.
In summary, I have completed an insightful exploratory data analysis on Trump’s rallies, and also created a predictive model on Trump’s topic trend, as a case study.
This careful analysis is created as Project 4 of the Metis Data Science Bootcamp (Singapore), and I would like to thank everyone, especially my instructor Neo Han Wei, who has guided me thus far. It has been an exhilarating experience, and an amazing learning journey. Thank you!
Here is the link to my GitHub, which contains all the codes and presentation slides for this project. Also reach me on my LinkedIn or comment here below to discuss!
Support me! — If you are not subscribed to Medium, and like my content, do consider supporting me by subscribing via my referral link here.