# Topic Scaling: A Joint Document Scaling-Topic Model Approach to Learn Time-Specific Topics
## 3 Data and Results  
### State of the Union Corpus  
We use the corpus of the State Of The Union (SOTU) speeches available in the R package _quanteda_[31], gathering 214 speeches of U.S. presidents from 1790 to 2019. We keep, in our analysis, documents starting from 1853 to ensure a party duality of Democratic-Republican in our corpus that later helps us study document-scaling variations, while words, previously lemmatized with the _spaCy_ language model [32], with fewer than 3 occurrences in the selected corpus were excluded to reduce the size of the document term matrix for more efficiency. Figure 1: Plate diagram of Supervised Latent Dirichlet allocation [11]. At first, document-scaling (_Wordfish_) was run on 176 documents, setting the score of [PERSON]'s address in 1981 as being greater than [PERSON]'s of the same year, to identify the scale parameter \(\psi\). Figures 2 and 3 show _Wordfish_ results with a clear time-effect for both parties and noticeable similarities in addresses over time (in blue the _Locally Weighted Scatterplot Smoothing loss_ curve [33]). Democratic addresses underwent a significant change in wording in the early twentieth century with the exception of [PERSON]'s speech of 1916 (Figure 2), but the modern addresses of [PERSON] and [PERSON] share close document positions, and hence similarities, in line with the tree-based topical word-term representation of [PERSON] [22]. On the other hand, Republican addresses show greater variability in terms of document positions (Figure 3) where it is common to have scattered positions of a single president, indicating a potential shift of interest in the addresses. Figure 4 shows the densities of document positions, by party, to be bimodal. Each party dominates one end of a scale, with interchangeability linked to the presidents' tenure in office, as Democratic addresses have a skewed distribution to the left, while Republican ones have a less skewed one to the right. The document position \(\hat{\psi}\approx 0.25\), corresponding to the three addresses of president [PERSON] in 1914, could be interpreted as a cutoff or turning point that separates the studied corpus into two dual periods related to the evolution of rhetorical presidency [27]. _Wordfish_ scores confirm the results of [PERSON] [22], who identified a distinctive style for each president since the speech of [PERSON] in 1934, while previous presidents shared many stylistic aspects. Figure 2: _Wordfish_ scores for Democratic presidents (blue line is the smoothed Loess curve). Word positions plot (Figure 5) shows the contribution of specific words to the estimated _Wordfish_ scale (words with higher \(|\beta_{k}|\)) and their specific effects (\(v_{k}\)). It appears that words related to security, mostly found in modern speeches, contribute mostly to documents with positive scores, i.e., recent addresses, even if their specific effects are relatively low. Figure 4: Density plots of _Wordfish_ scores by party. Figure 3: _Wordfish_ scores for Republican presidents (blue line is the smoothed Loess curve). In the second stage, supervised LDA was run through an Expectation Maximization algorithm (50 steps expectation and 20 steps maximization), with \(\alpha=1\) (Dirichlet hyperparameter for topic proportions), \(\eta=0.1\) (Dirichlet hyperparameter for topic multinomials) and an L2 regularization scheme to learn topic scores with a shrinkage parameter \(\lambda=0.01\). This setting was tested against a plain LDA with a dynamic topic model by keeping the same hyperparameter setting used for sLDA (\(\alpha=1\) and \(\eta=0.1\)). Figures 6 and 7 show the top 10 words in each learned _Topic-Scaling_ model with 10 and 15 topics, respectively, and confirms the existence of nested structures when increasing the number of topics. A model with 15 topics seems to be the winning solution as it maximizes the Renyi entropy (Table 1), while other metrics (\(R^{2}\) and log-likelihood) were not found to be informative with increasing number of topics.