Recursive Masking Reveals Subtle Differences in Campaign Style
Creators: name: , Musashi
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: American Politics
subject: Linguistics
subject: FOS: Languages and literature
subject: Models and Methods
subject: Political Science
subject: FOS: Political science
subject: Social and Behavioral Sciences
subject: Computational Linguistics

description: There are two levels to this proposal. The first is to use the integrated gradients (Sundarajan Taly and Yan 2017) extracted from multimodal transformers trained to classify advertisements as belonging to one political campaign or another to infer the visual and linguistic features that distinguish pairs of campaigns. The second more novel idea is to recursively mask the most informative features until the classifier fails, in order to observe "subtler" differences between campaigns. As noted in Lauderdale and Herzog (2016, and elsewhere), there are hierarchical sources of variance in a corpus of texts, from language (i.e. English vs French) at the highest level, down to topic, speaker style, and eventually stance (and so on). It is my expectation that a text-based classifiers trained to distinguish Trump and Biden advertisements will primarily pick up on topical differences between the two campaigns; if an advertisement mentions borders and crime, it is likely a Trump advertisement, whereas if it mentions healthcare it is more likely to be a Biden advertisement. Although at some level, it is interesting to be able to say that the _most important_ difference between the Trump and Biden advertisement campaigns was the issues they focused on, this leaves us unable to search for differences occurring at hierarchically lower levels of variance. For instance, we may explicitly be interested in non-topical differences between the Trump and Biden campaigns, such as use of emotive language and a broad range of other characteristics that might come under the heading of "style". In order to explore these subtler sources of variance, I propose the following procedure. We first fine-tune a transformer model with a binary classifier head on Trump and Biden advertisements while randomly permuting some portion of the text with special `[MASK]` tokens. Then for each advertisement in the unseen subset of the data, we apply the following recursive procedure: 1. Predict label of an unseen advertisement (first time without random masking). If incorrect, STOP. If correct, proceed to 2. 2. Identify the token that contributed the most information to the correct prediction using integrated gradients. 3. Replace this token with `[MASK]`. Return to 1. Several quantities of interest are returned from this procedure. The tokens that are masked, and the order in which they are masked, informs us of the hierarchical sources of variance differentiating the two campaigns at the token level. Although this is limiting to the extent that relevant linguistic phenomena (i.e. style or intent) likely do not occur at the token level, we can still infer the semantic and syntactic contributions of these tokens to the statement that they were masked from. A second quantity of interest is the point at which the classifier can no longer correctly predict the label of the advertisement. Depending on the interpretability of the remaining unmasked portion of the advertisement, analysing this output will reveal what the two campaigns have in common. Although my expectation is that this will be largely tokens that contribute to syntax than semantics, the extent to which these remainders contain value-laden terms will reveal common features of the two campaigns.