Social media emotion macroscopes reflect emotional experiences in society at large

Social media generate data on human behaviour at large scales and over long periods of time, posing a complementary approach to traditional methods in the social sciences1. Millions of texts from social media can be processed with computational methods to study emotions2,3 over time and across regions4,5. However, recent research has shown weak correlations between social media emotions and affect questionnaires at the individual level6,7 and between static regional aggregates of social media emotion and subjective well-being at the population level8, questioning the validity of social media data to study emotions. Yet, to date, no research has tested the validity of social media emotion macroscopes to track the temporal evolution of emotions at the level of a whole society. Here we present a pre-registered prediction study that shows how gender-rescaled time series of Twitter emotional expression at the national level substantially correlate with aggregates of self-reported emotions in a weekly representative survey in the United Kingdom. A follow-up exploratory analysis shows a high prevalence of third-person references in emotionally-charged tweets, indicating that social media data provide a way of social sensing the emotions of others9 rather than just the emotional experiences of users. These results show that, despite the issues that social media have in terms of representativeness10 and algorithmic confounding11, the combination of advanced text analysis methods with user demographic information in social media emotion macroscopes can provide measures that are informative of the general population beyond social media users. In our digital society, new technologies for social interaction are playing a key role in capturing and shaping our emotional experiences. Affect plays an central role in several important phenomena12, including affective polarization across political identities13, 14, emotional components in misinformation15, and sentiment in online attention and engagement16. At the same time, social technologies generate data on social interaction that has the potential to capture emotional expression at new scales and resolutions. This has motivated applications of social media data to affective research questions such as mental health17, emotional well-being8, anxiety7, collective emotions18, and emotion regulation19. An especially powerful approach is the implementation of emotion macroscopes that can aggregate affective information at large scales and fast temporal resolutions, often relying on text analysis4 or mental health tracker data20. However, social media data is not designed for behaviour research, bringing concerns about algorithmic and preformative behaviour issues1, 11, as well as sampling biases that make social media users not representative of the population at large in terms of demographics and other relevant attributes10. Testing the validity of social media emotion macroscopes as a measurement of aggregates of emotional experiences has remained an elusive task due to three challenges: scarcity of frequent representative surveys of emotions, the technical barriers to analyze large-scale longitudinal geo-located data from social media, and the low precision of emotion detection methods from social media text. Here we address these three challenges in a pre-registered validation study of social media macroscopes of several emotional states. We use a comprehensive dataset of social media text from the United Kingdom and apply both established and advanced text analysis methods to measure emotion aggregates. Results are then compared with two years of emotion data from a representative survey in the UK. We present a pre-registered study testing that Twitter aggregated emotion timelines positively correlate with weekly emotion reports when applying dictionary-based methods and taking gender into account (more details about pre-registration in the Supplementary Information). The first part of our pre-registration describes a test of our hypotheses for historical data up to October 2020. The second part of the pre-registration postulates the same hypotheses and analysis as a prediction for a future ar X iv :2 10 7. 13 23 6v 1 [ cs . S I] 2 8 Ju l 2 02 1 time period. This way we test the generalizability of our results across time, making a proper prediction rather than doing retrodictive data analysis alone21. Our analysis covers the period between June 2019 and June 2021, including a total of more than 1.5 Billion tweets posted by users in the United Kingdom (see Materials and Methods for more information). We designed the study to test two approaches to emotion detection from text: the most popular dictionary-based method with emotion word lists in English22, and a state-of-the-art supervised classifier that we trained against a large corpus of tweets with annotated emotions2 (see Methods). The fraction of tweets posted by men is above 60%, in line with the higher visibility of men on Twitter23. Because of this, gender-agnostic measures have the risk of overweighting male voices on Twitter. This can be a source of systematic error given the known gender differences in emotional experiences24–26. We use gender information about the users in our sample to calculate gender-rescaled emotional expression indices that are more representative of the population of the UK. The measurements of sadness in the survey and on Twitter are consistently correlated between historical and predicted periods of our pre-registration, as shown in Figure 1. The dictionary approach achieves substantial positive correlations (0.69 in the historical period and 0.672 in the prediction period) as well the supervised method (0.636 in the historical period and 0.653 in the prediction period). Similarly, for the case of scared in the survey versus anxiety or fear in Twitter, illustrated in Figure 1, shows strong correlations in the historical period (0.78 for dictionary method and 0.793 for supervised method). As preregistered, the dictionary method is positively correlated in the prediction period with a coefficient of 0.471, but the supervised method has a weaker correlation close to 0.30. A summary and statistical details of these correlations are reported on Table 1.