Donate or Create? Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts
Authors: Christopher Bagdon, Aidan Combs, Carina Silberer, Roman Klinger
Accepted at: acl25

Donate or Create? Comparing Data Collection Strategies for Emotion-labeled Multimodal Social Media Posts

Christopher Bagdon1, Aidan Combs1,2,3, Carina Silberer3, and Roman Klinger1
1Fundamentals of Natural Language Processing, University of Bamberg, Germany
2Department of Sociology, The Ohio State University, USA
3Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany
{christopher.bagdon,roman.klinger}@uni-bamberg.de
combs.494@osu.edu; carina.silberer@ims.uni-stuttgart.de

Abstract

Accurate modeling of subjective phenomena such as emotion expression requires data annotated with authors’ intentions. Commonly such data is collected by asking study participants to donate and label genuine content produced in the real world, or create content fitting particular labels during the study. Asking participants to create content is often simpler to implement and presents fewer risks to participant privacy than data donation. However, it is unclear if and how study-created content may differ from genuine content, and how differences may impact models. We collect study-created and genuine multimodal social media posts labeled for emotion and compare them on several dimensions, including model performance. We find that compared to genuine posts, study-created posts are longer, rely more on their text and less on their images for emotion expression, and focus more on emotion-prototypical events. The samples of participants willing to donate versus create posts are demographically different. Study-created data is valuable to train models that generalize well to genuine data, but realistic effectiveness estimates require genuine data.

1 Introduction

Emotions play a fundamental role in communication (Chen et al., 2022; Chung and Zeng, 2020), particularly in online settings (Derks et al., 2008). On contemporary social media sites, authors often express emotion through a combination of text and visual content (Illendula and Sheth, 2019; Li and Xie, 2020). Modeling the emotions expressed in social media posts therefore requires multimodal datasets labeled for author emotion. This is, however, a challenging task: Emotions are internal psychological states and external annotators can therefore only approximate the correct labels (Troiano et al., 2023; Nakagawa et al., 2022).

One approach to mitigate this annotator–author label mismatch is to ask study participants to create content fitting provided labels (Troiano et al., 2023, “Write a text that caused emotion X”).). While this is simple to implement, the resulting data may differ from real social media data. Such lack of generalizability may lead to limited model robustness (Degtjar and Rose, 2023; Elangovan et al., 2024; Ribeiro et al., 2020; Yang et al., 2023).

An alternative approach is to ask social media users to donate and label their real social media posts (Oprea and Magdy, 2020). While this approach may provide more realistic data, it requires more precautions to protect participant privacy (Keusch et al., 2024; Gomez Ortega et al., 2023).

Little is known about what precisely the differences between study-created and donated author-labeled content may be, or how significant differences may be for modeling. We provide a better understanding of the tradeoffs of these corpus collection methods with the goal of informing future author-labeled corpora collection efforts. To do this, we collect study-created and genuine multimodal social media posts labeled by their authors for emotion. We analyze differences between the events that inspire the posts, how they are labeled for emotion, and sample characteristics. Finally, we explore the impact of these differences on emotion modeling and prediction.

We implement three collection procedures:
1. Creation: Study-created data. Participants are asked to create posts about an event they experienced that elicited an emotion for which we prompt. This approach is clear-cut to conduct but potentially lacks generalizability.
2. Donation: Genuine data. Participants provide posts from their social media accounts about an event they experienced that elicited
a prompted emotion. This method yields real-world data balanced across emotions, but might come with privacy issues and participant’s self filtering, as well as potentially limited availability of posts.

3. RECENT: Genuine data. Participants submit their five most recent posts and then annotate each for emotion. While this method avoids potential experiment bias in emotion annotation, it may underrepresent emotions that are rarely shared on social media.

We find that (1) study-created data differs from genuine data in several ways. Notably, it is dominated by prototypical emotion triggers, while genuine data is more diverse. (2) The data collection procedures lead to different samples of participants. (3) Models trained on CREATION generalize well to genuine data, but DONATION test data is required to realistically estimate their effectiveness.

2 Related Work

Asking authors to label text is useful in areas where author intent is both important and unclear from the text alone. For example, author-annotated corpora exist for deception detection (Capuozzo et al., 2020; Veluthambath et al., 2024), sarcasm detection (Oprea and Magdy, 2020; Abu Farha et al., 2022), and, of particular interest to us, emotion detection (Kajiwara et al., 2021; Troiano et al., 2019; Scherer and Wallbott, 1997; Troiano et al., 2023). In this section, we review common methods for collecting annotations of authors’ internal states.

2.1 Genuine Data Collection

The standard approach in natural language processing and computer vision is to acquire genuine data from the world and request annotations from external annotators. These annotators may be trained experts or recruited through crowdsourcing platforms. However, since annotators do not have access to the original author’s internal state, their annotations are often inaccurate (Kajiwara et al., 2021; Troiano et al., 2023; Nakagawa et al., 2022).

To circumvent this issue, researchers may try to indirectly acquire labels from authors. For example, in social media data, author labels can sometimes be inferred using corresponding hashtags (Mohammad, 2012; Abbes et al., 2020). While such approaches are useful when reliable markers are available, they are also subject to error: Annotating texts with hashtags as an indicator of a particular label makes other such markers redundant.

If author intent is difficult to access in an indirect, rule-based manner, an alternative is to ask social media users to donate and label their own data (Oprea and Magdy, 2020; Kajiwara et al., 2021; Razi et al., 2022; Pfiffner et al., 2024). Directly obtaining labels from authors eliminates incorrect inferences as sources of error.

Collecting genuine social media data annotated by authors is challenging (van Driel et al., 2022). This is because of privacy concerns (Boeschoten et al., 2022; Carrière et al., 2024; Gomez Ortega et al., 2023), the required technical skill set of participants (Keusch et al., 2024), and the potentially limited availability of requested types of data.

![Data collection process flowchart](page_1012_186_388_246.png)

Figure 1: Data collection process.

2.2 Study-created Data Collection

An alternative to collecting real data is to construct author-labeled datasets by asking participants to role-play or write according to specific instructions that provide target labels. For example, participants may be asked to tell truths or lies (Ott et al., 2013; Capuozzo et al., 2020; Veluthambath et al., 2024; Lloyd et al., 2019), write about events that elicited certain emotions (Troiano et al., 2019, 2023), respond with specific coping strategies (Troiano et al., 2024), or act out hypothetical emotional scenarios (Busso et al., 2008, 2016).

Such approaches for obtaining study-created data have various advantages. They avoid the difficulties of finding a sufficient amount of content fitting uncommon labels “in the wild”. Participants have more control over what they contribute, and can better guard their privacy. Finally, the data can be less prone to recall bias as large time gaps between content creation and labeling are avoided.

Such methods may, however, be susceptible to other experiment effects (Vania et al., 2020). For instance, people behave differently when they know they are participating in a study, often to confirm
<table>
  <tr>
    <th>Label</th>
    <th>Question Text</th>
    <th>Options</th>
  </tr>
  <tr>
    <td colspan="3"><b>Emotion annotation (RECENT only)</b></td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>Please select the emotions that you felt as a result of this event [multiple]</td>
    <td>[Emo.]</td>
  </tr>
  <tr>
    <td>Intensity</td>
    <td>Please rate how intensely you felt each of these emotions as a result of this event [Emo.]</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td colspan="3"><b>Text–image relationship:</b> How much do these statements apply?</td>
  </tr>
  <tr>
    <td>Text describes image</td>
    <td>The text directly describes the image.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Text → image</td>
    <td>The text is required to understand the image.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Image → text</td>
    <td>The image is required to understand the text.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Image conveys emotion</td>
    <td>The image explicitly conveys the emotion you posted about.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Text conveys emotion</td>
    <td>The text explicitly conveys the emotion you posted about.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td colspan="3"><b>Event experience</b></td>
  </tr>
  <tr>
    <td>Event Duration</td>
    <td>How long did the event last?</td>
    <td>[Time]</td>
  </tr>
  <tr>
    <td>Emotion Duration</td>
    <td>How long did you experience emotion as a result of the event?</td>
    <td>[Time]</td>
  </tr>
  <tr>
    <td>Event Intensity</td>
    <td>How intense was your experience of the event?</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Emotion Intensity</td>
    <td>How intense was your experience of this emotion?</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td colspan="3"><b>Appraisal:</b> Think back to when the event happened and recall its details. Take some time to remember it properly. How much do these statements apply? Some statements might not fit the event exactly, please answer to the best you can.</td>
  </tr>
  <tr>
    <td>Familiarity</td>
    <td>The event was familiar.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Predictability</td>
    <td>I could have predicted the occurrence of the event.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Attention</td>
    <td>I had to pay attention to the situation.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Notconsider</td>
    <td>I tried to shut the situation out of my mind.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Pleasantness</td>
    <td>The event was pleasant for me.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Unpleasantness</td>
    <td>The event was unpleasant for me.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Goalrelevance</td>
    <td>I expected the event to have important consequences for me.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Ownresponsibility</td>
    <td>The event was caused by my own behavior.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Goalsupport</td>
    <td>I expected positive consequences for me.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Anticipconseq</td>
    <td>I anticipated the consequences of the event.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Owncontrol</td>
    <td>I was able to influence what was occurring during the event.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Otherscontrol</td>
    <td>Someone other than me was influencing what was occurring.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Acceptconseq</td>
    <td>I anticipated that I would easily live with the unavoidable consequences of the event.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Effort</td>
    <td>The situation required me a great deal of energy to deal with it.</td>
    <td>1...5</td>
  </tr>
  <tr>
    <td>Internalstandards</td>
    <td>The event clashed with my standards and ideals.</td>
    <td>1...5</td>
  </tr>
</table>

Table 1: Wording and response options for survey questions used in the analysis. [Emo.] refers to Anger, Disgust, Fear, Joy, Sadness, Surprise. [Time] refers to one of Seconds, Minutes, Days, Weeks, Months.

the hypotheses they think the researchers have (Mummolo and Peterson, 2019; Nichols and Maner, 2008). More emotionally intense events are remembered more easily, potentially leading participants to preferentially select them when writing posts about them after-the-fact (Kensinger, 2009). Writing differs based on audience, for example varying in the degree to which it is stereotype-consistent (Lyons and Kashima, 2006, 2003). Differences between study-created data and actual social media content are likely to harm the generalizability of models trained on study-created data (see Elangovan et al., 2024; Ribeiro et al., 2020; Yang et al., 2023, for a discussion of differences between holdout data and real-world data).

3 Data Acquisition Methods

In this section we discuss the data acquisition methods and their advantages and disadvantages.1

1All data, code, and surveys are available at https://www.uni-bamberg.de/en/nlproc/projects/item/

3.1 Process Overview

The data collection process is illustrated in Figure 1. Potential participants see the study details and choose to participate or decline. Those who accept are asked to provide a social media post using one of three data collection strategies – CREATION, DONATION, and RECENT – and then annotate it. Each participant provides five posts. Finally, participants answer questions about themselves. Participants may complete the study as many times as they wish, up to once per emotion.

3.2 Data Collection Strategies

We request posts that contain both text and an image and are authored by the participant. Other instructions vary by collection strategy.

CREATION. We ask participants to recall an event in which they felt a particular emotion – anger, disgust, joy, fear, sadness, or surprise – and which they remember well. They then write a social media post about it. Participants must select an image from the Flickr database of Creative Commons licensed
images2 that is similar to the one they would have used if they had created the post naturally. This approach diminishes the risk to participant privacy, but posts may differ from genuine data.

DONATION. We prompt participants for an emotion and ask them to share a genuine post from their timeline. They do so by copy–pasting the text and uploading the image from that post. This approach yields genuine posts while equally representing emotions regardless of their prevalence on social media. However, participants may struggle to find posts representing uncommon target emotions. This approach also raises greater privacy concerns.3

RECENT. RECENT is similar to DONATION, but we do not prompt participants for particular emotions. Instead, we ask them to share their five most recent multimodal posts and annotate each with all emotions, and the associated intensities, they felt in response to the event that inspired the post. We adapt this emotion annotation approach from Rhodes-Purdy et al. (2021).4 RECENT diminishes concerns about the accuracy of emotion annotations at the cost of potentially underrepresenting uncommon emotions. The same privacy concerns that affect DONATION also apply to RECENT.

3.3 Annotation Details

Post and Event Annotations. Table 1 shows the questions participants answer about each of their posts. To understand the roles of images and text, we ask about the relationship between the modalities. To understand the link between the event and the emotion category, we request appraisal labels (Scartantino, 2016; Scherer, 2005). Appraisals are a psychological theory of how events induce emotion (Troiano et al., 2023; Stranisci et al., 2022).

Participant Information. After participants annotate their posts, we ask about their age, gender, education, ethnicity, frequency of social media use and posting, and choice of social media platform. We additionally use demographic information provided by the research platform we use – namely, employment and student status. We summarize these questions in Table 5 in the Appendix.

Study Details. We recruit participants using Prolific. We require that they reside in the United Kingdom or Ireland, have resided there for at least five years, are 18 or older, and be native English speakers. We restrict participation to these demographics to avoid confounding variables, as emotion expression can differ from culture to culture. The survey is conducted via Google Forms. CREATION and DONATION take 30 minutes and we pay participants £4.50. RECENT takes 40 minutes and we pay participants £6.00. After removing posts that do not meet our requirements, our dataset contains 2,507 posts authored by 522 participants.

<table>
  <tr>
    <th rowspan="2">Emotion</th>
    <th colspan="3">Study</th>
  </tr>
  <tr>
    <th>CREATION</th>
    <th>DONATION</th>
    <th>RECENT</th>
  </tr>
  <tr>
    <td>Anger</td>
    <td>193</td>
    <td>17%</td>
    <td>174</td>
    <td>15%</td>
    <td>16</td>
    <td>8%</td>
  </tr>
  <tr>
    <td>Disgust</td>
    <td>202</td>
    <td>17%</td>
    <td>195</td>
    <td>17%</td>
    <td>10</td>
    <td>5%</td>
  </tr>
  <tr>
    <td>Fear</td>
    <td>193</td>
    <td>17%</td>
    <td>189</td>
    <td>16%</td>
    <td>4</td>
    <td>2%</td>
  </tr>
  <tr>
    <td>Joy</td>
    <td>189</td>
    <td>16%</td>
    <td>196</td>
    <td>17%</td>
    <td>158</td>
    <td>79%</td>
  </tr>
  <tr>
    <td>Sadness</td>
    <td>185</td>
    <td>16%</td>
    <td>197</td>
    <td>17%</td>
    <td>16</td>
    <td>8%</td>
  </tr>
  <tr>
    <td>Surprise</td>
    <td>197</td>
    <td>17%</td>
    <td>198</td>
    <td>17%</td>
    <td>30</td>
    <td>15%</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>1159</th>
    <th></th>
    <th>1149</th>
    <th></th>
    <th>199</th>
    <th></th>
  </tr>
</table>

Table 2: Distribution of posts in the dataset. In RECENT, posts are categorized into emotions based on which emotion the participant reported feeling most intensely. In the case of ties, the post is counted in both categories.

dom or Ireland, have resided there for at least five years, are 18 or older, and be native English speakers. We restrict participation to these demographics to avoid confounding variables, as emotion expression can differ from culture to culture. The survey is conducted via Google Forms. CREATION and DONATION take 30 minutes and we pay participants £4.50. RECENT takes 40 minutes and we pay participants £6.00. After removing posts that do not meet our requirements, our dataset contains 2,507 posts authored by 522 participants.

4 Analysis and Modeling

We answer research questions about differences in the data between collection strategies (RQ1), differences in the events that lead to the posts (RQ2), differences between samples (RQ3) and in how participants assign labels (RQ4), and the impact on model performance and generalization (RQ5).

4.1 RQ1: Are there differences in posts between data collection strategies?

Table 2 shows the post and label distributions. We see that CREATION and DONATION represent emotions nearly evenly by design. In RECENT, we obtain posts dominated by joy.

Text. Figure 2 shows the average lengths in character counts. CREATION posts are longer than DONATION and RECENT posts, especially for posts about joy and surprise. Controlling for differences in emotion distribution, CREATION posts are 51% longer than RECENT posts and 26% longer than DONATION posts (p<0.01 for both).5

Image Style. To investigate differences in image style, we manually label each image as a meme, screenshot, graphic, professional photo, personal

5Examples can be seen in Appendix A.2.2.
Figure 2: Length of the posts, in characters, by emotion and study. Means are represented by points. Outlined boxes indicate significant differences (one-way ANOVA, p<0.05 after Bonferroni correction).

Figure 3: Distribution of image style labels.

photo, or other.\footnote{Details and examples are in Appendix A.2.} Figure 3 shows the distribution of image styles across the three data collection strategies. They differ significantly between studies (\( \chi^2 \) test p<0.001). Personal photos are most prominent across studies. The remaining images for CREATION are dominated by professional photos. DONATION and RECENT have fewer professional photos and instead more screenshots, graphics, memes, and other images, which are less prevalent in the study-created data.

The lack of screenshots in CREATION is a consequence of requiring participants to select images from an existing database. Figure 4 illustrates this difference. This is backed by feedback from study participants who found the database limiting. Further, it shows that study-created data might not include all real-world triggers for social media posts.

Image Content. We label the images with GPT-4o to analyze the content.\footnote{https://openai.com/gpt-4, prompt and validation details are in Appendix A.2.} Table 3 shows the 10 most frequent labels for each study (Table 7 in the Appendix shows the 50 most frequent labels). Despite the clear differences in image style that we observed, the word lists indicate that the images show comparable content across all collection

<table>
  <tr>
    <th>Creation</th>
    <th>Donation</th>
    <th>Recent</th>
  </tr>
  <tr>
    <td>text</td>
    <td>43%</td>
    <td>55%</td>
    <td>text</td>
    <td>58%</td>
  </tr>
  <tr>
    <td>background</td>
    <td>12%</td>
    <td>tree</td>
    <td>11%</td>
    <td>tree</td>
    <td>14%</td>
  </tr>
  <tr>
    <td>tree</td>
    <td>12%</td>
    <td>background</td>
    <td>9%</td>
    <td>woman</td>
    <td>13%</td>
  </tr>
  <tr>
    <td>sky</td>
    <td>11%</td>
    <td>woman</td>
    <td>9%</td>
    <td>man</td>
    <td>12%</td>
  </tr>
  <tr>
    <td>grass</td>
    <td>11%</td>
    <td>sky</td>
    <td>9%</td>
    <td>sky</td>
    <td>12%</td>
  </tr>
  <tr>
    <td>building</td>
    <td>9%</td>
    <td>person</td>
    <td>8%</td>
    <td>background</td>
    <td>11%</td>
  </tr>
  <tr>
    <td>water</td>
    <td>8%</td>
    <td>people</td>
    <td>8%</td>
    <td>person</td>
    <td>10%</td>
  </tr>
  <tr>
    <td>people</td>
    <td>8%</td>
    <td>grass</td>
    <td>8%</td>
    <td>grass</td>
    <td>9%</td>
  </tr>
  <tr>
    <td>woman</td>
    <td>7%</td>
    <td>man</td>
    <td>8%</td>
    <td>smile</td>
    <td>8%</td>
  </tr>
  <tr>
    <td>car</td>
    <td>7%</td>
    <td>smile</td>
    <td>7%</td>
    <td>building</td>
    <td>8%</td>
  </tr>
</table>

Table 3: 10 most common image content words per study. Image content word extraction done by GPT-4o.

![Comparison of posts submitted for CREATION and RECENT. CREATION is a more general comment, while RECENT is a reaction to a specific news item. The latter style does not occur in the CREATION data.](page_1016_1012_388_180.png)

(a) CREATION post labeled as surprise.

(b) RECENT post labeled as anger.

Figure 4: Comparison of posts submitted for CREATION and RECENT. CREATION is a more general comment, while RECENT is a reaction to a specific news item. The latter style does not occur in the CREATION data.

strategies, but with differences in the order. The most common for all three was text, though this was less frequent for CREATION (43%) than for DONATION (55%) and RECENT (58%). This is consistent with the smaller number of screenshots and lack of memes in CREATION. The image labels are dominated by references to people or scenery.

Text–Image Relation. To understand the role of the image and the text in conveying an emotion, we asked participants questions about the relationship between the two. Figure 6 shows the response distributions. CREATION participants describe their post images as less necessary to understand the text and as conveying emotion less than DONATION and RECENT posts. CREATION post images are not as integral to the post as a whole. In contrast, in RECENT posts, the text and images are more dependent on one another.

4.2 RQ2: Are there differences in the events that inspire participants to write posts?

With this and the following research questions, we aim at understanding potential reasons for the differences that we observe in the posts.

Emotion Appraisals. Figure 5 shows the distribu-
Figure 5: Responses to event appraisal questions (columns) across emotions (rows) and studies (color). Responses were provided on 5-point Likert scales. Means are represented by points. Outline: Significant at p<0.05 according to an ANOVA with Bonferroni correction.

Figure 6: Participant ratings of the relationship between post text and images, on five point Likert scales. Means are represented by points. Outline indicates significance (one-way ANOVA, p<0.05 after Bonferroni correction).

In most cases, the events that inspired study-created posts are rated more highly on the appraisal dimensions than the events that inspire genuine posts. The cases which break this pattern tend to be those appraisal dimensions which are negatively associated with the emotion at hand. For example, participants rate disgust-causing events as less pleasant and less likely to have positive consequences in CREATION. RECENT posts, in contrast, tend to rate lowest on these appraisal dimensions, particularly in joy and surprise, where they are best represented. This suggests that participants use more prototypical events for particular emotions in CREATION than in genuine posts.

Duration and Intensity. Events that are recalled by prompting for a specific emotion may be dominated by the emotion and the duration and intensity of the event. To analyze this, we estimate mixed-effects models that include the study type and emotion as independent variables and random intercepts to account for grouping by participant. We find that, on a 5-point Likert scale, controlling for emotion, participants rate CREATION events as 0.34 points more intense than DONATION events, and their emotional responses to CREATION events as 0.36 points more intense (p < 0.001 for both). There are no intensity differences between DONATION and RECENT events or emotional responses.

8The reason is, presumably, that the imbalanced corpus in RECENT leads to more power to detect significant differences for Joy. When RECENT is excluded from consideration, there are similarly many significant differences between CREATION and DONATION for Joy as for the other emotions.
Figure 7: Differences in decline rates between studies and emotions. “Decline” means that a participant opted not to participate after reading the study information.

Participants report that emotion responses to CREATION events last significantly longer than responses to DONATION events (p < 0.001), and responses to DONATION events last significantly longer than responses to RECENT events (p < 0.01). There are no significant differences between CREATION and DONATION in the length of the events themselves. However, RECENT events are significantly shorter than DONATION events (p < 0.01).

It is possible that these observations are a consequence of the role that emotions play in memorizing events. In CREATION, participants may preferentially select events that are more memorable and more emotionally prototypical than those they actually post about on social media.

4.3 RQ3: Are there differences in participant characteristics?

We find that participants who completed CREATION are significantly older, less likely to be students, and more likely to be European than those who completed DONATION and RECENT.9

RECENT and DONATION require that participants be comfortable sharing their real social media posts with researchers, and, additionally, DONATION requires that those posts be about specific emotions. These differences in study requirements may change who is willing to participate. Figure 7 reports the decline rates after having read the instructions. Decline rates are considerably higher in DONATION and RECENT than in CREATION (\( \chi^2 \) test, p < 0.001). Participants are indeed hesitant to provide researchers with their social media posts.

In DONATION we find differences between decline rates across emotions (\( \chi^2 \) test p < 0.001). Participants finish the study more often when asked for joy-inducing event posts. The fact that the CREATION decline rate does not vary by emotion suggests that people do not find it particularly challenging to create posts about emotions other than joy, but rather they struggle to find appropriate posts in their social media feeds. Participants who declined DONATION and RECENT commonly cited privacy concerns and, in DONATION, they noted a lack of posts that reflected the desired emotion.

4.4 RQ4: Are there differences in how participants label posts for emotion?

Some emotions are harder to find on social media, and some posts may be about events which evoke more than one emotion. This may lead DONATION participants searching for uncommon emotions to submit posts which also or better represent common emotions. In such cases, the labels of the posts would be affected by the target labels presented to the participants, which is an undesirable potential source of bias. To investigate the potential scope of this issue, we allowed participants to label their posts freely with any number of emotions in RECENT. We presume that the annotations of multi-emotion posts are likely to be more affected by the target label presented to the participant.

Figure 8 shows the fractions of posts to which multiple emotions were assigned. Multi-emotion posts are common. Most anger, fear disgust, and surprise posts were additionally labeled with another emotion. We conclude that bias in DONATION emotion annotations is a possibility that should be investigated and mitigated. One approach would be to allow participants to label their posts with multiple emotion labels and the intensity of those emotions, as we did in RECENT.

4.5 RQ5: Do data differences affect model performance?

Our experiments aim to assess if CREATION and DONATION are equally suitable as training and as testing data for predictive models10. Specifically,

9Table 6 in the Appendix shows the sample demographics.

10We do not train or test on RECENT because of its smaller size and unbalanced emotion distribution.
Figure 8: Prevalence of multi-emotion posts in the RECENT study. Lines represent 95% confidence intervals. Posts were counted as multi-emotion under the Inclusive definition if the participant indicated the event evoked two or more emotions, and under the Strict definition if the participant indicated that two or more emotions were tied for greatest intensity.

do data differences between CREATION and DONATION have an effect on model performance when used as training data? Are there differences in effectiveness when testing on CREATION vs. DONATION? We fine-tune unimodal and multimodal models separately on CREATION and DONATION data subsets. We also evaluate multimodal foundation models in a zero-shot setup.

Setup. We divide the data such that the development and test sets each have 25 posts per emotion (300 posts per set) and use the remaining data for training (800 posts per strategy).11 As unimodal models, we use RoBERTa for text (Liu et al., 2019) and ViT for images (Dosovitskiy et al., 2021). For multimodal models, we use the dual encoder CLIP (Radford et al., 2021), applying early fusion by concatenating text and image embeddings, then adding a classification head on top. Appendix A.3 gives model and training details. All models are fine-tuned and tested five times.

For the zero-shot setup, we prompt llama3.2-vision, llava-llama3, and minicpm-v. We prompt each model five times and report results averaged across runs. We select the best model per modality on the development data and report the corresponding result on the test data12. We provide model details and prompts in Appendix A.3.

Results. Table 4 shows the main results.13 Models trained on CREATION and DONATION data perform equally well when tested on DONATION data (bottom block in Table 4). This suggests that the differences between CREATION and DONATION content may not be very important for model training. Performance scores on CREATION test data are higher (top block), and likely unrealistically optimistic, in comparison to scores on DONATION test data. This suggests that genuine data is required to reliably estimate model effectiveness. The zero-shot models’ results, which by design solely reflect differences in the test sets, underpin this finding.

Influence of Post and Respondent Features. Our analyses in Sections 4.1, 4.2, 4.3, and 4.4 show that collection strategies lead to differences in samples, and our analyses show that these differences carry over to model performances. We now investigate this relationship between post and respondent features and model performance further.

We fit three separate logistic regression models to test for predictors of correct emotion classification with three multimodal models whose performance is shown in Table 4. CLIP trained on DONATION and CREATION, and minicpm-v as a zero-shot approach. Our dependent variable is a binary variable indicating whether the model predicted the emotion in a given test set post correctly on a particular run. To account for non-independence between predictions for the same post from each of the five runs, we use clustered standard errors.

Independent variables are text length, image style, and text–image relation variables (cf. RQ1), emotion appraisals and event and emotion duration and intensity (RQ2), and participant sociodemographics (RQ3). Numeric and ordinal variables are scaled to [0; 1]. For categorical variables, we

<table>
  <tr>
    <th rowspan="2">Test F<sub>1</sub></th>
    <th rowspan="2">Mod.</th>
    <th colspan="3">Training</th>
  </tr>
  <tr>
    <th>DONATION</th>
    <th>CREATION</th>
    <th>Zero-shot</th>
  </tr>
  <tr>
    <td rowspan="3">Creatin</td>
    <td>V</td>
    <td>.16</td>
    <td>.18</td>
    <td>.24<sup>1</sup></td>
  </tr>
  <tr>
    <td>T</td>
    <td>.49</td>
    <td>.58</td>
    <td>.61<sup>1</sup></td>
  </tr>
  <tr>
    <td>T+V</td>
    <td>.60</td>
    <td>.62</td>
    <td>.56<sup>2</sup></td>
  </tr>
  <tr>
    <td rowspan="3">Donati</td>
    <td>V</td>
    <td>.19</td>
    <td>.18</td>
    <td>.19<sup>3</sup></td>
  </tr>
  <tr>
    <td>T</td>
    <td>.41</td>
    <td>.42</td>
    <td>.45<sup>1</sup></td>
  </tr>
  <tr>
    <td>T+V</td>
    <td>.38</td>
    <td>.40</td>
    <td>.43<sup>2</sup></td>
  </tr>
</table>

Table 4: Performance of models predicting emotion in multimodal social media posts using text alone (T), image alone (V), and both modalities combined (T+V). We report macro F<sub>1</sub> scores over 5 runs. Zero-shot models are chosen according to the best individual performance on development data. 1llama3.2-vision. 2minicpm-v. 3llava-llama3.

11RECENT is not included in the test set for reasons detailed in the Limitations Section.
12Three other zero-shot models were also tested, but returned results that were either worse in all instances or unparseable. See Appendix A.3 for details on these models.
13Full results including precision, recall, and F<sub>1</sub>-scores for individual emotions are in Appendix A.4, Tables 9 and 10.
Figure 9: Influence of post and respondent features on T+V model accuracy. Arrows represent predicted change in the probability of a correct post classification when the indicated variable is changed from its reference value (categorical variables) or from its minimum to its maximum (numeric variables). Only effects significant at the p<.05 level or better are shown.

choose reference levels (which serve as our baseline for each variable) following Johfre and Freese (2021).14 We control for the study the post is from and the emotion it shows. We omit 27 posts for which at least one independent variable is missing.

Figure 9 shows the effects of the statistically significant variables on the probability that a post will be classified correctly.15 Arrows begin at the probability of correct classification indicated by the model intercept: that is, when all independent variables are either 0 (numeric/ordinal) or at their reference values (categorical). Arrow heads indicate the change in this probability when the indicated independent variable is changed, either to its maximum value (numeric/ordinal) or to the indicated category (categorical).

Minicpm-v is less likely to predict emotion accurately when authors report the image is necessary to understand the post text. Figure 6 shows that CREATION posts have images that are less necessary for text understanding. Together, these two results suggest performance on CREATION data may be better in part because CREATION posts have images that are less integral to text interpretation.

Minicpm-v accuracy is also lower for posts authored by people with advanced degrees compared to those who have not gone to college. The CLIP model trained on DONATION is more likely to predict emotion correctly for posts authored by men. Notably, these effects are net of the effects of emotion and all other variables in the regression. These sociodemographic effects underscore the importance of being attentive to sample composition.

Consistent with our results above, Figure 9 shows that some emotions, and particularly Joy, are predicted more accurately than others. CLIP is less likely to accurately predict emotions for DONATION posts as compared to CREATION posts. That this effect is significant despite the presence of the other independent and control variables indicates there are further study differences that are not accounted for here.

We did not observe statistically significant impacts on classification outcomes for other variables not included in Figure 9. We note that our test set size is relatively small and effects are therefore conservative. The relationship between post/author features and model performance is an important avenue for future research.

5 Conclusion

In this paper, we compared methods for collecting study-created and genuine multimodal social media posts labeled by their authors for emotion. Our work is the first to directly compare these methods of collecting author-labeled corpora, and the first multimodal social media corpus labeled by its authors for emotion.

Our results show that CREATION posts are different in content and style and represent more prototypical events than genuine posts collected in DONATION and RECENT. RECENT leads to a more realistic emotion distribution, but does not evenly represent all emotions, which is a challenge for model development. More participants are comfortable participating in CREATION, suggesting that this corpus may better represent social media users. We note that all presented approaches may be easily scaled up to large quantities of posts. Corpus sizes are only limited by available participants on study platforms such as Prolific, and, for genuine data strategies, the amount of data they can provide.

Despite content differences, CREATION training data leads to models that perform on par with models trained on DONATION data. Modeling results on DONATION test data show worse – and likely more realistic – performance than on the CREATION test data. Therefore, we suggest that future studies consider using a strategy similar to DONATION for developing test sets and CREATION to collect corpora for model development as needed for model optimization.

14Refer to Appendix 11 for additional details.
15Full results are reported in Appendix Table 11.
Acknowledgments

We thank all ARR reviewers for their thorough reviews and valuable suggestions. We also thank our study participants for contributing to our work. This work has been supported by the Deutsche Forschungsgesellschaft (DFG) in the project “User’s Choice of Images and Text to Express Emotions in Twitter and Reddit” (ITEM, Project KL 2869/11-1, No. 513384754).

Limitations

We took precautions to allow for the best comparability of the data that we obtained by our studies, but some pragmatic decisions were required in the design. Most importantly, the studies were performed sequentially, each with multiple phases. As such, the time periods in which posts were collected differed and could potentially affect the content of posts. For example, there are more posts about the US presidential assassination attempt in the study which occurred days after the event than in studies which occured weeks later. We do however note that we do not find evidence in our analysis for such differences that would influence the conclusions of our work. In all three conditions there were posts on a variety of topics, containing a variety of images – posts about current events are not a dominant portion of our dataset. The image content analysis and our qualitative review of posts supports. The requirement in donation and creation to find or create posts about specific emotions likely pushed respondents to reach further back in their feeds, mitigating recency effects to some extent. For example, participants submitted posts related to COVID lockdowns which occurred 3–4 years prior to data collection. That said, we acknowledge it as a limitation, particularly in recent, and we encourage future collections of this type to bear it in mind and spread out their data collection if possible, as we did with recent.

Another limitation may be that we did not control for the (hypothetical or real) social media platform. Posts from different social media platforms may differ in content and style and may therefore not be directly comparable. While we do ask for the platform a post is from, we do not control for this. We do a short analysis of the distribution of platforms in our collected data in Appendix A.2.3. We consider platform effects to be an important direction for future work.

As mentioned in Section 3.2 and Section 4.4, for the RECENT study we asked participants to label their posts for all emotions they experienced in response to the event that inspired their post. While we do ask for emotion intensity ratings, in the case of ties we did not ask them to select a primary emotion. This is a limitation of our work, as it prevents us from using RECENT posts in our modeling analysis, as in the case of ties we cannot assign a single emotion label to the post, making them incompatible with posts from CREATION and DONATION. We suggest future work, for all collection strategies, both ask participants to label all emotions and intensities and to select a primary emotion in the case of ties.

The training and testing of models is limited by the size of our dataset. These experiments are designed to inform us the best methods to collect data, and as such we will use the findings to expand the dataset, allowing us to confirm our results in future work.

The focus of our modeling analysis is to compare training and test sets. As such, the pretrained models we use are selected based on their established performance and reliability, rather than seeking state-of-the-art performance.

Furthermore, for the zero-shot models, we chose to only use models which we could run locally for two reasons: (1) Reproducibility; changes to models such as GPT-4o are out of our control. There is no guarantee that we or other researchers will have access to the exact model used in our experiments. Furthermore, there is no guarantee that changes to the model will be disclosed to users. (2) Costs; When budgeting our research we prioritize increasing data collection efforts over using LLM API services. Especially given the above explanation about our model selection choices.

Ethical Considerations

This study was approved by the ethics review board at the University of Bamberg. In all data acquisition efforts, all participants have been informed about the collection procedure and the use of the data. Nevertheless, we would like to reflect on various potential challenges in this work.

While the participants have been informed about the use of the data, it may sometimes be the case that the content of a post does comprise anonymity, and the study participant may not be aware of the potential impact. This is not an issue with the study-created data, but may be a challenge in the RECENT
and DONATION data. Further, the collected data may contain information about other people not actively participating in the study. Because of these two challenges, we decided to only share the RECENT and DONATION data for research purposes upon request.

We use a Creative Commons licensed image database as to best avoid copyright issues. Copyright regulations vary widely by country. In some countries, fair use rules allow for using images protected by copyright for the purposes of academic research. However, in others research use of copyrighted images may be a copyright infraction. Given the international nature of emotion recognition research, we wish to prioritize data collection methods which produce data that can be used in as many legal contexts as possible. However, teams that wish to use image data must consider their own local context when designing data collections and using available corpora.

VS Code Copilot was used to assist in the writing of the code for the data analysis. Copilot was only used for debugging, documentation, refactoring, and code completion.

References

Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France. European Language Resources Association.

Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802–814. Association for Computational Linguistics.

Laura Boeschoten, Jef Ausloos, Judith E. Möller, Theo Araujo, and Daniel L. Oberski. 2022. A framework for privacy preserving digital trace data collection through data donation. Computational Communication Research, 4(2):388–423.

Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeanette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359.

Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception. IEEE Transactions on Affective Computing, 8(1):67–80.

Pasquale Capuozzo, Ivano Lauriola, Carlo Strapparava, Fabio Aiolli, and Giuseppe Sartori. 2020. DecOp: A multilingual and multi-domain corpus for detecting deception in typed text. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1423–1430, Marseille, France. European Language Resources Association.

Thijs C. Carrière, Laura Boeschoten, Bella Struminiskaya, Heleen L. Janssen, Niek C. De Schipper, and Theo Araujo. 2024. Best practices for studies using digital data donation. Quality & Quantity.

Junhan Chen, Yumin Yan, and John Leach. 2022. Are emotion-expressing messages more shared on social media? A meta-analytic review. Review of Communication Research, 10: -.

Wingyan Chung and Daniel Zeng. 2020. Dissecting emotion and user influence in social media communities: An interaction modeling approach. Information & Management, 57(1):103108. Big data and business analytics: A research agenda for realizing business value.

Irina Degtiar and Sherri Rose. 2023. A review of generalizability and transportability. Annual Review of Statistics and Its Application, 10(Volume 10, 2023):501–524.

Daantje Derks, Agneta H. Fischer, and Arjan E.R. Bos. 2008. The role of emotion in computer-mediated communication: A review. Computers in Human Behavior, 24(3):766–785. Instructional Support for Enhancing Students’ Information Problem Solving Ability.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.

Aparna Elangovan, Jiayuan He, Yuan Li, and Karin Verspoor. 2024. Principles from clinical research for NLP model generalization. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2293–2309, Mexico City, Mexico. Association for Computational Linguistics.

Alejandra Gomez Ortega, Jacky Bourgeois, Wiebke Toussaint Huttri, and Gerd Kortuem. 2023. Beyond data transactions: A framework for meaningfully informed data donation. AI & Society.

Anurag Illendula and Amit Sheth. 2019. Multimodal emotion classification. In Companion Proceedings of
The 2019 World Wide Web Conference, WWW ’19, page 439–449, New York, NY, USA. Association for Computing Machinery.

Sasha Shen Johfre and Jeremy Freese. 2021. Reconsidering the reference category. Sociological Methodology, 51(2):253–269.

Tomoyuki Kajiwara, Chenhui Chu, Noriko Take-mura, Yuta Nakashima, and Hajime Nagahara. 2021. WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2095–2104. Association for Computational Linguistics.

Elizabeth A. Kensinger. 2009. Remembering the Details: Effects of Emotion. Emotion Review, 1(2):99–113.

Florian Keusch, Paulina K. Pankowska, Alexandru Cer-nat, and Ruben L. Bach. 2024. Do You Have Two Minutes to Talk about Your Data? Willingness to Participate and Nonparticipation Bias in Facebook Data Donation. Field Methods, 36(4):279–293.

Yiyi Li and Ying Xie. 2020. Is a picture worth a thousand words? an empirical study of image content and social media engagement. Journal of Marketing Research, 57(1):1–19.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Cite arxiv:1907.11692.

E. Paige Lloyd, Jason C. Deska, Kurt Hugenberg, Allen R. McConnell, Brandon T. Humphrey, and Jonathan W. Kunstman. 2019. Miami University deception detection database. Behavior Research Methods, 51(1):429–439.

Anthony Lyons and Yoshihisa Kashima. 2003. How Are Stereotypes Maintained Through Communication? The Influence of Stereotype Sharedness. Journal of Personality and Social Psychology, 85(6):989–1005.

Anthony Lyons and Yoshihisa Kashima. 2006. Maintaining stereotypes in communication: Investigating memory biases and coherence-seeking in storytelling. Asian Journal of Social Psychology, 9(1):59–71.

Saif Mohammad. 2012. #emotional tweets. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 246–255, Montréal, Canada. Association for Computational Linguistics.

Jonathan Mummolo and Erik Peterson. 2019. Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113(2):517–529.

Tsubasa Nakagawa, Shunsuke Kitada, and Hitoshi Iy-atomi. 2022. Expressions Causing Differences in Emotion Recognition in Social Networking Service Documents. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 4349–4353. ACM.

Austin Lee Nichols and Jon K. Maner. 2008. The good-subject effect: Investigating participant demand characteristics. The Journal of General Psychology, 135(2):151–166. PMID: 18507315.

Silviu Oprea and Walid Magdy. 2020. iSarcasm: A Dataset of Intended Sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289. Association for Computational Linguistics.

Myle Ott, Claire Cardie, and Jeffrey T. Hancock. 2013. Negative deceptive opinion spam. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 497–501, Atlanta, Georgia. Association for Computational Linguistics.

Nico Pfiffner, Pim Witlox, and Thomas N. Friemel. 2024. Data Donation Module: A Web Application for Collecting and Enriching Data Donations. Computational Communication Research, 6(2):1.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.

Afsaneh Razi, Ashwaq Alsoubai, Seunghyun Kim, Nurun Naher, Shiza Ali, Gianluca Stringhini, Mumun De Choudhury, and Pamela J. Wisniewski. 2022. Instagram data donation: A case study on collecting ecologically valid social media data for the purpose of adolescent online risk detection. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, New York, NY, USA. Association for Computing Machinery.

Matthew Rhodes-Purdy, Rachel Navarre, and Stephen M Utych. 2021. Measuring simultaneous emotions: Existing problems and a new way forward. Journal of Experimental Political Science, 8(1):1–14.

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912. Online. Association for Computational Linguistics.

Andrea Scarantino. 2016. The philosophy of emotions and its impact on affective science. Handbook of emotions, 4:3–48.

Klaus Scherer and Harald Wallbott. 1997. The ISEAR questionnaire and codebook. Geneva Emotion Research Group.

Klaus R. Scherer. 2005. What are emotions? and how can they be measured? Social Science Information, 44(4):695–729.

Marco Antonio Stranisci, Simona Frenda, Eleonora Ceccaldi, Valerio Basile, Rossana Damiano, and Viviana Patti. 2022. APPReddit: a corpus of Reddit posts annotated for appraisal. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3809–3818, Marseille, France. European Language Resources Association.

Enrica Troiano, Sofie Labat, Marco Antonio Stranisci, Rossana Damiano, Viviana Patti, and Roman Klinger. 2024. Dealing with controversy: An emotion and coping strategy corpus based on role playing. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1634–1658, Miami, Florida, USA. Association for Computational Linguistics.

Enrica Troiano, Laura Oberländer, and Roman Klinger. 2023. Dimensional modeling of emotions in text with appraisal theories: Corpus creation, annotation reliability, and prediction. Computational Linguistics, 49(1).

Enrica Troiano, Sebastian Padó, and Roman Klinger. 2019. Crowdsourcing and validating event-focused emotion corpora for German and English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4005–4011, Florence, Italy. Association for Computational Linguistics.

Irene I. van Driel, Anastasia Giachanou, J. Loes Pouwels, Laura Boeschoten, Ine Beyens, and Patti M. Valkenburg. 2022. Promises and pitfalls of social media data donations. Communication Methods and Measures, 16(4):266–282.

Clara Vania, Ruijie Chen, and Samuel R. Bowman. 2020. Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 672–686. Association for Computational Linguistics.

Aswathy Velutharambath, Amelie Wührl, and Roman Klinger. 2024. Can factual statements be deceptive? the DeFaBel corpus of belief-based deception. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2708–2723, Torino, Italia. ELRA and ICCL.

Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, and Yue Zhang. 2023. Out-of-distribution generalization in natural language processing: Past, present, and future. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4533–4559, Singapore. Association for Computational Linguistics.
A Appendix

A.1 Survey Details

We provide the detailed questions on the participant information in Table 5 and the details of the participant information collected in Table 6. We allow participants to participate in each emotion/data collection strategy combination once. We ask them to only complete this information the first time. If they provide it more than once anyways, we use their most recent data. For the small number of participants who did not complete the study demographic information, we impute it using demographic data provided by Prolific.

<table>
  <tr>
    <th>Label</th>
    <th>Question Text</th>
    <th>Options</th>
  </tr>
  <tr>
    <th colspan="3">Participant information</th>
  </tr>
  <tr>
    <td>Age</td>
    <td>How old are you?</td>
    <td>\( \geq 18 \)</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>With which gender(s) do you identify? [multiple]</td>
    <td>Woman, Man, Nonbinary, Transgender, Other [write-in]</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>What is the highest level of education you completed?</td>
    <td>No formal qualifications, Secondary education, High school, Undergraduate degree (BA/BSc/other), Graduate degree (MA/MSc/Mphil/other), Doctorate degree (PhD/other)</td>
  </tr>
  <tr>
    <td>Ethnicity</td>
    <td>With which of the following ethnic groups do you identify the most? [multiple possible]</td>
    <td>Australian/New Zealander, North Asian, South Asian, East Asian, Middle Eastern, European, African, North American, South American, Hispanic/Latino, Indigenous, Other [write in]</td>
  </tr>
  <tr>
    <td>Social Media Use</td>
    <td>Approximately how often do you use social media (browse or participate)?</td>
    <td>Every day, 4-6 days a week, 2-3 days a week, Once per week, Occasionally, but less than once per week, Never</td>
  </tr>
  <tr>
    <td>Social Media Post</td>
    <td>Approximately how often do you post on social media?</td>
    <td>Every day, 4-6 days a week, 2-3 days a week, Once per week, Occasionally, but less than once per week, Never</td>
  </tr>
  <tr>
    <td>Preferred platform</td>
    <td>What is your preferred social media site?</td>
    <td>X (Twitter), Facebook, Instagram, Reddit, Tiktok, LinkedIn, Other [write-in]</td>
  </tr>
</table>

Table 5: Wording and response options for participant information.
<table>
  <tr>
    <th></th>
    <th>Creation</th>
    <th>Donation</th>
    <th>Recent</th>
  </tr>
  <tr>
    <th colspan="4">Gender identity</th>
  </tr>
  <tr>
    <td>Man</td>
    <td>109 (0.46)</td>
    <td>119 (0.5)</td>
    <td>22 (0.56)</td>
  </tr>
  <tr>
    <td>Woman</td>
    <td>126 (0.53)</td>
    <td>114 (0.48)</td>
    <td>17 (0.44)</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>2 (0.01)</td>
    <td>3 (0.01)</td>
    <td>0</td>
  </tr>
  <tr>
    <th colspan="4">Ethnicity *</th>
  </tr>
  <tr>
    <td>European</td>
    <td>196 (0.84)</td>
    <td>178 (0.76)</td>
    <td>29 (0.72)</td>
  </tr>
  <tr>
    <td>Non-European or multiethnic</td>
    <td>38 (0.16)</td>
    <td>57 (0.24)</td>
    <td>11 (0.28)</td>
  </tr>
  <tr>
    <th colspan="4">Education</th>
  </tr>
  <tr>
    <td>No college degree</td>
    <td>81 (0.34)</td>
    <td>73 (0.31)</td>
    <td>12 (0.32)</td>
  </tr>
  <tr>
    <td>Bachelor’s degree</td>
    <td>97 (0.4)</td>
    <td>99 (0.42)</td>
    <td>15 (0.39)</td>
  </tr>
  <tr>
    <td>Master’s degree or higher</td>
    <td>63 (0.26)</td>
    <td>62 (0.26)</td>
    <td>11 (0.29)</td>
  </tr>
  <tr>
    <th colspan="4">Employment status</th>
  </tr>
  <tr>
    <td>Full-Time</td>
    <td>136 (0.59)</td>
    <td>140 (0.6)</td>
    <td>27 (0.68)</td>
  </tr>
  <tr>
    <td>Part-Time</td>
    <td>40 (0.17)</td>
    <td>41 (0.18)</td>
    <td>7 (0.17)</td>
  </tr>
  <tr>
    <td>Not employed (unemployed, homemaker, retired, disabled)</td>
    <td>45 (0.19)</td>
    <td>40 (0.17)</td>
    <td>5 (0.12)</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>10 (0.04)</td>
    <td>12 (0.05)</td>
    <td>1 (0.03)</td>
  </tr>
  <tr>
    <th colspan="4">Student *</th>
  </tr>
  <tr>
    <td>Yes</td>
    <td>32 (0.14)</td>
    <td>60 (0.26)</td>
    <td>9 (0.22)</td>
  </tr>
  <tr>
    <td>No</td>
    <td>199 (0.86)</td>
    <td>175 (0.74)</td>
    <td>31 (0.78)</td>
  </tr>
  <tr>
    <th colspan="4">Social media use</th>
  </tr>
  <tr>
    <td>Every day</td>
    <td>203 (0.85)</td>
    <td>202 (0.86)</td>
    <td>35 (0.88)</td>
  </tr>
  <tr>
    <td>2–6 days a week</td>
    <td>30 (0.13)</td>
    <td>29 (0.12)</td>
    <td>3 (0.07)</td>
  </tr>
  <tr>
    <td>Once per week or less</td>
    <td>6 (0.03)</td>
    <td>5 (0.02)</td>
    <td>2 (0.05)</td>
  </tr>
  <tr>
    <th colspan="4">Social media post</th>
  </tr>
  <tr>
    <td>Every day</td>
    <td>19 (0.08)</td>
    <td>32 (0.14)</td>
    <td>2 (0.05)</td>
  </tr>
  <tr>
    <td>2–6 days a week</td>
    <td>65 (0.27)</td>
    <td>63 (0.27)</td>
    <td>15 (0.38)</td>
  </tr>
  <tr>
    <td>Once per week or less</td>
    <td>157 (0.65)</td>
    <td>141 (0.6)</td>
    <td>23 (0.58)</td>
  </tr>
  <tr>
    <th colspan="4">Preferred platform</th>
  </tr>
  <tr>
    <td>Facebook</td>
    <td>86 (0.35)</td>
    <td>78 (0.33)</td>
    <td>16 (0.4)</td>
  </tr>
  <tr>
    <td>Instagram</td>
    <td>79 (0.32)</td>
    <td>78 (0.33)</td>
    <td>14 (0.35)</td>
  </tr>
  <tr>
    <td>X (Twitter)</td>
    <td>37 (0.15)</td>
    <td>52 (0.22)</td>
    <td>7 (0.17)</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>42 (0.17)</td>
    <td>30 (0.13)</td>
    <td>3 (0.07)</td>
  </tr>
  <tr>
    <th colspan="4">Mean age (std. dev.) *</th>
  </tr>
  <tr>
    <td></td>
    <td>38.4 (12.3)</td>
    <td>33.2 (10.7)</td>
    <td>36 (11.8)</td>
  </tr>
</table>

Table 6: Sample composition of the three studies. For categorical variables, values are counts and proportions are shown in parentheses. For age, values are sample means and standard deviations are shown in parentheses. Proportions sum to less than one due to small numbers of participants missing information on given variables. Starred categories are those for which a \( \chi^2 \) test (categorical variables) or one-way ANOVA (age) shows statistically significant study differences at at least the p<0.05 level.

A.2 Analysis Details

A.2.1 Image Label and Content Analysis

Image Labels: To investigate differences in image style, we manually label each image as a meme, screenshot, graphic, professional photo, personal photo, or other. We inductively developed this list from our observations of our data. The list of labels is hierarchical, such that if the image is a meme and a personal photo, it will be labeled as meme. Examples of each can be seen in Figure 10. Memes use an established meme format edited to the authors’ specific use. Screenshots are images of the authors’ computer or smart phone screen. These includes shots of programs, websites, news stories, other social media posts, etc. Graphic refers to any image drawn, painted, or otherwise created and is not a photograph, but includes photographs which have been edited with graphic overlays. Professional photos are photographs which are clearly a stock photo or created for professional purposes such as news, sports media, or advertisements. All other photographs are labeled as personal photo. Images which do not fit into any of these are labeled as other. This includes images with inspirational quotes and still shots of movies or TV shows.
(a) Meme
(b) Screenshot
(c) Graphic
(d) Professional Photo
(e) Personal photo
(f) Other

Figure 10: Examples of image labels
Image Content: To analyze image content we use the following prompt with GPT-4o to extract a list of words describing the objects and scenes in each image:

“Please analyze the attached image and list the contents of the image using 1-2 word phrases. If the image contains text, only list ‘text’, do not repeat the text contained in the image. Output the list one content per line, with no other information but the 1-2 word description.”

We use GPT-4o for its combination of ease of use, quality of output, and low cost. We validate the results by manually checking the output for 100 randomly selected images. Of 673 content words only 10 were false positives, 98.5% precision. The top 50 most common content words for each study can be seen in Table 7.

<table>
  <tr>
    <th>Creation</th>
    <th>Donation</th>
    <th>Recent</th>
  </tr>
  <tr><td>text</td><td>496</td><td>text</td><td>627</td><td>text</td><td>115</td></tr>
  <tr><td>background</td><td>140</td><td>tree</td><td>125</td><td>tree</td><td>27</td></tr>
  <tr><td>tree</td><td>137</td><td>background</td><td>108</td><td>woman</td><td>26</td></tr>
  <tr><td>sky</td><td>128</td><td>woman</td><td>105</td><td>man</td><td>23</td></tr>
  <tr><td>grass</td><td>127</td><td>sky</td><td>98</td><td>sky</td><td>23</td></tr>
  <tr><td>building</td><td>100</td><td>person</td><td>94</td><td>background</td><td>21</td></tr>
  <tr><td>water</td><td>97</td><td>people</td><td>93</td><td>person</td><td>20</td></tr>
  <tr><td>people</td><td>94</td><td>grass</td><td>92</td><td>grass</td><td>18</td></tr>
  <tr><td>woman</td><td>77</td><td>man</td><td>89</td><td>smile</td><td>16</td></tr>
  <tr><td>car</td><td>77</td><td>smile</td><td>84</td><td>building</td><td>16</td></tr>
  <tr><td>light</td><td>72</td><td>light</td><td>74</td><td>people</td><td>15</td></tr>
  <tr><td>blue</td><td>71</td><td>building</td><td>72</td><td>dog</td><td>14</td></tr>
  <tr><td>cloud</td><td>71</td><td>blue</td><td>63</td><td>sunglasses</td><td>14</td></tr>
  <tr><td>black</td><td>69</td><td>water</td><td>62</td><td>shirt</td><td>14</td></tr>
  <tr><td>wall</td><td>67</td><td>table</td><td>59</td><td>wall</td><td>13</td></tr>
  <tr><td>man</td><td>64</td><td>crowd</td><td>53</td><td>flower</td><td>13</td></tr>
  <tr><td>green</td><td>64</td><td>red</td><td>53</td><td>water</td><td>13</td></tr>
  <tr><td>table</td><td>61</td><td>number</td><td>52</td><td>table</td><td>12</td></tr>
  <tr><td>red</td><td>61</td><td>car</td><td>50</td><td>blue</td><td>12</td></tr>
  <tr><td>smile</td><td>54</td><td>wall</td><td>49</td><td>chair</td><td>11</td></tr>
  <tr><td>person</td><td>53</td><td>cloud</td><td>47</td><td>cloud</td><td>11</td></tr>
  <tr><td>chair</td><td>52</td><td>flower</td><td>46</td><td>sofa</td><td>9</td></tr>
  <tr><td>street</td><td>50</td><td>child</td><td>42</td><td>crowd</td><td>9</td></tr>
  <tr><td>White</td><td>48</td><td>clothing</td><td>42</td><td>light</td><td>9</td></tr>
  <tr><td>sign</td><td>46</td><td>shirt</td><td>39</td><td>red</td><td>9</td></tr>
  <tr><td>rock</td><td>44</td><td>black</td><td>39</td><td>cat</td><td>8</td></tr>
  <tr><td>floor</td><td>43</td><td>hand</td><td>38</td><td>floor</td><td>8</td></tr>
  <tr><td>hand</td><td>43</td><td>sign</td><td>37</td><td>clothing</td><td>8</td></tr>
  <tr><td>flower</td><td>37</td><td>street</td><td>37</td><td>pavement</td><td>7</td></tr>
  <tr><td>child</td><td>37</td><td>floor</td><td>37</td><td>mountain</td><td>7</td></tr>
  <tr><td>face</td><td>36</td><td>green</td><td>36</td><td>male_child</td><td>6</td></tr>
  <tr><td>hair</td><td>36</td><td>expression</td><td>36</td><td>hand</td><td>6</td></tr>
  <tr><td>expression</td><td>35</td><td>drink</td><td>36</td><td>rock</td><td>6</td></tr>
  <tr><td>flag</td><td>35</td><td>dog</td><td>36</td><td>hat</td><td>6</td></tr>
  <tr><td>dog</td><td>34</td><td>hair</td><td>34</td><td>sunset</td><td>6</td></tr>
  <tr><td>shirt</td><td>33</td><td>rock</td><td>34</td><td>number</td><td>6</td></tr>
  <tr><td>leaf</td><td>33</td><td>chair</td><td>32</td><td>drink</td><td>6</td></tr>
  <tr><td>plant</td><td>33</td><td>White</td><td>30</td><td>orange</td><td>5</td></tr>
  <tr><td>yellow</td><td>32</td><td>decoration</td><td>29</td><td>child</td><td>5</td></tr>
  <tr><td>clothing</td><td>32</td><td>glass</td><td>26</td><td>spectacles</td><td>5</td></tr>
  <tr><td>drink</td><td>31</td><td>smoke</td><td>25</td><td>black</td><td>5</td></tr>
  <tr><td>eyes</td><td>30</td><td>bag</td><td>25</td><td>top</td><td>5</td></tr>
  <tr><td>hat</td><td>29</td><td>food</td><td>25</td><td>hair</td><td>5</td></tr>
  <tr><td>suit</td><td>29</td><td>dress</td><td>25</td><td>two</td><td>5</td></tr>
  <tr><td>hands</td><td>28</td><td>blanket</td><td>25</td><td>outdoor</td><td>5</td></tr>
  <tr><td>window</td><td>28</td><td>window</td><td>25</td><td>window</td><td>5</td></tr>
  <tr><td>field</td><td>27</td><td>plant</td><td>24</td><td>White</td><td>5</td></tr>
  <tr><td>beach</td><td>27</td><td>player</td><td>24</td><td>beach</td><td>5</td></tr>
  <tr><td>crowd</td><td>26</td><td>orange</td><td>24</td><td>colorful</td><td>5</td></tr>
  <tr><td>bag</td><td>25</td><td>suit</td><td>24</td><td>shadow</td><td>5</td></tr>
</table>

Table 7: Top 50 most common image content words for each approach, generated by GPT-4o.
My Dads in the UK!! I didn't expect when I answered the door to my sister that my dad was gonna be surprising me too. He was supposed to be coming in may but they both decided to surprise me a month later. I'm so happy and can't wait to catch up on all the lost time.

(a) CREATION post labeled as surprise.

Can't wait for this to inevitably not arrive! #evri

(b) RECENT post labeled as joy.

Figure 11: Comparison of posts submitted for CREATION and RECENT. CREATION gives a long detailed explanation of the event, including descriptions of their emotions, while RECENT uses only a single emoji to describe a similar event, spending time with their dad.

Why so many scammers online?

(a) CREATION post labeled as anger.

Can't wait for this to inevitably not arrive! #evri

(b) DONATION post labeled as anger.

Figure 12: Comparison of posts submitted for CREATION and DONATION. CREATION uses a stock photo, while DONATION demonstrates a reaction post in which the author uses a screenshot of an event they experienced to illustrate their emotion.

A.2.2 Examples of Posts

Here we look at specific examples of posts which highlight the differences described in Section 4.

Figure 11 shows how length and detail of description vary by collection strategy. In post (a) from CREATION you can see how the author explicitly describes both the event and the emotion they experienced. In post (b) from RECENT the author uses only a single emoji to describe a similar event, spending time with their Dad. The event can only be gleaned from the additional annotation in which the author describes it as “I felt happy because I was spending time with my dad”.

Figure 12 highlights how image types can affect posts: post (a) uses a stock photo to stand in as a representation of the event which triggered anger in them. The specific event is not detailed and the generic image does not help give any more clues to the reader. The post from DONATION, however, uses a screenshot that directly references the event which triggered the author’ anger. These reaction posts are not possible in CREATION, since the participants only have access to the image database, and not their own personal images. This is also another example of CREATION posts being about general events, where DONATION tend to be about specific events.

Figure 13 illustrates how posts can rely more on the text or the image to convey emotion. In post (a), a CREATION post, the image conveys almost no emotion, while the text explicitly describes the author’s
I really find interviews scary. I have one coming up and I genuinely shake when I think about it.

this is not a movie scene

(a) CREATION post labeled as fear. (b) DONATION post labeled as fear.

Figure 13: Comparison of posts submitted for CREATION and DONATION. CREATION’s image conveys no emotion and relies entirely, on the text to convey emotion, while DONATION’s text is emotion neutral and relies on the image to convey the experience of fear. experienced to illustrate their emotion.

Rest in peace, nan. 1936-2022

Mindfullness

(a) CREATION post labeled as sadness. (b) DONATION post labeled as sadness.

Figure 14: Comparison of posts submitted for CREATION and DONATION. CREATION is a prototypical event for sadness, while DONATION is a more general sense of sadness.

emotion. Post (b), a DONATION post, on the other hand, conveys no emotion in the text, relying on the image to convey the author’s emotion of fear.

In Figure 14 we can see that post (a) was inspired by a prototypical event for sadness, the death of a loved one. Post (b) is less obvious what event inspired the author to post, or indeed which emotion they are expressing. The author describes the event as "I was feeling upset that day and so was on a walk around a local nature reserve to unwind". In this case the event is more difficult to define.

A.2.3 Platform Distribution between Approaches
Figure 15 shows the distribution of posts between social media platforms. Participants in DONATION and RECENT were asked which platform their donated post was originally posted on. Participants in CREATION were asked which platform(s) they would have posted their study-created post on.
Figure 15: Distribution of post platforms under the CREATION, DONATION, and RECENT approaches. As multiple responses were allowed in CREATION, proportions are normalized such that they sum to one.

A.3 Model Details
All models are trained using four NVIDIA L40 GPUs. Each supervised model is fine-tuned five times, using the exact same setup and environment. For zero-shot models, every prompt is run five times per model. The average and range of scores is reported. All code can be found in the supplementary materials.16

A.3.1 Fine-tuned Models
Text-only: We use RoBERTa-base pretrained model for both tokenizer and model, via the transformers Python package.17 All text-only models are trained using 10 epochs, 1e-5 training rate, 16 batch size, 0.01 weight decay, and early stopping.
Vision-only: We fine-tune the vit-base-patch16-224 model using a combination of transformers and torchvision Python packages.18 Images are preprocessed by resizing them to 224x224. All image-only models are trained using 5 epochs, 5e-5 learning rate, 32 batch size, cross entropy loss, and early stopping.
Text+Vision: We fine-tune the clip-vit-base-patch32 model using a combination of transformers and torchvision Python packages.19 Both images and text are encoded via the CLIP processor, and then fused using a simple concatenation method. All text+vision models are trained using 10 epochs, 1e-5 learning rate, batch size 8, cross entropy loss, and early stopping.

A.3.2 Zero-shot Models
We use six multimodal models made available through the Ollama Python package20. These are llama3.2-vision21 (11b, ID 085af1da525), minicpm-v22 (8b, v2.6, ID c92bfad01205), llava23 (7b, v1.6, ID 8dd30f6b0cb1), llava-llama324 (8b, ID 44c161b1f465), bakllava25 (7b, ID 3dd68bd4447c), and llava-phi326 (3.8b, ID c7edd7b87593). Bakllava and llava-phi3 returned nearly entirely unparseable results and were hence eliminated from further analysis.

We perform basic cleaning on responses (converting to lowercase, removing punctuation and whitespace, removing the word “emotion”). We continue to prompt the model until we received five valid emotion predictions, with “valid” defined as equalling one of the six provided emotion options after cleaning. We prompt models up to a maximum of 30 times for each post. In the small number of cases in which we did not receive five valid predictions in 30 tries, we count missing responses as incorrect for the purposes of performance statistic calculations.

16https://www.uni-bamberg.de/en/nlproc/projects/item/
17https://huggingface.co/transformers/v2.9.1/model_doc/roberta.html
18https://huggingface.co/google/vit-base-patch16-224
19https://huggingface.co/openai/clip-vit-base-patch32
20https://pypi.org/project/ollama/0.4.6/
21https://ollama.com/library/llama3.2-vision
22https://ollama.com/library/minicpm-v
23https://ollama.com/library/llava
24https://ollama.com/library/llava-llama3
25https://ollama.com/library/bakllava
26https://ollama.com/library/llava-phi3
<table>
  <tr>
    <th>Section</th>
    <th>Text</th>
    <th>Vision</th>
    <th>Text + Vision</th>
  </tr>
  <tr>
    <td>Task Descr.</td>
    <td>Which emotion does the following text from a social media post convey most strongly?</td>
    <td>Which emotion does the following image from a social media post convey most strongly?</td>
    <td>Which emotion does the following text and image from a social media post convey most strongly?</td>
  </tr>
  <tr>
    <td>Labels</td>
    <td>Please choose one of the set [anger, disgust, fear, joy, sadness, surprise].</td>
    <td>Please choose one of the set [anger, disgust, fear, joy, sadness, surprise].</td>
    <td>Please choose one of the set [anger, disgust, fear, joy, sadness, surprise].</td>
  </tr>
  <tr>
    <td>Format Instr.</td>
    <td>Only provide a single word indicating the emotion. Do not provide explanation or analysis. Only provide plain text.</td>
    <td>Only provide a single word indicating the emotion. Do not provide explanation or analysis. Only provide plain text.</td>
    <td>Only provide a single word indicating the emotion. Do not provide explanation or analysis. Only provide plain text.</td>
  </tr>
  <tr>
    <td>Data Input</td>
    <td>Post text: {text}</td>
    <td></td>
    <td>Post text: {text}</td>
  </tr>
  <tr>
    <td>Image</td>
    <td></td>
    <td>this_image.{image type}</td>
    <td>this_image.{image type}</td>
  </tr>
</table>

Table 8: Prompts for text, vision, and text + vision modalities. Variables are typeset in {curly brackets}. Image filenames were changed to a default name before sending to the LLM to avoid the model being able to gain additional information.

A.4 Additional Modeling Results

<table>
  <tr>
    <th rowspan="2">Test</th>
    <th rowspan="2">Creatin</th>
    <th colspan="3">llama3.2-vision</th>
    <th colspan="3">minicpm-v</th>
    <th colspan="3">llava</th>
    <th colspan="3">llava-llama3</th>
  </tr>
  <tr>
    <th>Modality</th>
    <th>F<sub>1</sub></th>
    <th>Range</th>
    <th>Modality</th>
    <th>F<sub>1</sub></th>
    <th>Range</th>
    <th>Modality</th>
    <th>F<sub>1</sub></th>
    <th>Range</th>
    <th>Modality</th>
    <th>F<sub>1</sub></th>
    <th>Range</th>
  </tr>
  <tr>
    <td rowspan="3">Creatin</td>
    <td>Vision</td>
    <td></td>
    <td>.24</td>
    <td>20–30</td>
    <td></td>
    <td>.21</td>
    <td>.20–.24</td>
    <td></td>
    <td>.25</td>
    <td>.21–.31</td>
    <td></td>
    <td>.25</td>
    <td>.21–.33</td>
  </tr>
  <tr>
    <td>Text</td>
    <td></td>
    <td>.61</td>
    <td>.60–.63</td>
    <td></td>
    <td>.58</td>
    <td>.54–.65</td>
    <td></td>
    <td>.55</td>
    <td>.52–.58</td>
    <td></td>
    <td>.49</td>
    <td>.41–.55</td>
  </tr>
  <tr>
    <td>T+V</td>
    <td></td>
    <td>.53</td>
    <td>.50–.57</td>
    <td></td>
    <td>.56</td>
    <td>.52–.60</td>
    <td></td>
    <td>.44</td>
    <td>.42–.48</td>
    <td></td>
    <td>.43</td>
    <td>.37–.49</td>
  </tr>
  <tr>
    <td rowspan="3">Donation</td>
    <td>Vision</td>
    <td></td>
    <td>.21</td>
    <td>.17–.25</td>
    <td></td>
    <td>.25</td>
    <td>.21–.30</td>
    <td></td>
    <td>.23</td>
    <td>.19–.28</td>
    <td></td>
    <td>.19</td>
    <td>.14–.24</td>
  </tr>
  <tr>
    <td>Text</td>
    <td></td>
    <td>.45</td>
    <td>.41–.48</td>
    <td></td>
    <td>.45</td>
    <td>.42–.49</td>
    <td></td>
    <td>.42</td>
    <td>.39–.45</td>
    <td></td>
    <td>.44</td>
    <td>.41–.47</td>
  </tr>
  <tr>
    <td>T+V</td>
    <td></td>
    <td>.39</td>
    <td>.35–.43</td>
    <td></td>
    <td>.43</td>
    <td>.41–.45</td>
    <td></td>
    <td>.33</td>
    <td>.28–.38</td>
    <td></td>
    <td>.26</td>
    <td>.18–.30</td>
  </tr>
</table>

Table 9: Macro F<sub>1</sub> of multimodal zero-shot models for predicting emotion on Creation and Donation posts using post text (T), post image (V), and both text and image (T+V).
<table>
  <tr>
    <th rowspan="2">Modality</th>
    <th rowspan="2">Emotion</th>
    <th colspan="3">CREATION</th>
    <th colspan="3">DONATION</th>
    <th colspan="3">Zero-shot</th>
  </tr>
  <tr>
    <th>F<sub>1</sub></th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>1</sub></th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>1</sub></th>
    <th>P</th>
    <th>R</th>
  </tr>
  <tr>
    <td rowspan="6">Vision</td>
    <td>anger</td>
    <td>.25</td>
    <td>.27</td>
    <td>.21</td>
    <td>.17</td>
    <td>.17</td>
    <td>.14</td>
    <td>.01<sup>1</sup></td>
    <td>.10<sup>1</sup></td>
    <td>.01<sup>1</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.16</td>
    <td>.07</td>
    <td>.16</td>
    <td>.15</td>
    <td>.10</td>
    <td>.16</td>
    <td>.19<sup>1</sup></td>
    <td>.30<sup>1</sup></td>
    <td>.14<sup>1</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.12</td>
    <td>.12</td>
    <td>.12</td>
    <td>.18</td>
    <td>.10</td>
    <td>.25</td>
    <td>.14<sup>1</sup></td>
    <td>.31<sup>1</sup></td>
    <td>.09<sup>1</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.11</td>
    <td>.20</td>
    <td>.12</td>
    <td>.06</td>
    <td>.09</td>
    <td>.05</td>
    <td>.39<sup>1</sup></td>
    <td>.26<sup>1</sup></td>
    <td>.78<sup>1</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.23</td>
    <td>.29</td>
    <td>.25</td>
    <td>.20</td>
    <td>.14</td>
    <td>.19</td>
    <td>.26<sup>1</sup></td>
    <td>.26<sup>1</sup></td>
    <td>.26<sup>1</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.22</td>
    <td>.23</td>
    <td>.24</td>
    <td>.17</td>
    <td>.14</td>
    <td>.17</td>
    <td>.18<sup>1</sup></td>
    <td>.17<sup>1</sup></td>
    <td>.20<sup>1</sup></td>
  </tr>
  <tr>
    <td rowspan="6">Text</td>
    <td>anger</td>
    <td>.46</td>
    <td>.42</td>
    <td>.40</td>
    <td>.47</td>
    <td>.50</td>
    <td>.49</td>
    <td>.65<sup>1</sup></td>
    <td>.60<sup>1</sup></td>
    <td>.71<sup>1</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.57</td>
    <td>.41</td>
    <td>.67</td>
    <td>.45</td>
    <td>.31</td>
    <td>.60</td>
    <td>.55<sup>1</sup></td>
    <td>.76<sup>1</sup></td>
    <td>.43<sup>1</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.50</td>
    <td>.52</td>
    <td>.49</td>
    <td>.34</td>
    <td>.33</td>
    <td>.29</td>
    <td>.59<sup>1</sup></td>
    <td>.89<sup>1</sup></td>
    <td>.44<sup>1</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.73</td>
    <td>.67</td>
    <td>.77</td>
    <td>.70</td>
    <td>.58</td>
    <td>.65</td>
    <td>.76<sup>1</sup></td>
    <td>.63<sup>1</sup></td>
    <td>.95<sup>1</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.61</td>
    <td>.57</td>
    <td>.64</td>
    <td>.57</td>
    <td>.41</td>
    <td>.66</td>
    <td>.57<sup>1</sup></td>
    <td>.44<sup>1</sup></td>
    <td>.83<sup>1</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.57</td>
    <td>.79</td>
    <td>.50</td>
    <td>.28</td>
    <td>.17</td>
    <td>.22</td>
    <td>.20<sup>1</sup></td>
    <td>.59<sup>1</sup></td>
    <td>.12<sup>1</sup></td>
  </tr>
  <tr>
    <td rowspan="6">Text + Vision</td>
    <td>anger</td>
    <td>.55</td>
    <td>.62</td>
    <td>.51</td>
    <td>.50</td>
    <td>.45</td>
    <td>.50</td>
    <td>.58<sup>2</sup></td>
    <td>.51<sup>2</sup></td>
    <td>.70<sup>2</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.55</td>
    <td>.55</td>
    <td>.61</td>
    <td>.53</td>
    <td>.42</td>
    <td>.58</td>
    <td>.53<sup>2</sup></td>
    <td>.66<sup>2</sup></td>
    <td>.44<sup>2</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.57</td>
    <td>.52</td>
    <td>.60</td>
    <td>.56</td>
    <td>.65</td>
    <td>.54</td>
    <td>.56<sup>2</sup></td>
    <td>.75<sup>2</sup></td>
    <td>.45<sup>2</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.79</td>
    <td>.74</td>
    <td>.82</td>
    <td>.77</td>
    <td>.83</td>
    <td>.78</td>
    <td>.71<sup>2</sup></td>
    <td>.56<sup>2</sup></td>
    <td>.97<sup>2</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.65</td>
    <td>.63</td>
    <td>.66</td>
    <td>.62</td>
    <td>.59</td>
    <td>.65</td>
    <td>.60<sup>2</sup></td>
    <td>.54<sup>2</sup></td>
    <td>.68<sup>2</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.59</td>
    <td>.77</td>
    <td>.50</td>
    <td>.61</td>
    <td>.71</td>
    <td>.55</td>
    <td>.20<sup>2</sup></td>
    <td>.39<sup>2</sup></td>
    <td>.13<sup>2</sup></td>
  </tr>
  <tr>
    <td rowspan="6">Test</td>
    <td>Vision</td>
    <td>.13</td>
    <td>.28</td>
    <td>.10</td>
    <td>.22</td>
    <td>.12</td>
    <td>.21</td>
    <td>.34<sup>3</sup></td>
    <td>.34<sup>3</sup></td>
    <td>.34<sup>3</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.14</td>
    <td>.25</td>
    <td>.14</td>
    <td>.16</td>
    <td>.27</td>
    <td>.14</td>
    <td>.03<sup>3</sup></td>
    <td>.08<sup>3</sup></td>
    <td>.02<sup>3</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.19</td>
    <td>.24</td>
    <td>.18</td>
    <td>.23</td>
    <td>.15</td>
    <td>.28</td>
    <td>.07<sup>3</sup></td>
    <td>.10<sup>3</sup></td>
    <td>.06<sup>3</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.11</td>
    <td>.20</td>
    <td>.12</td>
    <td>.03</td>
    <td>.06</td>
    <td>.03</td>
    <td>.36<sup>3</sup></td>
    <td>.23<sup>3</sup></td>
    <td>.92<sup>3</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.21</td>
    <td>.15</td>
    <td>.19</td>
    <td>.26</td>
    <td>.22</td>
    <td>.26</td>
    <td>.00<sup>3</sup></td>
    <td>.00<sup>3</sup></td>
    <td>.00<sup>3</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.27</td>
    <td>.25</td>
    <td>.33</td>
    <td>.19</td>
    <td>.13</td>
    <td>.21</td>
    <td>.10<sup>3</sup></td>
    <td>.22<sup>3</sup></td>
    <td>.06<sup>3</sup></td>
  </tr>
  <tr>
    <td rowspan="6">Donation</td>
    <td>Text</td>
    <td>.36</td>
    <td>.41</td>
    <td>.31</td>
    <td>.47</td>
    <td>.43</td>
    <td>.44</td>
    <td>.54<sup>1</sup></td>
    <td>.57<sup>1</sup></td>
    <td>.52<sup>1</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.48</td>
    <td>.42</td>
    <td>.50</td>
    <td>.34</td>
    <td>.27</td>
    <td>.41</td>
    <td>.30<sup>1</sup></td>
    <td>.44<sup>1</sup></td>
    <td>.22<sup>1</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.35</td>
    <td>.33</td>
    <td>.36</td>
    <td>.18</td>
    <td>.10</td>
    <td>.13</td>
    <td>.40<sup>1</sup></td>
    <td>.68<sup>1</sup></td>
    <td>.29<sup>1</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.55</td>
    <td>.38</td>
    <td>.81</td>
    <td>.61</td>
    <td>.48</td>
    <td>.74</td>
    <td>.53<sup>1</sup></td>
    <td>.40<sup>1</sup></td>
    <td>.79<sup>1</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.43</td>
    <td>.50</td>
    <td>.39</td>
    <td>.45</td>
    <td>.39</td>
    <td>.44</td>
    <td>.39<sup>1</sup></td>
    <td>.30<sup>1</sup></td>
    <td>.54<sup>1</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.20</td>
    <td>.30</td>
    <td>.14</td>
    <td>.33</td>
    <td>.31</td>
    <td>.32</td>
    <td>.27<sup>1</sup></td>
    <td>.53<sup>1</sup></td>
    <td>.18<sup>1</sup></td>
  </tr>
  <tr>
    <td rowspan="6">Text + Vision</td>
    <td>anger</td>
    <td>.34</td>
    <td>.37</td>
    <td>.34</td>
    <td>.33</td>
    <td>.25</td>
    <td>.34</td>
    <td>.51<sup>2</sup></td>
    <td>.48<sup>2</sup></td>
    <td>.54<sup>2</sup></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>.29</td>
    <td>.32</td>
    <td>.27</td>
    <td>.28</td>
    <td>.24</td>
    <td>.28</td>
    <td>.27<sup>2</sup></td>
    <td>.41<sup>2</sup></td>
    <td>.21<sup>2</sup></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>.37</td>
    <td>.36</td>
    <td>.38</td>
    <td>.35</td>
    <td>.44</td>
    <td>.34</td>
    <td>.33<sup>2</sup></td>
    <td>.51<sup>2</sup></td>
    <td>.25<sup>2</sup></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>.57</td>
    <td>.52</td>
    <td>.68</td>
    <td>.57</td>
    <td>.47</td>
    <td>.67</td>
    <td>.50<sup>2</sup></td>
    <td>.35<sup>2</sup></td>
    <td>.87<sup>2</sup></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>.42</td>
    <td>.48</td>
    <td>.42</td>
    <td>.38</td>
    <td>.45</td>
    <td>.38</td>
    <td>.44<sup>2</sup></td>
    <td>.43<sup>2</sup></td>
    <td>.46<sup>2</sup></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>.36</td>
    <td>.38</td>
    <td>.32</td>
    <td>.34</td>
    <td>.32</td>
    <td>.30</td>
    <td>.25<sup>2</sup></td>
    <td>.53<sup>2</sup></td>
    <td>.17<sup>2</sup></td>
  </tr>
</table>

Table 10: Model predictive performance by emotion. Results shown are averaged over 5 runs. Zero-shot models are chosen for each data-test set combination based on overall performance on development data, and are the same here as reported in the main text. 1llama3.2-vision. 2minicpm-v. 3llava-llama3.
Table 11: Coefficients for logistic regressions predicting correct model emotion classification, in log odds units. Numeric and ordinal variables are scaled to range from 0–1. Reference levels – which serve as baselines for each variable – are indicated with hyphens in the table. They are chosen according to Johfre and Freese (2021), and are in general either the lowest category conceptually (in the case of inherently ordered variables like education) or lowest category numerically in terms of their effect on the dependent variable (in the case of unordered variables like emotion). Clustered standard errors (reported in parentheses) are used to account for modeling multiple predictions of the same posts. *p<.1. **p<.05. ***p<.01. **** p<.001.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">CLIP: creation-trained</th>
    <th colspan="2">CLIP: donation-trained</th>
    <th colspan="2">minicpm-v</th>
  </tr>
  <tr>
    <th>Coef</th>
    <th>SE</th>
    <th>Coef</th>
    <th>SE</th>
    <th>Coef</th>
    <th>SE</th>
  </tr>
  <tr>
    <td>(Intercept)</td>
    <td>−1.27</td>
    <td>(0.98)</td>
    <td>−0.49</td>
    <td>(0.94)</td>
    <td>−2.85</td>
    <td>(1.49)+</td>
  </tr>
  <tr>
    <th>Emotion</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>surprise</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
  </tr>
  <tr>
    <td>anger</td>
    <td>0.42 (0.54)</td>
    <td>−0.19 (0.54)</td>
    <td>2.26 (0.70)**</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>disgust</td>
    <td>0.64 (0.51)</td>
    <td>0.26 (0.48)</td>
    <td>1.02 (0.67)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>fear</td>
    <td>0.85 (0.53)</td>
    <td>0.40 (0.53)</td>
    <td>0.87 (0.67)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>joy</td>
    <td>1.79 (0.45)***</td>
    <td>1.76 (0.43)***</td>
    <td>5.16 (0.89)***</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>sadness</td>
    <td>1.05 (0.56)+</td>
    <td>0.47 (0.55)</td>
    <td>2.22 (0.74)**</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Study</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>creation</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
  </tr>
  <tr>
    <td>donation</td>
    <td>−0.53 (0.26)*</td>
    <td>−0.51 (0.25)*</td>
    <td>−0.59 (0.37)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Text-image relation</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Text describe image</td>
    <td>−0.30 (0.37)</td>
    <td>0.13 (0.34)</td>
    <td>0.31 (0.50)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Text understand image</td>
    <td>−0.28 (0.36)</td>
    <td>−0.36 (0.34)</td>
    <td>−0.49 (0.46)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Image understand text</td>
    <td>−0.50 (0.36)</td>
    <td>−0.46 (0.33)</td>
    <td>−1.02 (0.45)*</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Image conveys emotion</td>
    <td>−0.14 (0.43)</td>
    <td>−0.65 (0.47)</td>
    <td>0.08 (0.53)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Text conveys emotion</td>
    <td>0.41 (0.47)</td>
    <td>0.67 (0.47)</td>
    <td>−0.03 (0.51)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Emotion appraisals</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Familiarity</td>
    <td>0.18 (0.32)</td>
    <td>−0.01 (0.32)</td>
    <td>0.17 (0.39)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Predictability</td>
    <td>−0.20 (0.40)</td>
    <td>0.07 (0.40)</td>
    <td>−0.74 (0.58)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Pleasantness</td>
    <td>0.30 (0.77)</td>
    <td>−0.20 (0.75)</td>
    <td>−0.49 (1.04)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Unpleasantness</td>
    <td>0.34 (0.66)</td>
    <td>0.28 (0.69)</td>
    <td>1.07 (0.78)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Goalrelevance</td>
    <td>0.09 (0.44)</td>
    <td>0.00 (0.44)</td>
    <td>−0.14 (0.58)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Ownresponsibility</td>
    <td>−0.04 (0.44)</td>
    <td>0.01 (0.41)</td>
    <td>0.10 (0.54)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Anticip.conseq</td>
    <td>0.24 (0.44)</td>
    <td>0.09 (0.44)</td>
    <td>0.69 (0.59)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Goalsupport</td>
    <td>0.27 (0.49)</td>
    <td>−0.01 (0.50)</td>
    <td>0.28 (0.72)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Own control</td>
    <td>−0.08 (0.49)</td>
    <td>0.05 (0.47)</td>
    <td>−0.38 (0.62)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Others control</td>
    <td>0.25 (0.33)</td>
    <td>−0.12 (0.32)</td>
    <td>−0.07 (0.42)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Accept conseq.</td>
    <td>−0.35 (0.37)</td>
    <td>−0.48 (0.39)</td>
    <td>−0.52 (0.50)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Internal standards</td>
    <td>−0.73 (0.44)+</td>
    <td>−0.70 (0.42)+</td>
    <td>−0.99 (0.51)+</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Attention</td>
    <td>0.41 (0.46)</td>
    <td>0.60 (0.46)</td>
    <td>0.52 (0.56)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Not consider</td>
    <td>0.18 (0.44)</td>
    <td>0.09 (0.41)</td>
    <td>0.62 (0.52)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Effort</td>
    <td>0.09 (0.45)</td>
    <td>−0.29 (0.46)</td>
    <td>−0.58 (0.55)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Participant gender identity</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Nonman</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Man</td>
    <td>0.40 (0.27)</td>
    <td>0.68 (0.27)*</td>
    <td>0.63 (0.36)+</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Age</td>
    <td>1.43 (0.77)+</td>
    <td>1.22 (0.73)+</td>
    <td>0.70 (0.96)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Participant ethnicity</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Non-European or multiethnic</td>
    <td>____</td>
    <td>____</td>
    <td>____</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>European</td>
    <td>0.16 (0.34)</td>
    <td>−0.03 (0.32)</td>
    <td>0.08 (0.39)</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

continued …
Table 11: Coefficients for logistic regressions predicting correct model emotion classification, in log odds units. Numeric and ordinal variables are scaled to range from 0–1. Reference levels – which serve as baselines for each variable – are indicated with hyphens in the table. They are chosen according to Johfre and Freese (2021), and are in general either the lowest category conceptually (in the case of inherently ordered variables like education) or lowest category numerically in terms of their effect on the dependent variable (in the case of unordered variables like emotion). Clustered standard errors (reported in parentheses) are used to account for modeling multiple predictions of the same posts. *p<.1. **p<.05. ***p<.01. **** p<.001.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">CLIP: creation-trained</th>
    <th colspan="2">CLIP: donation-trained</th>
    <th colspan="2">minicpm-v</th>
  </tr>
  <tr>
    <th>Coef</th>
    <th>SE</th>
    <th>Coef</th>
    <th>SE</th>
    <th>Coef</th>
    <th>SE</th>
  </tr>
  <tr>
    <th colspan="7">Participant education</th>
  </tr>
  <tr>
    <td>No college</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>B.Sc.</td>
    <td>0.20</td>
    <td>(0.28)</td>
    <td>0.10</td>
    <td>(0.28)</td>
    <td>0.07</td>
    <td>(0.33)</td>
  </tr>
  <tr>
    <td>Masters or more</td>
    <td>−0.33</td>
    <td>(0.35)</td>
    <td>0.15</td>
    <td>(0.35)</td>
    <td>−1.03</td>
    <td>(0.42)*</td>
  </tr>
  <tr>
    <th colspan="7">Participant employment status</th>
  </tr>
  <tr>
    <td>Not full-time</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Full-Time</td>
    <td>−0.06</td>
    <td>(0.26)</td>
    <td>−0.06</td>
    <td>(0.26)</td>
    <td>−0.01</td>
    <td>(0.33)</td>
  </tr>
  <tr>
    <th colspan="7">Participant is a student</th>
  </tr>
  <tr>
    <td>No</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Yes</td>
    <td>0.02</td>
    <td>(0.32)</td>
    <td>−0.39</td>
    <td>(0.30)</td>
    <td>0.21</td>
    <td>(0.43)</td>
  </tr>
  <tr>
    <th colspan="7">Participant social media frequency</th>
  </tr>
  <tr>
    <td>Use social media</td>
    <td>−0.04</td>
    <td>(0.44)</td>
    <td>0.06</td>
    <td>(0.46)</td>
    <td>1.16</td>
    <td>(0.86)</td>
  </tr>
  <tr>
    <td>Post on social media</td>
    <td>−0.31</td>
    <td>(0.42)</td>
    <td>−0.21</td>
    <td>(0.38)</td>
    <td>−0.03</td>
    <td>(0.52)</td>
  </tr>
  <tr>
    <th colspan="7">Participant preferred platform</th>
  </tr>
  <tr>
    <td>Facebook</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Instagram</td>
    <td>0.63</td>
    <td>(0.33)+</td>
    <td>0.50</td>
    <td>(0.31)</td>
    <td>0.67</td>
    <td>(0.45)</td>
  </tr>
  <tr>
    <td>X (Twitter)</td>
    <td>0.30</td>
    <td>(0.40)</td>
    <td>−0.17</td>
    <td>(0.38)</td>
    <td>0.45</td>
    <td>(0.54)</td>
  </tr>
  <tr>
    <td>Other platform</td>
    <td>0.33</td>
    <td>(0.41)</td>
    <td>0.19</td>
    <td>(0.38)</td>
    <td>0.31</td>
    <td>(0.50)</td>
  </tr>
  <tr>
    <th colspan="7">Image style</th>
  </tr>
  <tr>
    <td>Personal Photo</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Pro Photo</td>
    <td>0.10</td>
    <td>(0.31)</td>
    <td>0.02</td>
    <td>(0.33)</td>
    <td>−0.02</td>
    <td>(0.43)</td>
  </tr>
  <tr>
    <td>Other image style</td>
    <td>−0.33</td>
    <td>(0.36)</td>
    <td>−0.28</td>
    <td>(0.36)</td>
    <td>0.02</td>
    <td>(0.42)</td>
  </tr>
  <tr>
    <th colspan="7">Post text length</th>
  </tr>
  <tr>
    <td>Event duration</td>
    <td>−0.94</td>
    <td>(0.64)</td>
    <td>−0.55</td>
    <td>(0.63)</td>
    <td>−0.11</td>
    <td>(0.75)</td>
  </tr>
  <tr>
    <td>Event intensity</td>
    <td>0.14</td>
    <td>(0.78)</td>
    <td>−0.08</td>
    <td>(0.80)</td>
    <td>1.59</td>
    <td>(1.06)</td>
  </tr>
  <tr>
    <td>Emotion duration</td>
    <td>0.37</td>
    <td>(0.75)</td>
    <td>0.12</td>
    <td>(0.77)</td>
    <td>−0.58</td>
    <td>(0.84)</td>
  </tr>
  <tr>
    <td>Emotion intensity</td>
    <td>−0.04</td>
    <td>(0.73)</td>
    <td>0.38</td>
    <td>(0.72)</td>
    <td>−1.37</td>
    <td>(1.06)</td>
  </tr>
  <tr>
    <td>Num.Obs.</td>
    <td>1365</td>
    <td></td>
    <td>1365</td>
    <td></td>
    <td>1362</td>
    <td></td>
  </tr>
  <tr>
    <td>RMSE</td>
    <td>0.45</td>
    <td></td>
    <td>0.45</td>
    <td></td>
    <td>0.39</td>
    <td></td>
  </tr>
</table>

Reviewer paper summaries: This study examines three methods for gathering multimodal data for emotion analysis. In the first approach, participants generate fictional posts that convey specific emotions. The second approach involves participants sharing posts from their social media profiles reflecting the prompted emotion. In the third approach, participants submit their five most recent posts and annotate each. The goal is to compare artificially created content with authentic social media posts.
Summary of strengths: This work introduces a valuable dataset for multimodal emotion detection using three approaches. 

The study examines the effectiveness of synthetic data compared to genuine posts and its potential for future research. 

The writing is clear, and the organization is well-structured.
Summary of weaknesses: The paper presents a limited analysis of the data findings and their potential use in future research.
Comments, suggestions and typos: Try to add figures and tables closer to where they are mentioned in the text.

"We will make these data available for research after taking appropriate anonymization steps." -> When are you planning to anonymize the data? This process is an important step, and the paper should include the anonymization statement and process before realizing the data to ensure data protection.

Post and Event Annotations -> It is unclear if participants were required to fill out this form for each post (I assume it is) or if it was 1 for everything. For RECENT, it explains that the last five posts are used, but for CREATION and DONATION, it doesn't specify how many posts they are asked to donate or create.

"We require that they reside in the United Kingdom or Ireland, have resided there for at least five years, are 18 or older, and be native English speakers." -> I imagine it is because of the ease of payment. Or is there some other reason? It is essential to clarify this as this would bias the work to this specific area, and it is important to mention it so that it is not generalized that it is the same all over the world without being sure of it.

Footnote: "are similarly many significant differences between CREATION
and DONATION for Joy as for the other emotions." -> Reduce text so it does not make page breaks and is easier to read.

"Strict definition if the participant indicated that two or more emotions were tied for greatest intensity." -> It is unclear to me what you mean by greatest intensity.

Table 4: average F1 scores -> macro f1 of all emotions?
Overall assessment scores: 4