Venue: NAACL 2025

# Improving and Assessing the Fidelity of Large Language Models Alignment

to Online Communities

 Minh Duc Chu   Zihao He  Rebecca Dorn    Kristina Lerman

USC Information Sciences Institute

{mhchu, zihaoh, rdorn}@usc.edu, lerman@isi.edu

###### Abstract

Large language models (LLMs) have shown promise in representing individuals and communities, offering new ways to study complex social dynamics. However, effectively aligning LLMs with specific human groups and systematically assessing the fidelity of the alignment remains a challenge. This paper presents a robust framework for aligning LLMs with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language, including authenticity, emotional tone, toxicity, and harm. We demonstrate the utility of our approach by applying it to online communities centered on dieting and body image. We administer an eating disorder psychometric test to the aligned LLMs to reveal unhealthy beliefs and successfully differentiate communities with varying levels of eating disorder risk. Our results highlight the potential of LLMs in automated moderation and broader applications in public health and social science research 1 .

Footnote 1: Our data and code are available at [https://github.com/Davidchu11381/llm_align_eval](https://github.com/Davidchu11381/llm_align_eval).

0
Footnote 1: Our data and code are available at [https://github.com/Davidchu11381/llm_align_eval](https://github.com/Davidchu11381/llm_align_eval).

0
Footnote 2: By “base LLMs” we refer to models not finetuned for instruction following

## 1 Introduction

[Warning: This paper discusses eating disorders, which some may find distressing.]

Large language models (LLMs) have demonstrated an exceptional ability to generate nuanced responses to natural language prompts, suggesting their potential for creating high-fidelity proxies of people Simmons and Hare (2023). Digital representations of human groups (digital twins) are computational models that mimic collective behaviors, social interactions, and communication patterns in real-world communities Rossetti et al. (2024b). Leveraging LLMs to build these representations offers powerful tools for studying human behavior, enhancing human-computer interactions, and moderating online spaces to foster prosociality and safety.

To create such digital twins, researchers align LLMs to subgroups through steering--prompting the LLM to mimic the target subgroup with its key characteristics Santurkar et al. (2023); Durmus et al. (2023). However, this approach does not fully resolve the misalignment between LLMs and the target subgroup. Another method is finentuning base LLMs2, such as GPT-2, on data from specific subgroups Jiang et al. (2022b); He et al. (2024d). While this can produce models reflecting subgroup linguistic patterns, these finetuned models often lack the flexibility to follow diverse instructions, limiting their broader applicability.

Footnote 2: By “base LLMs” we refer to models not finetuned for instruction following

Another key challenge in developing digital representations of human subgroups is evaluating the alignment between the LLM and the target group. Traditional methods compare the LLM's responses to surveys with those of the target group Santurkar et al. (2023); Durmus et al. (2023), but this approach misses critical aspects of human expression like emotional reactions He et al. (2024b). Additionally, surveys are not scalable due to their cost and time requirements, particularly for marginalized or hard-to-reach groups. In addition, mapping diverse online communities to clear demographic identities greatly complicates alignment evaluation.

To address these challenges, we propose a framework for aligning LLMs with online communities on instructions that are created in a fully unsupervised manner. Additionally, we introduce a comprehensive evaluation framework to assess alignment. This enables the creation of high-fidelity digital representations of online communities, paving the way for new research into human behavior Jiang et al. (2022c), content moderation He et al. (2024c), public mental health Sharma et al. (2024), and social science Grossmann et al. (2023). As one example, we can administer psych

to these digital replicas to identify at-risk communities prone to psychopathologies.

Specifically, our alignment method takes a corpus of social media posts (e.g., tweets) from an online community and creates a set of demonstrations (instruction-response pairs) based on the posts. In each demonstration, as shown in Figure 1, the instruction specifies the task (e.g., tweet generation) with the response being the exact tweet. We then finetune an LLM on these demonstrations to align it with the community. To assess alignment, we generate a synthetic text corpus using the finetuned LLM and compare it to the original posts along four key aspects: 1) authenticity, 2) emotional tone, 3) toxicity, and 4) harm. These dimensions capture the essential features of online social communication, ensuring the aligned LLM accurately reflects the semantics, affect, and style of the target group's discourse.

To demonstrate our framework's utility, we analyze Twitter discussions in diet and body image communities, where harmful body image attitudes persist. Twitter's emphasis on user connection and lax content moderation allows communities to organically form and freely express their voice. While these communities can offer support and encouragement, they often promote unhealthy behaviors and normalize beliefs that put people at risk for developing eating disorders (EDs). Applying traditional psychometric instruments to screen individuals in those online spaces for EDs is impractical and potentially unethical (see Ethics Statement). Instead, we use our framework to align LLMs with these communities through automatically generated demonstrations and evaluate alignment to show that the finetuned LLMs outperform baseline LLMs in creating high-fidelity proxies of online communities. We then apply an ED screening questionnaire to community-aligned LLMs, revealing significant differences between communities: pro-anorexia communities show a high risk of unhealthy behaviors, while those critical of the diet culture exhibit the lowest risk. These findings highlight our framework's potential for automated moderation by distinguishing communities with varying levels of ED risk.

Our framework offers a scalable approach to modeling and analyzing online communities, with broad implications for understanding and mitigating harmful behaviors. By applying this method to diet and body image communities, we demonstrate its potential to contribute to public health and social science research, highlighting the value of LLMs in studying complex social dynamics.

## 2 Related Work

**Digital Representations of Human Subgroups** Digital twins--precise virtual replicas of complex real-world systems--are increasingly employed for advanced analysis and experimentation Tao and Qi (2019); Grieves (2011), particularly in monitoring human behaviors and health outcomes Ferdousi et al. (2022); Shengli (2021); El Saddik et al. (2019). Recent advances leverage social media data to provide deeper insights into human interactions Olad and Valilai (2020), exemplified by Rossetti et al. (2024a)'s Y social, which uses LLMs to simulate social media interactions and study network dynamics in controlled environments.

Building on this foundation, researchers have explored various approaches to align LLMs with diverse human subgroups Simmons and Hare (2023). While some have attempted prompt-based steering towards specific demographic groups Santurkar et al. (2023); Durmus et al. (2023); He et al. (2024b), this approach has shown limitations in achieving true alignment, particularly with organically-formed communities. More promising results have come from finetuning approaches: Jiang et al. (2022b) developed communityLM by finetuning GPT-2 models Radford et al. (2019) on politically divergent tweets, while He et al. (2024d) extended this to examine broader community interactions. Most recently, He et al. (2024a) proposed using advanced LLMs to distill community knowledge into instruction-response pairs for finetuning, though this approach faces cost constraints.

Our study aims to advance this field by developing a framework that uses LLMs to create digital representations of online communities, specifically focusing on analyzing collective mental well-being.

**Evaluating LLMs' Alignment to Subgroups** Existing works Santurkar et al. (2023); Durmus et al. (2023) measure an LLM's alignment with a target subgroup using multi-choice surveys. Specif

Figure 1: An example of a demonstration from a oreating disorder community, where the response is a tweet from the community.

 

ically, they prompt the LLM to respond to a survey question from the perspective of a subgroup and then compare the LLM-generated distribution over the different options of the question to that of the survey respondents belonging to the target group. However, collecting survey responses can be costly and time-consuming. Also, responses on sensitive topics, such as mental health, may be biased due to stigma and social desirability bias Gordon (1987). Our framework that evaluates LLM alignment by comparing the LLM-generated synthetic text to the original text written by humans is significantly more scalable, unbiased, and cost-effective.

LLMs and Psychometric TestsLLMs can respond to psychometric instruments designed for humans, with researchers using these tools to examine LLMs' decision-making, reasoning, and cognitive traits--a practice termed "AI Psychometrics" Pellert et al. (2024). Research shows LLMs can engage with various psychometric tools, from anxiety questionnaires Coda-Forno et al. (2023) to moral reasoning assessments Tanmay et al. (2023) and personality tests Jiang et al. (2022); Lu et al. (2023); Serapio-Garcia et al. (2023). Our work differs by using these instruments via a finetuned LLM to analyze specific online communities, helping identify unhealthy beliefs and potential pathologies like eating disorders-related cognitions.

Online Eating Disorders CommunitiesPro-ED (pro-anorexia) communities are online spaces that frame EDs as a lifestyle rather than an illness. While they provide social support, a sense of belonging, and empathy for stigmatized individuals Juarascio et al. (2010); Oksanen et al. (2016); Yeshua-Katz and Martins (2013); McCormack (2010), they also promote harmful behaviors, such as weight loss tips and "thinspiration" imagery, exacerbating EDs and psychological distress Ging and Garvey (2018); Mento et al. (2021).

Previous research has focused on identifying harmful content and at-risk users within these communities. For example, Chancellor et al. (2016) develop a lexical classifier to predict posts moderated by Instagram for self-harm content, comparing pro-recovery and pro-ED communities Chancellor et al. (2016, 2016). In contrast, our study examines the collective mindset of these communities as expressed through their discussions, using advanced language models to assess attitudes toward mental health and body image issues.

## 3 Communities in Online Discussions

We collect online conversations related to EDs and identify organically-formed communities within the broader context of weight loss, dieting, and fitness discussions.

### Data Collection

We collected 2.6M tweets from 557K users from October 2022 to March 2023 using ED-related keywords to query Twitter. For keywords, we start with a set of terms that promote ED Chancellor et al. (2016); Peter et al. (2016), such as _thingo_ (thin inspiration), _proana_ (pro-anorexia), and _promia_ (pro-bulimia), among others. We remove spam terms yielding unrelated content, such as _skinny_. We expanded the query set to include closely related topics such as diet and weight loss through terms such as (_ketodiet_, _weightloss_, \(\ldots\)), and antidiet culture (_bodypositivity_, _dietculture_, \(\ldots\)). See Appendix A.1 for the full set of search terms and why we selected them.

### Identifying Communities

We construct a retweet network where nodes are users, and (undirected) edges link users who retweet each other. Visualization of the retweet network is shown in Figure 7 in Appendix A.2. We use Louvain modularity maximization Blondel et al. (2008) to identify dense clusters of users who frequently retweet one another. These clusters are organically formed based on shared interests, consisting of users who pay attention to each other. Detailed statistics and content of the clusters are shown in Table 3 and Figure 6 in Appendix A.2. Based on the thematic profiling of discussions (Table 4 in Appendix A.2), we categorize the clusters into six communities: _Pro-ED_, _Keto & Diet_, _Weight Loss Drugs_, _Body Image_, _Healthty Lifestyle & Weight Loss_, and _Anti-ED_. This categorization is intended to label the communities for easy reference in subsequent analyses, and the labels do not cover the full spectrum of discussions in the communities.

After identifying communities in the retweet network, we clean the tweets by removing URLs, mentions, hashtags, and emojis, and we filter out retweets and comments, only keeping the original tweets. To ensure high-quality data, we compute the perplexities of the tweets using BERTweet Nguyen et al. (2020) that is pretrained on tweets, and select a maximum of 10K highest quality (i.e., 

lowest perplexity) tweets from each community. If there are fewer than 10K tweets from the community, we keep all of them. The numbers of tweets from the community _Pro-ED_, _Keto & Diet_, _Body Image_, _Anti-ED_, _Healthy Lifestyle & Weight Loss_, and _Weight Loss Drugs_ are 10K, 10K, 3.3K, 2.9K, 10K, and 10K respectively.

## 4 Aligning LLMs to Communities

There are \(n\) online communities {\(C_{1}\), \(C_{2}\), ...., \(C_{n}\)} on a topic (e.g., EDs), each characterized by their own beliefs and perspectives. Members of a community \(C_{i}\) produce a body of text \(D_{i}\) (e.g., tweets) that reflects their collective opinions and behaviors. Our objective is to align an LLM \(f\) to each specific community \(C_{i}\) by training it on the corresponding text corpus \(D_{i}\). The resulting model, \(f^{\prime}_{i}\), should capture the community's unique collective mindset, enabling it to generate responses that authentically represent the community's voice.

### Constructing Instruction-Response Pairs

To align an LLM \(f\) to a particular community \(C\), we employ a finetuning process using a set of demonstrations (instruction-response pairs). We propose creating demonstrations based on the community's raw text corpus \(D\), which is cost-efficient, and yet curated demonstrations can be used to fine-tune a foundational LLM (e.g., Llama-3) effectively.

For each community \(C_{i}\), we use tweets in \(D_{i}\) as the responses verbatim in the demonstrations. To create instructions that can be answered by the tweets, we focus on the tweet generation task. We curate an instruction pool of 20 different instruction templates (Table 5 in Appendix B.1). We diversify the prompts to improve the model's text-generation capabilities and enhance its robustness to linguistic variations Salinas and Morstatter (2024). For a community, a tweet is paired with an instruction randomly sampled from the instruction pool. As a result, the community has a maximum of 10K demonstrations \(Z_{i}=\{(x_{j},y_{j})\}_{j=1}^{m}\) for tweet generation, where \(m\) is the size of the community's text corpus \(D\).

For each community, we augment the demonstrations of tweet generation with the 52K Alpaca Taori et al. (2023) demonstrations that cover a wide range of tasks to retain the instruction-following capabilities of the LLM and not restrict it to only generating tweets. Ultimately, there are a maximum of 62K demonstrations in the demonstration corpus for a community.

### Instruction Tuning LLMs

For each community \(C_{i}\), we align a Llama-3 model \(f^{\prime}_{i}\) Dubey et al. (2024) to the community using its demonstration corpus \(Z_{i}\). The LLM is finetuned on 4 Tesla H100-80GB GPUs with batch size 8 for 3 epochs, which takes about 3 hours.

## 5 Assessing Alignment

To assess how effectively a finetuned LLM \(f^{\prime}_{i}\) aligns with its target community \(C_{i}\), we measure the model's ability to mimic the responses of community members. We first generate a synthetic corpus \(D^{ft}_{i}\) using \(f^{\prime}_{i}\) and compare it to the original text corpus \(D_{i}\) from the community. The more closely \(D^{ft}_{i}\) resembles \(D_{i}\), the better aligned the LLM is with the community. We evaluate the similarity between \(D^{ft}_{i}\) and \(D_{i}\) across 1) authenticity, 2) emotional tone, 3) toxicity, and 4) harm. While not exhaustive, these aspects capture the essential features of online social communication. Authenticity ensures that the aligned LLM accurately reflects the meaning, content, and linguistic patterns of the target population's language and generates contextually appropriate responses. Emotional tone captures the affective aspects of communication, helping to convey nuances that may not be evident from semantics alone. Toxicity measures the model's ability to reflect hostility and aggression in the population's discourse. Finally, recognizing that certain online conversations can negatively impact users, we compare the types and levels of harm in language across groups. Although in this paper we focus on the domain of EDs, we argue that our LLM alignment framework is naturally generalizable to online communities in other domains.3

Footnote 3: We acknowledge that evaluating harm is more tailored to the ED domain, but other evaluation aspects should be widely applicable.

### Synthetic Corpus Generation

Given a community \(C_{i}\), we create a synthetic corpus \(D^{ft}_{i}\) by prompting an LLM \(f^{\prime}_{i}\) aligned to the community to generate tweets. To diversify the LLM generations on different topics, we compile a set of 27 topics relevant to ED discussions, such as _thinspo_, _fitspo_, and _bonespo_ (Appendix C.1), and prompt LLMs to generate tweets on these topics. When generating tweets on a topic, we reuse the instructions from the instruction pool (Table 5 

in Appendix B.1), with topic-oriented generation. An example instruction is "What would you tweet about **fasting?**" For each topic, the LLM initially generates 1000 tweets, resulting in a synthetic corpus \(D_{i}^{ft}\) with 27,000 tweets for all 27 topics (see Appendix C.4 for examples). To encourage diversity in \(D_{i}^{ft}\), we remove duplicate tweets. In addition, to ensure coherence, a generated tweet is filtered out if its perplexity score is above 400, as evaluated by BERTweet.

A natural concern is that \(D_{i}^{ft}\) is simply a duplicate of \(D_{i}\), as \(f_{i}^{\prime}\) is finetuned on \(D_{i}\). To this end, we detail the number of tweets in the community's original text corpus \(D_{i}\) that contain the keyword(s) for each of the 27 topics listed in Table 7 (see Appendix C.1). We observe that \(D_{i}\) includes very few tweets discussing these topics because we eliminate hashtags during tweet processing, and these keywords typically appear in the hashtags. Consequently, when the LLM is finetuned on \(D_{i}\), it is not extensively exposed to tweets directly related to these topics.

To further ensure that the synthetic corpus \(D_{i}^{ft}\) does not simply replicate \(D_{i}\), we omit generated tweets that are substantially syntactically similar to the human-written ones. Specifically, a tweet is removed from \(D_{i}^{ft}\), if its ROUGE-L similarity to any existing tweet in \(D_{i}\) is greater than 0.7. As a result, when inspecting the synthetic corpus \(D_{i}^{ft}\), we are essentially examining if the finetuned model \(f_{i}^{\prime}\) is able to extrapolate from existing data in \(D_{i}\) and predict how community members might

Figure 2: The framework of our method. (1) We align an LLM (Llama-3) to an online community by finetuning the LLM to follow instructions on the task of generating tweets written by users in the community. (2) To prove the effectiveness of alignment, we compare three tweet corpora for each community: human-written tweets \(D_{i}\), LLM-Context-generated tweets \(D_{i}^{context}\), and finetuned LLM-generated tweets \(D_{i}^{fit}\). We show that \(D_{i}^{ft}\) is closer to \(D_{i}\) than \(D_{i}^{context}\) is, along the following aspects: (a) A classifier trained to classify the tweet origin (what community the tweet belongs to) on \(\mathbb{D}=\{D_{i}\}_{i=1}^{n}\) performs better on \(\mathbb{D}^{fr}=\{D_{i}^{ft}\}_{i=1}^{n}\), than on \(\mathbb{D}^{context}=\{D_{i}^{context}\}_{i=1}^{n}\); (b) the emotion and toxicity distributions of \(D_{i}^{ft}\) are much closer to that of \(D_{i}\) compared to \(D_{i}^{context}\); (c) the semantic embeddings of \(D_{i}^{ft}\) are closer to that of \(D_{i}\) in the embedding space than that of \(D_{i}^{context}\) are; (d) a human annotator decides that \(D_{i}^{ft}\) is more aligned to the underlying distribution of \(D_{i}\) than \(D_{i}^{context}\) is; (e) two ED experts determine that \(D_{i}^{ft}\) carries harmful narratives that are more similar to \(D_{i}\) than \(D_{i}^{context}\) does. (3) As the LLM is aligned with the community and can speak in the voice of that community, we administer an ED questionnaire to screen the community for EDs.

 

discuss these previously unseen topics.

Finally, to ensure class balance, we sample 6000 generated tweets from each community. More details are provided in Appendix C.4.

**Baseline** We use the LLM with in-context learning (LLM-Context) as a baseline. We do not finetune this baseline model. For a community \(C_{i}\), when prompting the model to generate synthetic tweets on topic \(t\), we retrieve 250 tweets from \(D_{i}\) as in-context examples, consisting of (1) the tweets containing the topic keyword(s), if available, and (2) randomly sampled tweets from \(D_{i}\). Each retrieved tweet is truncated at 20 tokens. We include the retrieved tweets in the prompt, instruct the model to learn the community's mindset from the tweets, and generate synthetic tweets. See Appendix C.2 for the complete prompting template. The synthetic corpus from LLM-Context is denoted as \(D^{context}_{i}\).

### Alignment Dimensions

#### 5.2.1 Automatic Evaluation

Tweet Origin ClassificationWe train a classifier to determine the community from which a tweet originated by finetuning Llama-3 using demonstrations with the following template "Instruction: _From these communities: Pro Eating Disorder, Keto & Diet, Body Image, Anti Eating Disorder, Healthy lifestyle & Weight Loss, and Weight Loss Drugs; which community does this Tweet belong to? [Tweet]_ Response: _[Community name]_". We randomly sample 3,000 original tweets from each community's corpus \(D_{i}\) and construct a total of 18,000 demonstrations for finetuning. We train the classifier using 95% demonstrations and use the remaining 5% to test, with test accuracy of 0.74. We classify the finetuned LLM-generated tweets in \(\mathbb{D}^{ft}=\{D^{ft}_{i}\}_{i=1}^{n}\) and LLM-Context-generated tweets \(\mathbb{D}^{content}=\{D^{context}_{i}\}_{i=1}^{n}\), leading to an F1 accuracy score of 0.53 and 0.40, respectively. These results indicate that the classifier trained on original tweets accurately recognizes the tweets generated by the finetuned LLM. However, it performs poorly on the tweets generated by the LLM-Context, demonstrating that the finetuned LLMs better capture community-specific linguistic characteristics.

Semantic ComparisonWe compute the semantic embeddings of \(D_{i}\), \(D^{ft}_{i}\), and \(D^{context}_{i}\) using BERTweet Nguyen et al. (2020). We then measure the distance between these embeddings using the Frrechet Inception Distance (FID) Heusel et al. (2017). This metric provides a quantitative measure of the semantic distance between two text corpora. We implement it using the IBM comparing-corpora package Kour et al. (2022). \(FID(D_{i},D^{ft}_{i})\) and \(FID(D_{i},D^{context}_{i})\) for different communities are shown in Table 1. We see that \(FID(D_{i},D^{ft}_{i})\) is much smaller than \(FID(D_{i},D^{context}_{i})\) for almost all communities, implying that the finetuned LLM outputs are more semantically similar responses to the original posts compared to the LLM-Context.

Emotion and Toxicity AnalysisEmotions and toxicity are vital aspects of online social interactions Prescott et al. (2019). They can reveal the underlying tone, intent, and style of communication of online users. Within ED communities, these elements heavily impact self-perception of body image Brytek-Matera and Schiltz (2011) and can exacerbate body dissatisfaction Kast (2018).

We analyze the emotions of tweets using Demux Chochlakis et al. (2023). For each tweet, Demux returns a vector of confidence scores of eleven emotions: anger, anticipati

\begin{table}
\begin{tabular}{p{113.8pt} p{56.9pt} p{56.9pt}} \hline
**Community** & \(FID(D_{i},D^{context}_{i})\) & \(FID(D_{i},D^{ft}_{i})\) \\ \hline Pro-ED & 1.16 & **0.82** \\ Body Image & 1.25 & **0.74** \\ Keto \& Diet & 1.19 & **0.50** \\ Anti-ED & 0.84 & **0.42** \\ Healthy Lifestyle \& Weight Loss & 1.11 & **0.82** \\ Weight Loss Drugs & **0.90** & 1.99 \\ \hline \end{tabular}
\end{table}
Table 1: Fréchet Inception Distances (FID) (1) between human-written tweets \(D_{i}\) and LLM-Context generated tweets \(D^{context}_{i}\), and (2) between human-written tweets \(D_{i}\) and finetuned LLM generated tweets \(D^{ft}_{i}\). A smaller distance indicates more similarity.

Figure 3: Emotional agreement (a) between human-written tweets and LLM-Context-generated tweets, and (b) between human-written tweets and finetuned LLM-generated tweets. The differences in the emotional alignment between pairs within each community are statistically significant at a 95% confidence level.

 

joy, love, optimism, pessimism, sadness, surprise, and trust. For a community \(C_{i}\), we sum the emotion confidence vectors of all tweets (i.e., the ones \(D_{i}\), \(D^{ft}_{i}\), or \(D^{context}_{i}\)) and normalize them, resulting in an emotion distribution vector \(\mathbf{e}_{i}\). We then compute the agreement between \(\mathbf{e}_{i}^{ft}\) and \(\mathbf{e}_{i}\), and between \(\mathbf{e}_{i}^{context}\) and \(\mathbf{e}_{i}\). The emotional alignment is measured as one minus the Jensen-Shannon distance between the two distribution vectors He et al. (2024b). As illustrated in Figure 3, for most communities, \(D^{ft}_{i}\) more closely resembles the emotional tone of \(D_{i}\) compared to \(D^{context}_{i}\). This demonstrates that finetuning LLMs can effectively capture the authentic emotional tone of posts from communities.

We use Detoxify Hanu and Unitary team (2020) to measure toxicity in tweets Rajadesingan et al. (2020); Sheth et al. (2022). For a tweet, Detoxify returns a value between 0 and 1 indicating the level of toxicity4. Figure 4 shows the distributions of toxicity scores of human-written tweets \(D_{i}\), LLM-Context-generated tweets \(D^{context}_{i}\) and finetuned LLM-generated tweets \(D^{ft}_{i}\). We observe that the toxicity distribution of \(D^{ft}_{i}\) matches more closely to that of \(D_{i}\) compared to \(D^{context}_{i}\) for most communities, and tweets from the anti-ED community have the highest toxicity.

Footnote 4: We only include tweets with toxicity levels equal to or greater than 0.05 for clarity and to reduce noise.

#### 5.2.2 Human Evaluation

Authenticity ComparisonAn annotator with expertise in EDs on social media was presented with 300 triplets, 50 from each community, where a triplet consists of a community name, a LLM-Context-generated tweet \(d^{context}_{i,j}\in D^{context}_{i}\), and a finetuned LLM-generated tweet \(d^{ft}_{i,k}\in D^{ft}_{i}\). Both tweets in a triplet are on the same topic and from the same community. For each triplet, the annotator was asked to decide which tweet was more aligned with the given community, by referring to the following characteristics: mis/use of ingroup language, references to themes in underlying distribution (e.g. the Body Image community often references nudity), use of capitalization, and coherence of message. In 248 of 300 triplets, the annotator chose \(d^{ft}_{i,j}\) as a better match, showing better alignment of finetuned LLM with the community.

Harm CategorizationOnline ED communities pose significant risks by promoting and normalizing harmful behaviors Lerman et al. (2023a). Harm and toxicity are distinct in online discourse where toxicity detection algorithms may mistakenly flag explicit yet harmless language as toxic Sanchez et al. (2024). We come up with a dimension tailored to this ED domain where we assess harm by focusing on the underlying semantic content, as opposed to surface-level style. Our goal is for the finetuned LLM, \(f^{\prime}_{i}\), to accurately capture the level of harm within these communities.

There are no existing classifiers for automatic harm detection in the context of EDs. In collaboration with ED experts, we developed a comprehensive taxonomy of harm specific to ED online content, covering dimensions such as body image, relationships with food and exercise, and self-disclosure. Harm is defined as the promotion or glorification of unhealthy dieting and body objectionation.

We sampled 60 tweets from each community (360 for all six communities), with 20 each from \(D_{i}\), \(D^{context}_{i}\), and \(D^{ft}_{i}\). Two annotators with ED expertise labeled these tweets based on two tasks: (1) determine whether a tweet is harmful, and (2) classify harmful tweets into one of three fine-grained categories--_body image objectification_, _relationship to food and exercise_, or _self-disclosure_. Annotators achieved a Cohen's Kappa score of 0.453 for identifying if harm was present, and 0.617 for classifying fine-grained harm categories, indicating fair to moderate agreement (see more details in Limitations).

A tweet was assigned to a harm category if both annotators agreed. Out of 360 tweets, 41 were classified into harm categories across \(\mathbb{D}\), \(\mathbb{D}^{context}\), and \(\mathbb{D}^{ft}\). Figure 5 shows the distribution of these categories, demonstrating that finetuned LLMs better replicate the distribution of harm found in the community's conversations.

Figure 4: Toxicity distributions across different communities of human-written tweets, LLM-Context-generated tweets, and finetuned LLM-generated tweets.

 

Case Study: Screening Online Communities for Eating Disorders

In SS5, we demonstrate that the finetuned LLM learns a more accurate representation of the community than the baseline in-context learning method. This motivates us to apply psychometric instruments designed to evaluate an individual's risk of EDs to online communities to help uncover unhealthy body and eating concerns within them.

Eating Disorder ScreenerThe Stanford-Washington University Eating Disorder Screener (SWED) Graham et al. (2019) is a concise screening tool for ED behaviors. The screener has been widely used in both men and women Fitzsimmons-Craft et al. (2018) and incorporated into an online tool NEDA (NEDA, 2019) by the National Eating Disorders Association (NEDA, n.d). SWED consists of 11 questions (see Appendix D.1), both multiple-choice and open-ended, covering demographics, height, and weight, ED behaviors, weight and shape concerns, and impairment.

We focus on a subset of SWED questions and evaluate responses using four key criteria Fitzsimmons-Craft et al. (2018): C1, C2, C3, and C4. These items indicate a higher risk of EDs when the score is elevated (C1) or when being true (C2, C3, C4). For details, see Appendix D.2 and D.3.

Screening Online Communities via Finetuned LLMsWe prompt finetuned LLMs to respond to questions on the SWED screener. To account for randomness, for each item on the SWED questionnaire, the finetuned LLM generates 50 responses. The responses (Table 9 in Appendix D.5) are aggregated using a majority vote for each question. The results, as presented in Table 2, indicate that the _Pro Eating Disorder_ community exhibits the highest levels of body image concerns, followed by the _Keto & Diet_ community. Furthermore, both communities meet all three criteria signaling a high risk of ED pathology, whereas responses of the _Anti Eating Disorder_ community are consistent with a low risk of ED.

These findings align with our empirical observations. The content shared by the _Pro Eating Disordered_ community glorifies thinness and includes tips to promote disordered behaviors and body dysmorphin. Conversely, the _Anti Eating Disorder_ community is critical of the diet culture and people who glorify EDs. The relatively high-risk score of the _Keto & Diet_ community is a concerning indicator that this community may serve as a gateway to EDs. Experts caution that the keto diet's emphasis on restrictive dieting eliminating carbs could expose vulnerable individuals to binge eating disorder and anorexia Kelvas (2023); Polivy and Herman (1985). In contrast, the Body Image community, which mostly posts about body positivity, has a low risk of EDs, as does the Healthy Lifestyle & Weight Loss community. Although the latter focuses on weight loss, it appears to achieve this goal through healthy behaviors.

## 7 Conclusion

We demonstrate that aligning LLMs to online communities helps create high-fidelity digital proxies, which can be queried to reveal the implicit mindsets of these communities. When applied to online diet and body image communities, the method uncovers communities with unhealthy body image and dieting beliefs that put their members at risk of eating disorders. This is important, as harmful communities that indoctrinate users into extreme ideologies Schmitz et al. (2022) or glorify eating disorders and self-harm Goldenberg et al. (2022)

\begin{table}
\begin{tabular}{c|c c c c} \hline
**Community** & **C1** & **C2** & **C3** & **C4** \\ \hline Pro Eating Disorder & **45.0** & T & T & T \\ Keto \& Diet & 33.3 & T & T & T \\ Weight Loss Drugs & 16.7 & F & F & T \\ Body Image & 15.0 & F & F & F \\ Healthy Lifestyle \& Weight Loss & 13.3 & T & F & F \\ Anti Eating Disorder & 13.3 & F & F & F \\ \hline \end{tabular}
\end{table}
Table 2: Eating disorder risk assessment on the fine-tuned LLMs for different communities, using four criteria–C1 through C4. For C1, a higher score indicates a higher risk of an ED. For C2, C3, and C4, being positive implies higher risk.

Figure 5: Distribution of the three fine-grained harm categories



often evade moderation by using coded language that is opaque to outsiders or obfuscate harmful content via coded language and misspellings (Cancellor et al., 2016; Cobb, 2017; Bickham et al., 2024). As social data are increasingly abundant, our method is generalizable to study various online communities in different fields and can assist on-line platforms in overcoming challenges to foster safe and supportive environments.

### Limitations

Dataset Bias.The anonymized version of our dataset may contain implicit biases reflecting societal prejudices or unequal representation of demographic subgroups. More specifically, ED symptoms have a history of being under-diagnosed in African American and Hispanic adolescents, in part due to stereotypical representation of ED being Caucasian adolescent girls (Gordon et al., 2002). This historical bias could be inadvertently learned by our model, resulting in discriminatory behavior. In our future work, we hope to evaluate the model's fairness across different user groups, allowing us to properly mitigate dataset biases.

Evolving Nature of Online Communities.Capturing the evolving nature of online communities is potentially difficult. Online discourse is dynamic, with language, topics, and sentiments shifting over time. Our finetuning process may not fully account for these temporal changes, which could result in misalignment when the model is applied to current discussions within the community.

Synthetic Corpus Artifacts.The synthetic corpus generated by the LLM might also introduce artifacts that do not fully represent the authentic discourse of the community. Although we strive for diversity in the generated content, the model's predictions on previously unseen topics may not always accurately reflect how community members would engage with those topics in real-life scenarios.

Evaluation Metrics.While the aspects of authenticity, emotional tone, toxicity, and harm capture important aspects of online communication, they may not encompass all the subtle and complex features of human discourse. As a result, some aspects of community interaction may be underrepresented or overlooked in our evaluation process.

Low Inter-Annotator Agreement for ED Harm Categorization.The annotators in our harm categorization achieve a Cohen's Kappa score from 0.384 to 0.519, which indicates fair to moderate agreement (Landis, 1977). Since no prior work has specifically focused on categorizing harmful ED content, we develop a harm taxonomy in collaboration with psychologists and clinicians specializing in eating disorders. This ongoing process has introduced uncertainty in defining some categories, leading to annotation discrepancies. Additionally, content discussing eating disorders on social media can be nuanced and implicit--while some posts may appear benign, they can subtly normalize harmful behaviors or contain triggering details for vulnerable users, further complicating annotation.

Complete Coverage of Eating Disorders.This paper looks at the discussions of ED in online communities. We focus on a conglomeration of ED, including bulimia nervosa, anorexia nervosa, and binge eating disorder. Besides ED, our dataset captures other discussions related to weight concerns, such as weight loss, diet, body positivity, etc. Unfortunately, our data does not comprehensively represent all existing ED. However, our methods ensure that if a large ED community has some overlap with our keyword list, the community will be identified.

## Ethics Statement

Risk of Finetuning Models Towards HarmIn our study, we expect the finetuned LLMs to replicate harmful narratives from online communities. This is conducted solely to demonstrate that, through our alignment framework, LLMs can accurately capture the nuanced and authentic language of these communities, including harmful content. Our objective is not to create models with malicious intent; however, we acknowledge the potential scenario of vicious actors exploiting this framework to extract, amplify, and regenerate harmful information from social data. To mitigate this risk, we will release our code only upon eligible and transparent requests, ensuring that it is shared responsibly with researchers who have legitimate purposes and adhere to strict ethical and legal guidelines. We strongly advise that any future replication of this work be conducted with the utmost caution to prevent misuse and protect against the spread of harmful content 

Individual-Level DiagnosisExisting computational frameworks that diagnose or predict mental illness based on individuals' online activity and content raise significant ethical concerns due to the lack of user consent. These approaches often analyze personal data--even when publicly accessible--and infer sensitive individualized information like mental well-being without explicit permission from the users. By aggregating users' data for community-level diagnosis, we can address these privacy concerns without infringing on individual autonomy, allowing for valuable insights to inform community-wide policy creation.

Community-Level Diagnosis.Diagnosing psychiatric illness at the community level comes with the risk of falsely diagnosing some community members. This could lead to unjust actions against users, such as unwarranted bans or removal of content. Furthermore, psychological profiling of online communities sets a precedence for a slippery slope of increasingly intrusive monitoring and potentially creates a chilling effect on free speech in these spaces. Community members will anticipate and normalize heavy surveillance and thus self-censor or withdraw entirely from community discussions due to fear of revealing sensitive information that could lead to unintended consequences such as involuntary interventions. This would severely harm mental health online spaces by undermining the core values of trust and community. Additionally, approximating community behavior inherently excludes minority group members. Simultaneously, anorexia is one of the deadliest mental health disorders5 and participation in online pro-ED spaces heightens one's disease risk (Mento et al., 2021). By evaluating psychiatric illness on the community level, we can identify toxic communities, helping content moderation experts deploy proper interventions to promote healthy and safe online environments. We encourage the use of human moderators to review and validate the decisions made by our model, particularly in cases with low confidence scores.

Footnote 5: [https://www.state.sc.us/dmh/anorexia/statistics.htm](https://www.state.sc.us/dmh/anorexia/statistics.htm)

Topic Sensitivity and PrivacyThe sensitive nature of our topic means that our outputs could be misused, such as targeted advertising. Additionally, our dataset includes some tweets that disclose deeply personal information such as medical diagnoses, weight information, and personal struggles. Many of these tweets are posted under the assumption of anonymous identity. By collecting these tweets, user-specific information may be pieced together thus de-anonymizing some users. For these reasons, we take precautions to anonymize the social media posts before feeding them to the language models. Additionally, researchers can be granted access to generated tweets upon detailed inquiry.

Hallucination Risk.Our finetuned models can exhibit hallucinations, generating incorrect or non-sensical information. Hallucination in the context of community alignment can lead to community misrepresentation. In future work, we hope to utilize some factual-based evaluation datasets to measure model hallucination.

## Acknowledgements

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112290021 and Air Force Office of Scientific Research (AFOSR) under grant No. FA9550-23-1-0551. The authors are grateful to Ellen Fitzsimmons-Kraft for providing access to the eating disorders screener and to Aryan Karnati for aid with data analysis.

## References

* Bickham et al. [2024] Charles Bickham, Kia Kazemi-Nia, Luca Luceri, Kristina Lerman, and Emilio Ferrara. 2024. Hidden in plain sight: Exploring the intersections of mental health, eating disorders, and content moderation on itktok. _Preprint_, arXiv:2404.15457.
* Blondel et al. [2008] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. _Journal of Statistical Mechanics: Theory and Experiment_, 2008(10):P10008.
* Brytek-Matera and Schiltz [2011] Anna Brytek-Matera and Lony Schiltz. 2011. Association between attitudes towards body image, negative emotions about one's own body and self-state representations in a clinical sample of eating disordered women. _Archives of Psychiatry and Psychotherapy_, 2:37-43.
* Chancellor et al. [2016a] Stevie Chancellor, Zhiyuan Lin, and Munmun De Choudhury. 2016a. "this post will just get taken down" characterizing removed pro-eating disorder social media content. In _CHI_, pages 1157-1162.
* Chancellor et al. [2016b] Stevie Chancellor, Zhiyuan Lin, Erica L Goodman, Stephanie Zerwas, and Munmun De Choudhury. 2016b. Quantifying and predicting mental illness 

severity in online pro-eating disorder communities. In _CSCW_, pages 1171-1184.
* Chancellor et al. (2016c) Stevie Chancellor, Tanushree Mitra, and Munmun De Choudhury. 2016c. Recovery amid pro-anorexia: Analysis of recovery in social media. In _Proceedings of the 2016 CHI conference on human factors in computing systems_, pages 2111-2123.
* Chancellor et al. (2016d) Stevie Chancellor, Jessica Annette Pater, Trustin Clear, Eric Gilbert, and Munmun De Choudhury. 2016d. # thyghgapp: Instagram content moderation and lexical variation in pro-eating disorder communities. In _Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing_, pages 1201-1213.
* 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5.
* Cobb (2017) Gemma Cobb. 2017. "this is not pro-ana": Denial and disguise in pro-anorexia online spaces. _Fat Studies_, 6(2):189-205.
* Coda-Forno et al. (2023) Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. 2023. Inducing anxiety in large language models increases exploration and bias. _arXiv preprint arXiv:2304.11111_.
* Cuccolo et al. (2022) Kelly Cuccolo, Rachel Kramer, Thomas Petros, and McKena Thoennes. 2022. Intermittent fasting implementation and association with eating disorder symptomatology. _Eating disorders_, 30(5):471-491.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishekadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Durmus et al. (2023) Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the representation of subjective global opinions in language models. _arXiv preprint arXiv:2306.16388_.
* El Saddik et al. (2019) Abdulmotaleb El Saddik, Hawazim Badawi, Roberto Alejandro Martinez Velazquez, Fedwa Laamatri, Rogelio Gamez Diaz, Namrata Bagaria, and Juan Sebastian Arteaga-Falconi. 2019. Dtwins: A digital twins ecosystem for health and well-being. _IEEE COMSOC MMTC Commun. Front_, 14(2):39-43.
* Elran-Barak et al. (2015) Roni Elran-Barak, Maya Sztainer, Andrea Goldschmidt, Scott Crow, Carol Peterson, Laura Hill, Ross Crosby, Pauline Powers, James Mitchell, and Daniel Grange. 2015. Dietary restriction behaviors and binge eating in anorexia nervosa, bulinia nervous and binge eating disorder: Trans-diagnostic examination of the restraint model. _Eating Behaviors_, 18.
* Ferdousi et al. (2022) Rahatara Ferdousi, Fedwa Laamatri, M Anwar Hossain, Chunsheng Yang, and Abdulmodelabel El Saddik. 2022. Digital twins for well-being: an overview. _Digital Twin_, 1:7.
* Fitzsimmons-Craft et al. (2018) Ellen Fitzsimmons-Craft, Marie-Laure Firebaugh, Andrea Graham, Dawn Eichen, Grace Monterubio, Katherine Balantekin, Anna Karam, Annie Seal, Burkhard Funk, C. Taylor, andDenise Wilfley. 2018. State-wide university implementation of an online platform for eating disorders screening and intervention. _Psychological Services_, 16.
* Ganson et al. (2022) Kyle T Ganson, Kelly Cuccolo, Laura Hallward, and Jason M Nagata. 2022. Intermittent fasting: Describing engagement and associations with eating disorder behaviors and psychopathology among canadian adolescents and young adults. _Eating behaviors_, 47:101681.
* Gang and Garvey (2018) Debbiie Gang and Sarah Garvey. 2018. 'written in these scars are the stories i can't explain': A content analysis of pro-ana and thinspiration image sharing on instagram. _New Media & Society_, 20:1181-1200.
* Goldenberg et al. (2022) Alex Goldenberg, John Farmer, Lee Jussim, Loree Sutton, Danit Finkelstein, Cristian Ramos, Pamela Paresky, and Joel Finkelstein. 2022. Online communities of adolescents and young adults celebrating, glorifying, and encouraging self-harm and suicide are growing rapidly on twitter. Technical report, NCRI.
* Gordon et al. (2002) Kathryn H Gordon, Marisol Perez, and Thomas E Joiner Jr. 2002. The impact of racial stereotypes on eating disorder recognition. _International Journal of Eating Disorders_, 32(2):219-224.
* Gordon (1987) Randall A Gordon. 1987. Social desirability bias: A demonstration and technique for its reduction. _Teaching of Psychology_, 14(1):40-42.
* Graham et al. (2019) AK Graham, M Trockel, H Weisman, EE Fitzsimmons-Craft, KN Balantekin, DE Wilfley, and CB Taylor. 2019. A screening tool for detecting eating disorder risk and diagnostic symptoms among college-age women. _Journal of American College Health_, 67(4):357-366. Epub 2018 Oct 9.
* Grives (2011) Michael Grivees. 2011. _Virutually perfect: driving innovative and lean products through product lifecycle management_, volume 11. Space Coast Press Cocoa Beach.
* Grossmann et al. (2023) Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. 2023. Ai and the transformation of social science research. _Science_, 380(6650):1108-1109.
* Hanu and team (2020) Laura Hanu and Unitary team. 2020. Detoxify. Github. [https://github.com/unitaryai/detoxify](https://github.com/unitaryai/detoxify).
* He et al. (2024a) Zihao He, Minh Duc Chu, Rebecca Dorn, Siyi Guo, and Kristina Lerman. 2024a. Community-cross-invstruct: Unsupervised instruction generation for 

aligning large language models to online communities. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 17001-17019, Miami, Florida, USA. Association for Computational Linguistics.
* He et al. (2024b) Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. 2024b. Whose emotions and moral sentiments do language models reflect? In _Findings of the Association for Computational Linguistics: ACL 2024_, pages 6611-6631, Bangkok, Thailand. Association for Computational Linguistics.
* He et al. (2024c) Zihao He, Jonathan May, and Kristina Lerman. 2024c. Cpl-novid: Context-aware prompt-based learning for norm violation detection in online communities. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 18, pages 569-582.
* He et al. (2024d) Zihao He, Ashwin Rao, Siyi Guo, Negar Mokhberian, and Kristina Lerman. 2024d. Reading between the tweets: Deciphering ideological stances of interconnected mixed-ideology communities. _arXiv preprint arXiv:2402.01091_.
* Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. _Advances in neural information processing systems_, 30.
* Jiang et al. (2022a) Guangyuan Jiang, Manjie Xu amd Song-Chun Zhu, Wenjun Han, and Yixin Zhu Chi Zhang. 2022a. Evaluating and inducing personality in pre-trained language models. _arXiv preprint arXiv:2206.07550_.
* Jiang et al. (2022b) Hang Jiang, Doug Beeeferman, Brandon Roy, and Deb Roy. 2022b. Communitylm: Probing partisan worldviews from language models. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 6818-6826.
* Jiang et al. (2022c) Hang Jiang, Doug Beeeferman, Brandon Roy, and Deb Roy. 2022c. CommunityLM: Probing partisan worldviews from language models. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 6818-6826, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Juarasico et al. (2010) Adrienne S Juarasico, Amber Shoaib, and C Alix Timko. 2010. Pro-eating disorder communities on social networking sites: a content analysis. _Eating disorders_, 18(5):393-407.
* Kast (2018) Hinde Kast. 2018. The unspoken power of toxic words on body image. Master's thesis, University of Southern California.
* Kelvas (2023) Danielle Kelvas. 2023. Anorexia nervosa ketoacidosis symptoms. [https://withinhealth.com/learn/articles/anorexia-nervosa-ketoacidosis-symptoms](https://withinhealth.com/learn/articles/anorexia-nervosa-ketoacidosis-symptoms). Accessed on [10/11/2023].
* Killen et al. (1994) JD Killen, C Hayward, DM Wilson, CB Taylor, LD Hammer, I Litt, B Simmonds, and F Haydel. 1994. Factors associated with eating disorder symptoms in a community sample of 6th and 7th grade girls. _International Journal of Eating Disorders_, 15(4):357-367.
* Killen et al. (1996) Joel Killen, C Taylor, Chris Hayward, Katherine Haydel, Lawrence Hammer, Helena Kraemer, Anne Blair-Greiner, and Diane Strachowski. 1996. Weight concerns influence the development of eating disorders: A 4-year prospective study. _Journal of consulting and clinical psychology_, 64:936-40.
* Killen et al. (1993) Joel D. Killen, C. Barr Taylor, Lawrence D. Hammer, Iris Litt, Darrell M. Wilson, Tia Rich, Chris Hayward, Beverly Simmonds, Helen Kraemer, and Ann Varady. 1993. An attempt to modify unhealthful eating attitudes and weight regulation practices of young adolescent girls. _International Journal of Eating Disorders_, 13(4):369-384.
* Kour et al. (2022) George Kour, Samuel Ackerman, Eitan Farchi, Orna Raz, Boaz Carmeli, and Ateret Analyo-Tavor. 2022. Measuring the measuring tools: An automatic evaluation of semantic metrics for text corpora. In _Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2022)_. Association for Computational Linguistics.
* Landis (1977) JR Landis. 1977. The measurement of observer agreement for categorical data. _Biometrics_.
* Lerman et al. (2023a) Kristina Lerman, Aryan Karnati, Shuchan Zhou, Siyi Chen, Sudesh Kumar, Zihao He, Joanna Yau, and Abigail Horn. 2023a. Radicalized by thinness: Using a model of radicalization to understand pro-anorexia communities on twitter. _arXiv preprint arXiv:2305.11316_.
* Lerman et al. (2023b) Kristina Lerman, Aryan Karnati, Shuchan Zhou, Siyi Chen, Sudesh Kumar, Zihao He, Joanna Yau, and Abigail Horn. 2023b. Radicalized by thinness: Using a model of radicalization to understand pro-anorexia communities on twitter. _Preprint_, arXiv:2305.11316.
* Lu et al. (2023) Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang. 2023. Illuminating the black box: A psychometric investigation into the multifaceted nature of large language models. _Preprint_, arXiv:2312.14202.
* McCormack (2010) Abby McCormack. 2010. Individuals with eating disorders and the use of online support groups as a form of social support. _CIN: Computers, Informatics, Nursing_, 28:12-19.
* Mento et al. (2021) Carmela Mento, Maria Catena Silvestri, Maria Rosaria Anna Muscatello, Amelia Rizzo, Laura Celebre, Martina Pratico, Rocco Antonio Zoccali, and Antonio Bruno. 2021. Psychological impact of pro-anorexia and pro-eating disorder websites on adolescent females: A systematic review. _International Journal of Environmental Research and Public Health_, 18:2186 

* NEDA (2019) NEDA. 2019. Eating disorder online screening tool. [https://www.nationaleatingdisorders.org/screening-tool/](https://www.nationaleatingdisorders.org/screening-tool/). [Online; accessed 14-May-2024].
* NEDA (2020) NEDA. n.d. Eating disorder statistics & research.
* Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 9-14.
* Oksanen et al. (2016) Atte Oksanen, David Garcia, and Pekka Rasanen. 2016. Proanorexia communities on social media. _Pediatrics_, 137(1).
* Olad and Valilai (2020) Abdiladif Ahmed Olad and O Fatahi Valilai. 2020. Using of social media data analytics for applying digital twins in product development. In _2020 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM)_, pages 319-323. IEEE.
* Pater et al. (2016) Jessica A Pater, Oliver L Haimson, Nazanin Andalibi, and Elizabeth D Mynatt. 2016. "Inuger hurts but starving works" characterizing the presentation of eating disorders online. In _CSCW_, pages 1185-1200.
* Pellert et al. (2024) Max Pellert, Clemens M. Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. _Perspectives on Psychological Science_, 0(0):17456916231214460. PMID: 38165766.
* Pereira and Alvarenga (2007) Raquel Pereira and Marle Alvarenga. 2007. Disorderedeating: Identifying, treating, preventing, and differentiating it from eating disorders. _Diabetes Spectrum_, 20.
* Polivy and Herman (1985) Janet Polivy and C Peter Herman. 1985. Dieting and binging: A causal analysis. _American psychologist_, 40(2):193.
* Prescott et al. (2019) Julie Prescott, Terry Hanley, and Katalin Ujhelyi Gomez. 2019. Why do young people use online forums for mental health and emotional support? benefits and challenges. _British Journal of Guidance & counselling_, 47(3):317-327.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Rajadesingan et al. (2020) Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. 2020. Quick, community-specific learning: How distinctive toxicity norms are maintained in political subreddits. In _Proceedings of the International AAAI Conference on Web and Social Media_, volume 14, pages 557-568.
* Rossetti et al. (2020) Giulio Rossetti, Massimo Stella, Remy Czabet, Katherine Abramski, Erica Sau, Salvatore Citraro, Andrea Failla, Riccardo Improta, Virginia Morini, and Valentina Panasanella. 2024a. Y social: an llm-powered social media digital twin. _arXiv preprint arXiv:2408.00818_.
* Rossetti et al. (2024) Giulio Rossetti, Massimo Stella, Remy Czabet, Katherine Abramski, Erica Sau, Salvatore Citraro, Andrea Failla, Riccardo Improta, Virginia Morini, and Valentina Panasanella. 2024b. Y social: an llm-lpwered social media digital twin. _Preprint_, arXiv:2408.00818.
* Salinas and Morstatter (2024) Abel Salinas and Fred Morstatter. 2024. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. In _Annual Meeting of the Association for Computational Linguistics_.
* Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cino Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In _International Conference on Machine Learning_, pages 29971-30004. PMLR.
* Schmitz et al. (2022) Matheus Schmitz, Goran Muric, and Keith Burghardt. 2022. Quantifying how hateful communities radicalize online users. In _ASONAM_, pages 139-146. IEEE.
* Serapio-Garcia et al. (2023) Greg Serapio-Garcia, Mustafa Safdari, Clement Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Mataric. 2023. Personality traits in large language models. _Preprint_, arXiv:2307.00184.
* Sharma et al. (2024) Ashish Sharma, Kevin Rushton, Inna Wangvin Lin, Theresa Nguyen, and Tim Althoff. 2024. Facilitating self-guided mental health interventions through human-language model interaction: A case study of cognitive restructuring. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_, pages 1-29.
* Shengli (2021) Wei Shengli. 2021. Is human digital twin possible? _Computer Methods and Programs in Biomedicine Update_, 1:100014.
* Sheth et al. (2022) Amit Sheth, Valerie L Shalin, and Ugur Kursuncu. 2022. Defining and detecting toxicity on social media: context and knowledge are key. _Neurocomputing_, 490:312-318.
* Simmons and Hare (2023) Gabriel Simmons and Christopher Hare. 2023. Large language models as subpopulation representative models: A review. _arXiv preprint arXiv:2310.17888_.
* Sanchez et al. (2024) Cinthia Sanchez, Minh Duc Chu, Zihao He, Rebecca Dorn, Stuart Murray, and Kristina Lerman. 2024. Feelings about bodies: Emotions on diet and fitness forums reveal gendered stereotypes and body image concerns. _Preprint_, arXiv:2407.03551.
* Tanmay et al. (2023) Kumar Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and Monojit Choudhury. 2023. Probing the moral development of large language models through defining issues test. _Preprint_, arXiv:2309.13356 

* [Tao and Qi2019] Fei Tao and Qinglin Qi. 2019. Make more digital twins. _Nature_, 573(7775):490-491.
* Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Taylor et al. [2006] C Taylor, Susan Bryson, Kristine Luce, Darby Cunningham, Angela Doyle, Liana Abascal, Roxanne Rockwell, Parvati Dev, Andrew Winzelberg, and Denise Wilfley. 2006. Prevention of eating disorders in at-risk college-age women. _Archives of general psychiatry_, 63:881-8.
* Yeshua-Katz and Martins [2013] Daphna Yeshua-Katz and Nicole Martins. 2013. Communicating stigma: The pro-ana paradox. _Health Communication_, 28(5):499-508.

## Appendix A Online Communities in ED Discussions

### Search Terms

The terms used for tweet collection are: _antips, bodygoals, bodyimage, bodypositivity, chloetingchallange, cleaneating, cleanupgan, eatingdisorder, edrecovery, edwt, edvent, fatspo, fearfood, foodistheenemy, healthyliving, intermittentfasting, willlbeskinny, juicceleanse, ketodiet, losingweight, lowcalcrestriction, meangpo, midriff, oempic, proana, proanatis, redracetpro, semaglutide, skinyneckel, slimmingworld, sweetspo, thighgapworkout, thinspo, hinspoa, watercleanse, weigovy, weightlossjourney, weightlossmotivation, whateatinaday, bonespo, fatacceptance, keto, primia, skinnydiet, dietculture, m34nspo, weightloss, weightlossips._

Disordered eating behaviors exist along a spectrum between normal eating patterns and clinically diagnosable eating disorders Pereira and Alvarenga (2007). Previous studies have shown that restrictive diets, such as keto ("ketodiet") or intermittent fasting ("intermittentfasting"), often intended for weight loss, are linked to a heightened risk of developing EDs Elran-Barak et al. (2015); Ganson et al. (2022); Cuccolo et al. (2022). While these behaviors may not meet clinical diagnostic criteria, they can act as a gateway, steering individuals toward more harmful online communities that promote disordered eating Lerman et al. (2023b). Additionally, we wanted to explore discussions from the opposite perspective ("bodypositivity"), where critics of diet culture and advocates for positive body image and healthy eating raise their voices.

Below are the explanations of these keywords used in the context of the online ED community:

* chloroquinechallenge: a popular fitness trend created by YouTube Chloe Ting.
* edtwt: refers to the general ED community on Twitter/X.
* fatspo: promotes body positivity and acceptance of larger body sizes.
* feafood: a term for foods that cause anxiety or avoidance in those with ED.
* rebracketpro: refers to the bracelet patients wear at a treatment facility when they are medically unstable or fragile.
* meanspo, m34nspo: be deliberately mean or insulting to motivate someone to do something.
* midriff: refers to the area of the body between the chest and the waist. It often shows one's ribcage and is closely associated with being skinny.
* ozempic, wegovy, semaglitude: refers to a medication primarily used to treat type 2 diabetes but has gained attention for its use as a weightloss drug
* thighgapworkout: refers to exercises aimed at achieving a gap between the thighs, a controversial and unrealistic body goal often associated with unhealthy body image standards.
* thinspo: short for "thinspiration," referring to content or imagery that promotes extreme thinness.
* bonespo: refers to content that glorifies extreme thinness by focusing on images of prominent bones.
* promia: the promotion of bulimia-related behaviors, often found in harmful online communities.

### Profiling Communities

The statistics of the top 20 largest user clusters detected by Louvain modularity maximization are shown in Table 3. The word clouds of tweets in these 20 clusters are shown in Figure 6. The retweet network, with users from different clusters showing different colors, is shown in Figure 7.

To profile discussions, we provide a random sample of 200 posts from each user cluster to GPT-4 

with the prompt: "_Given this list of posts, summarize the main ideas in 1 sentence_". We observe that using different random samples of posts leads to substantially similar summaries. After reviewing generated summaries, we note significant thematic and content overlaps and group the clusters based on their common topics of discussion into clusters: _Pro-ED_, _Keto & Diet_, _Body Image_, _Anti-ED_, _Healthy Lifestyle & Weight Loss_, _Weight Loss Drugs_, and _spam_ (not included).

Members of clusters 0, 7, 8, and 9 use "edtwt" to self-identify as part of the ED community, and their posts promote disordered behaviors. Interestingly, members of clusters 8 and 9 post in Spanish and Portuguese, respectively. They are also placed close to pro-ED clusters 0, 7 in Figure 7. Cluster 2, although also uses "edtwt" label, is well separated from the rest. This cluster takes a critical--anti-ED--stance on ED, as seen from the summary in Table 4.

The remaining clusters are loosely connected in the retweet network and less insular than the pro-ED cluster. Clusters 1, 15, 16, 18 discuss the risks and benefits of the keto diet; clusters 3, 6 and 19 focus on issues surrounding the use of weight loss drugs like Ozempic and Wegovy; Clusters 4, 13, 17 examine issues of healthy lifestyle and weight loss, while clusters 5, 10 cover body image topics, like body positivity and self-acceptance. Clusters 11, 12, and 14 are on other random issues not relevant to ED, as can be observed from word clouds and thus we exclude them in our subsequent analysis.

## Appendix B Aligning LLMs

### Demonstration Template for LLM finetuning

The instructions for finetuning LLMs are shown in Table 5. For tweet generation demonstrations, each tweet is paired with a randomly sampled instruction from the table. An example prompt template is shown below. More demonstrations for different communities are shown in Table 6.

Instruction: What would you tweet? Response: {Tweet}

## Appendix C Assessing Alignment

### Topics for Creating Synthetic Tweets

The 27 topics used for creating the synthetic tweets are: _thinspo_, _fitspo_, _bonespo_, _deathspo_, _caloric restriction_, _meaning_, _ozempic_, _wegovy_, _fatspo_, _fatphobia_, _highgap_, _caloric counting_, _purging_, _food rules_, _extreme diet_, _food fear_, _hiding food_, _fasting_, _starving_, _steroid_, _excessive exercising_, _body dysmorphia_, _working out_, _anorexia_, _bulimia_, _orthorexia_, _binge eating_.

The number of tweets mentioning the topics for each community is shown in Table 7.

### Prompt Template for Tweet Generation by LLM-Context

An example prompt template is shown below. You're part of an online community now. To help you describe this online community, here are the tweets made by members in this community about the topic of {topic}. Tweet 1: {tweet_1} Tweet 2: {tweet_1}  ... Tweet 250: {tweet_250} What would you tweet about {topic}? Learn the ideas and mindset of the community from these tweets and speak like a member from this community. Only generate one tweet.

### Demonstration Template for Tweet Origin Classification

Instruction: From these communities: Pro Eating Disorder, Keto & Diet, Body Image, Anti Eating Disorder, Healthy lifestyle & Weight Loss, and Weight Loss Drugs, which community does this Tweet belong to? {Tweet} Response: {community_name}

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline
**Comm** & **0** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** \\ \hline \# of users & 61,954 & 24,400 & 21,887 & 20,631 & 9,901 & 9,031 & 9,000 & 8,084 & 7,702 & 7,020 \\ \# of tweets & 805,249 & 112,674 & 32,883 & 37,788 & 193,348 & 24,395 & 21,369 & 82,702 & 70,764 & 71,970 \\ \hline
**Comm** & **10** & **11** & **12** & **13** & **14** & **15** & **16** & **17** & **18** & **19** & **total** \\ \hline \# of users & 6,477 & 6,158 & 5,181 & 4,528 & 3,682 & 3,672 & 3,360 & 3,163 & 3,086 & 2,865 & 221,887 \\ \# of tweets & 15,796 & 9,254 & 7,019 & 103,177 & 260,971 & 5,338 & 4,881 & 5,065 & 4,612 & 7,021 & 1,876,276 \\ \hline \end{tabular}
\end{table}
Table 3: Number of users (community size) and tweets in the top 20 largest user clusters respectively and in total.

 



### LLM Tweet Generation

Table 8 shows examples of LLM generated tweets. These examples across different communities and topics demonstrate that the finetuned LLM generates tweets more aligned with the assigned topic, effectively capturing the community's consensus, norms, and linguistic patterns. In contrast, the LLM-Context-generated tweets show less specificity and coherence with the community's established discourse. This highlights the finetuned model's superior ability to reflect the language and cultural context of the target group.

Figure 8 shows the distribution of ROUGE-L scores between tweets in a community's synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\) and their most similar tweets within the corpus. Figure 9 shows the distribution of perplexity scores of tweets in a community's synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\). Figure 10 shows the distribution of ROUGE-L scores between tweets in a community's synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\) and their most similar tweets in the community's original corpus \(D_{i}\).

## Appendix D Screening Online Communities

### Stanford-Washington University Eating Disorder (SWED) 3.0 Screener

The 11 questions in the questionnaire are shown below.

1. Are you currently in treatment for an eating disorder? 1. No 2. Yes 3. Not currently, but I have been in the past
2. What was your lowest weight in the past year, including today, in pounds?
3. What is your current weight in pounds?
4. What is your current height in inches?
5. How much more or less do you feel you worry about your weight and body shape than other people your age? 1. I worry a lot less than other people 2. I worry a little less than other people 3. I worry about the same as other people 4. I worry a little more than other people 5. I worry a lot more than other people
6. How afraid are you of gaining 3 bounds? 1. Not afraid of gaining 2. Slightly afraid of gaining 3. Moderately afraid of gaining 4. Very afraid of gaining 5. Terrified of gaining
7. When was the last time you went on a diet? 1. I have never been on a diet 2. I was on a diet about one year ago 3. I was on a diet about 6 months ago 4. I was on a diet about 3 months ago

\begin{table}
\begin{tabular}{p{14.2pt}|p{142.3pt}|p{142.3pt}} \hline
**Community Tag** & \multicolumn{1}{c|}{**Summary of Community Discussions**} & **User Cluster ID** \\ \hline Pro Eating Disordering tips, hinsupo (thin inspiration), meango (mean inspiration), fasting strategies, and discussing body image and weight loss goals, often in a way that promotes disordered eating behaviors. & 0,7,8,9 \\ Keto \& Diet & This community focuses on a range of topics related to ketogenic diets, weight loss, metabolic health, and low-carb recipes, with discussions on the effectiveness of keto for various health conditions, debates on prescribing obesity drugs to children, and personal testimonials about the benefits of a keto. & 1,15,16,18 \\ Body Image & This community dives into a variety of personal updates, including fitness activities, body positivity, nudism, modeling, and social interactions, with some tweets promoting content or expressing motivational thoughts. & 5, 10 \\ Anti Eating Disordering & This community expresses strong negative sentiments towards “edtwt” (presumably ”eating disorder Twitter”), criticizing it for being toxic, fatphobic, and harmful, with calls to abolish it and stop interacting with its content. & 2 \\ Healthy Lifestyle \& Weight Loss & This community covers a variety of health and wellness topics, including weight loss methods, dietary plans, fitness advice, healthy eating, keto diet, fasting, moxibustion, and motivational messages for maintaining a healthy lifestyle. & 4,13,17 \\ Weight Loss Drugs & This community discusses the controversial use of the diabetes drug Ozempic for weight loss, the impact of its shortage on diabetic patients, the cost of the medication, and related topics such as body positivity, keto diets, and the role of influencers and celebrities in promoting certain health trends and products. & 3,6,19 \\ \hline  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 4: Summary of posts in the communities with GPT-4. Similar communities are merged.



* I was on a diet about 1 month ago
* I was on a diet less than 1 month ago
* I'm on a diet now
8. Compared to other things in your life, how important is your weight to you? 1. My weight is not important compared to other things in my life 2. My weight is a little more important than some other things 3. My weight is more important than most, but not all, things in my life 4. My weight is the most important thing in my life
9. Do you ever feel fat? 1. Never 2. Rarely 3. Sometimes 4. Often 5. Always
10. In the past 3 months, how many times have you had a sense of loss of control AND you also ate what most people would regard as an unusually large amount of food at one time, defined as definitely more than most people would eat under similar circumstances?
11. In the past 3 months, how many times have you done any of the following as a means to control your weight and shape: 1. Made yourself throw up? 2. Used diuretics or laxatives? 3. Exercised excessively? i.e. pushed yourself very hard; had to stick to a specific exercise schedule no matter what - for example even when you were sick/injured or if it meant missing a class or other important obligation; felt compelled to exercise 4. Fasted? i.e. intentionally not eating anything at all for at least 24 hours in an attempt to prevent weight gain (e.g., that is feared as a result of binge eating) or to lose weight
12. Have you experienced significant weight loss (or are at a low weight for your age and height) but are not overly concerned with the size and shape of your body? 1. Yes 2. No

\begin{table}
\begin{tabular}{c l} \hline \multicolumn{1}{l}{**Index**} & \multicolumn{1}{r}{**Instruction**} \\ \hline
**1** & What would you tweet? \\
**2** & What tweet would you send out? \\
**3** & What’s your tweet today? \\
**4** & What would you want to tweet about? \\
**5** & What’s on your mind to tweet? \\
**6** & What tweet would you drop? \\
**7** & What would you say? \\
**8** & What’s your tweet? \\
**9** & Tweet something. \\
**10** & Share your thought with a tweet. \\
**11** & What kind of tweet would you send out to engage with fellow members? \\
**12** & Draft a tweet that captures the interests and spirit of the community. \\
**13** & Craft a relatable tweet that resonates with members. \\
**14** & Share a tweet that sparks conversation on relevant topics. \\
**15** & Compose a tweet that reflects the shared voice and passions. \\
**16** & Author an insightful tweet that inspires dialogue among members. \\
**17** & Tweet something that provokes intellectual discourse. \\
**18** & Tweet an observation or perspective that contributes meaningfully. \\
**19** & Craft a tweet that elevates the ongoing conversations. \\
**20** & Compose a tweet that encourages enriching engagement. \\ \hline \end{tabular}
\end{table}
Table 5: Instructions used to finetune the LLMs.

 



### Weight Concerns Scale

The Weight Concerns Scale (WCS) [13] is a brief, validated psychometric instrument designed to measure concerns about weight and body shape, fear of gaining weight, dieting history, and feelings of fatness. The scale was created to assess risk factors for ED and has demonstrated excellent stability and sensitivity to treatment differences. Longitudinal studies have shown that high WCS scores predict the onset of ED, making it a valuable instrument for identifying at-risk individuals [14, 20, 9]. Our work deploys this questionnaire to gain insights into body-related concerns of members of online communities.

### Diagnostic Criteria

C1 is a numerical score, the Weight Concerns Scale (WCS) [15] - a metric that measures concerns about weight and shape. This criterion composes answers from questions 5 to 9 in SWED, maps the options to numerical values to a 0-100 scale, and calculates the average score across the 5 questions. The higher the WCS score is, the higher the risk factor for body and shape concerns.

C2, C3, and C4 are boolean values, and being true implies a higher risk of EDs. C2 is True if the model responds "more/most important" (options c or d) to Q8 "Compared to other things in your life, how important is your weight to you?". C3 is True if the model responds "moderately afraid" or more (options c, d, or e) to Q6: "How afraid are you of gaining 3 pounds?". C4 is True if the model responds "Yes" at least 3 times to Q11: "In the past 3 months, how many times have you done any of the following as a means to control your weight and shape: a. Made yourself throw up? b. Used diuretics or laxatives? c. Exercised excessively? d. Fasted?"

### Prompt Template for SWED Question Answering

{Question} Respond to the following question only with the letter at the beginning of each option or with a number.

### LLM Responses to SWED Questions

Responses from the finetuned LLMs to SWED questions are shown in Table 9.

\begin{table}
\begin{tabular}{|p{71.1pt}|p{142.3pt}|p{142.3pt}|} \hline
**Community** & **Instruction** & **Response** \\ \hline Pro-ED & What would you tweet? & most of the time the only thing i want in the whole world is to be skinny and lose weight \\ Keto \& Diet & What tweet would you send out? & ready to jumpstart your weight loss journey? Try these tips to help you lose weight in a month \\ Body Image & What’s your tweet today? & everyone has something about their body they ’re not completely happy with. Don’t focus on that! Love the body you have! \\ Anti-ED & What would you want to tweet about? & do not follow me if you’re on edtwt. I don’t know how many times I have to say this we don’t stop exercising because we grow old, we grow old because we stop exercising. \\ Height Lifestyle & What’s on your mind to tweet? & we don’t stop exercising because we grow old, we grow old because we stop exercising. \\ Weight Loss Drugs & What tweet would you drop? & are our keto diet pills effective and safe to use? The truth about keto diet pills benefits, risks, and effectiveness \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 6: Demonstration examples for LLM finetuning for different communities.





\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline
**Topic** & **Pro-ED** & \begin{tabular}{c} **Keto** \\ **and Diet** \\ \end{tabular} & \begin{tabular}{c} **Body** \\ **Image** \\ \end{tabular} & **Anti-ED** & \begin{tabular}{c} **Healthy lifestyle** \\ **and Weight Loss** \\ \end{tabular} & 
\begin{tabular}{c} **Weight Loss** \\ **Drugs** \\ \end{tabular} \\ \hline thinspo & 20 & 0 & 0 & 24 & 0 & 2 \\ \hline fitspo & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline bonespo & 4 & 0 & 0 & 0 & 0 & 0 \\ \hline deathspo & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline caloric restriction & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline calorie counting & 0 & 0 & 0 & 1 & 0 & 0 \\ \hline purging & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline food rules & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline extreme diet & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline food fear & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline hiding food & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline fasting & 0 & 1 & 0 & 1 & 0 & 2 \\ \hline starving & 1 & 0 & 1 & 1 & 0 & 1 \\ \hline steroid & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline meenspo & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline ozempic & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline wegovy & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline fatspo & 2 & 0 & 0 & 3 & 0 & 0 \\ \hline fatphobia & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline thigh gap & 4 & 0 & 0 & 0 & 0 & 0 \\ \hline excessive exercising & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline body dysmorphia & 0 & 0 & 1 & 1 & 0 & 0 \\ \hline working out & 1 & 2 & 2 & 0 & 0 & 1 \\ \hline anorexia & 0 & 0 & 0 & 2 & 0 & 0 \\ \hline bulimia & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline orthorexia & 0 & 0 & 0 & 0 & 0 & 0 \\ \hline binge eating & 1 & 0 & 0 & 0 & 0 & 0 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 7: Number of tweets mentioning topic keyword(s) from each community.





## Appendix A Proofs

### Proof of Theorem 1.2

Figure 6: Wordclouds of popular terms appearing in the original tweets posted within each user cluster.



Figure 7: Retweet network, where nodes are individual users and edges indicate the retweeting activities. Node colors represent different user clusters identified by the Louvain modularity method.

 

Figure 8: Distribution of ROUGE-L scores between tweets in a community’s synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\) and their most similar tweets within the corpus.

Figure 9: Distribution of perplexity scores of tweets in a community’s synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\).

 

Figure 10: Distribution of ROUGE-L scores between tweets in a community’s synthetic corpus \(D_{i}^{ft}\) or \(D_{i}^{context}\) and their most similar tweets in the community’s original corpus \(D_{i}\).

 



\begin{table}
\begin{tabular}{|l|p{142.3pt}|p{85.4pt}|p{76.9pt}|p{76.9pt}|} \hline
**Community** & **Topic** & **In-context** & **Finetuned** \\ \hline Anti Eating Disorder & body dysmorbia & some of yall on edtwt need to mind fkg business??? & edtwt is so bizarre 2 me; how are you making abt how ugly you think people are? dont you hate yourself? id worry abt first, \\ \hline Body Image & extreme diet & everyone has something about their body they’re not completely happy with don’t focus on that! love yourself, love your body, love your life. & im down to 15 stone 12!!!! so s proud of myself, i know its been a long journey but i kept going and now im reaping the rewards. change happens over time and patience is the key. \\ \hline Weight Loss Drugs & wegovy & what is the best diet for weight loss? too many diet rules doesn’t work! & wegovy has helped many overcome obesity and drop excess pounds. recently, the hashtag has been trending as a side effect of taking wegovy. however, is this a side effect of taking the medication or simply due to rapid weight loss? full article \\ \hline Eating Disorder & anorexia & when his celebrty crush is my thinspo and my celebry crush is his thinspo groush is his thinspo & im going to the doctors soon so im gon na have to lose some weight before i go \\ \hline \end{tabular}
\end{table}
Table 8: LLM generated outputs across different communities and topics.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline
**Question** & \begin{tabular}{c} **Pro** \\ **ED** \\ \end{tabular} & **Keto \& Diet** & **Body Image** & \begin{tabular}{c} **Anti** \\ **ED** \\ \end{tabular} & \begin{tabular}{c} **Healthy Lifestyle** \\ **\& Weight Loss** \\ **\end{tabular} & 
\begin{tabular}{c} **Weight Loss** \\ **Drugs** \\ \end{tabular} \\ \hline
5 & b & c & b & a & a & b \\ \hline
6 & c & c & a & c & a & b \\ \hline
7 & c & a & b & b & a & a \\ \hline
8 & c & c & b & a & c & b \\ \hline
9 & c & a & a & a & a & a \\ \hline
11a & c & a & a & c & a & a \\ \hline
11b & c & a & a & c & a & a \\ \hline
11c & a & b & b & b & b & a \\ \hline
11d & a & b & b & b & b & a \\ \hline \end{tabular}
\end{table}
Table 9: Responses from the finetuned LLMs to the questions in SWED that are used to compute the diagnosis criteria. The responses displayed are the majority of answers for each question.