Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts
Year: 2023

Abstract:
Many cognitive approaches to well-being, such as recognizing and reframing unhelpful thoughts, have received considerable empirical support over the past decades, yet still lack truly widespread adoption in self-help format. A barrier to that adoption is a lack of adequately specific and diverse dedicated practice material. This work examines whether current language models can be leveraged to both produce a virtually unlimited quantity of practice material illustrating standard unhelpful thought patterns matching specific given contexts, and generate suitable positive reframing proposals. We propose PATTERNREFRAME, a novel dataset of about 10k examples of thoughts containing unhelpful thought patterns conditioned on a given persona, accompanied by about 27k positive reframes. By using this dataset to train and/or evaluate current models, we show that existing models can already be powerful tools to help generate an abundance of tailored practice material and hypotheses, with no or minimal additional model training required.

heading: 1 Introduction
text: Cognitive Behavioral Therapy (CBT) (Beck, 1963, 1976) is one of the most robustly validated approaches in psychology (Hofmann et al., 2012; David et al., 2018). A core pillar of CBT consists in identifying and reframing unhelpful ways of thinking. Low-intensity CBT interventions have shown promise in self-help formats (Shafran et al., 2021; Williams, 2001), yet a lack of sufficient practice material suited to people’s specific circumstances is a barrier to adoption (Helgadóttir et al., 2009).
Through prompting, control tokens, or adequate conditioning, modern language models can guide generation of language towards desired outcomes, such as conforming to a given persona (Zhang et al., 2018), style (Ziems et al., 2022), or level of confidence (Mielke et al., 2022). This makes them a potentially powerful practice aid for learning cog-
nitive reframing techniques. A major barrier is the lack of publicly available data. Most existing work in natural language processing (NLP) for CBT focuses on interactions between patients and mental health professionals, which are not publicly available (Mieskes and Stiegelmayr, 2018; Rojas-Barahona et al., 2018; Shreevastava and Foltz, 2021). Ziems et al. (2022) released the first public dataset for reframing tweets marked with a hashtag indicating stress, using known reframing techniques, but it does not specifically look at the categories of unhelpful thinking used in CBT, and uses existing tweets rather than allowing the generation of examples suited to a particular situation.
In this work, we propose1 a novel dataset, PATTERNREFRAME, consisting in ∼10k crowdsourced examples of thoughts containing ten classical types of unhelpful thought patterns (Burns, 1980), conditioned on personas, matched with crowdsourced proposals of reframing that do not exhibit the patterns. We introduce two controllable text-to-text generation tasks on the dataset: (1) generating and (2) reframing unhelpful thoughts, given a persona and pattern as the context. We also define a classification task to identify the unhelpful thought pattern, given a persona and a thought. We train and evaluate different fine-tuned and few-shot approaches for the tasks, and show that these approaches perform reasonably well on the tasks.

heading: 2 Related Work

heading: 2.1 NLP for Mental Health
text: Recent work has used linguistic features and pretrained language models to identify mental health conditions such as anxiety (Owen et al., 2020; Shreevastava and Foltz, 2021; Fine et al., 2020),
1The dataset and task have been released through the ParlAI framework (Miller et al., 2017) and are available at https://github.com/facebookresearch/ ParlAI/tree/main/projects/reframe_ thoughts
13641
depression (Wolohan et al., 2018; Poświata and Perełkiewicz, 2022; Ji et al., 2022), schizophrenia (Jiang et al., 2020b; Mitchell et al., 2015; Sarioglu Kayi et al., 2017), and post-traumatic stress disorder (Coppersmith et al., 2015). Most of these works annotate social media posts to create datasets for the task, and then train and evaluate different classification models. Shreevastava and Foltz (2021) and Rojas-Barahona et al. (2018) created datasets for identifying unhelpful thoughts by annotating patient-therapist interactions and finetuned different pretrained models for the task. However, these datasets are not publicly available.
The closest work to ours is that of Ziems et al. (2022), which introduces a reframing task, releases a parallel corpus of reframed sentences, and uses controllable text generation models to reframe social media content from Twitter that was marked as expressing stress. However, the source social media material is not conditioned on personas, or focused on the classical unhelpful thought patterns from CBT. Our work introduces conditioning on personas and classical unhelpful thought patterns, and extends the reframing task to identifying and generating thoughts matching a given persona and unhelpful pattern.

heading: 2.2 Controllable Text Generation
text: Controllable text generation approaches using pretrained language models (PLMs) typically fall into four categories: (i) prompt-based methods that either construct templates for PLMs to complete (Jiang et al., 2020a; Schick and Schütze, 2021a,b) or finetune a task-specific layer to guide the generation (Li and Liang, 2021; Lester et al., 2021), (ii) finetuning methods that either use labelled data prepended with controlled attributes (Ziems et al., 2022; Fan et al., 2018; Martin et al., 2020; Ross et al., 2022) or define a task-specific reward function using reinforcement learning (Ziegler et al., 2019; Liu et al., 2020), (iii) post-processing methods that train discriminator models to guide the generation towards a specific criterion during decoding (Dathathri et al., 2019; Hua and Wang, 2020; Xu et al., 2020), and (iv) pretraining methods that pretrain PLMs from the start with different control tokens prepended to the input (Keskar et al., 2019). In our work, we experiment with prompt-based and finetuning methods.

heading: 3 Identifying and Reframing Unhelpful Thoughts
text: We use the ten categories of unhelpful thought patterns described in lay terms in a widely used CBT self-help book used for bibliotherapy (Burns, 1980). Table 1 lists these categories and provides examples for each category. For reframing unhelpful thoughts, we follow Ziems et al. (2022), who describe five reframing strategies based on positive psychology (Harris et al., 2007): (i) Growth Mindset: Focusing on learning from challenges and improving the skills needed to deal with a difficult situation; (ii) Optimism: Directing the attention towards the positive aspects of the situation and expressing gratitude while still acknowledging the negative aspects; (iii) Impermanence: Understanding that adversities are inevitable and temporary and focusing on accepting the situation; (iv) Neutralizing: Challenging unhelpful thoughts that are far from reality and replacing them with realistic neutral alternatives; (v) Self-affirmation: Reflecting on core values to ground oneself in a difficult situation. Note that other reframing strategies exist, such as “being mindful” (Robertson, 2012), or “focusing on forgiveness and compassion” (Gilbert, 2010). We provide the above five strategies only as a starting point, but crowd workers are free to use other strategies.

heading: 4 PATTERNREFRAME Dataset

heading: 4.1 Data Collection
text: We briefly explain the four-step data collection process used to crowdsource the dataset. We provide further data collection details and snapshots of the interface in Appendix A and B.

heading: 4.1.1 Task 1: Writing Unhelpful Thoughts
text: In order to generate unhelpful thoughts that match a diversity of contexts and situations, we use personas from the PERSONA-CHAT dataset (Zhang et al., 2018) as context for writing unhelpful thoughts. We give a persona and one of the ten unhelpful thought patterns to the crowdsource workers, and ask them to write sentences that both are consistent with the given persona, and exhibit the given unhelpful thought pattern.

heading: 4.1.2 Task 2: Categorizing Unhelpful Thoughts
text: Unhelpful thoughts can exhibit multiple patterns, and the patterns themselves are overlapping rather
than distinct (Burns, 1980). In order to capture this, as well as filter out low-quality crowdsourced data, we use a second crowdsourcing task requesting workers to label the previously generated thoughts. Workers are given a thought and the list of unhelpful patterns, and select all the patterns that appear in the thought. The annotators can choose a “None” option in case the thought is irrelevant or nonsensical. We collect five annotations for each thought, and discard the thoughts that are marked “None” by a majority of annotators.

heading: 4.1.3 Task 3: Reframing Unhelpful Thoughts
text: In a third task, we ask crowdworkers to rewrite thoughts containing unhelpful patterns, in a more helpful way, similar to the task in Ziems et al.
(2022). We give crowdworkers a thought and the persona and unhelpful pattern that were used to generate it, and ask them to rewrite the thought in a way that still aligns with the context, but does not contain the unhelpful pattern. We also show the five reframing strategies described in §3 to aid the workers in reframing the thoughts, and ask them to select what strategy they used, if any. Note that the strategies are only provided as suggestions, and the workers are free to reframe the thought in other appropriate ways. We collect three rewrites for each thought.

heading: 4.1.4 Task 4: Evaluating the Rewrites of Unhelpful Thoughts
text: Finally, we assess the quality of the rewrites as follows: workers are given a persona, unhelpful thought pattern, generated thought, along with three rewrites. They are asked to select which rewrites successfully remove the unhelpful pattern while not logically contradicting the source (following Ziems et al. (2022)). If worker selects a valid rewrite, we further ask them to identify which of the five proposed reframing strategies were used, if any. We collect five annotations for each set, and include only the rewrites that are marked as “valid” by a majority of annotators.

heading: 4.2 Data Quality
text: We use the Mephisto2 and Amazon Mechanical Turk3 platforms to collect crowdsource data. We use the labeling tasks (2nd and 4th task) to select a pool of high-quality workers (that is, crowdsource workers whose generative work was validated by a majority of separate annotators in a separate labeling task), after first seeding the set of annotators through manual inspection of a first batch of data. We use only selected annotators for evaluation tasks (tasks 2 and 4). We first kept the generative text tasks (tasks 1 and 3) open to all workers. We expanded the list of selected workers after every iteration by adding new workers that had completed at least five generative text tasks with at least 80% of generated text validated through the evaluation tasks. We ended up with 524 qualified workers after nine rounds of the entire pipeline, where each iteration started with a batch of 500 thoughts. Once we gathered > 500 qualified workers, we restricted all the tasks to the selected pool. In the final dataset, we included only the annotations provided by these selected workers.
Along with the selected pool of workers, we also included onboarding tasks (details in §A) to ensure that the workers adequately understood the concept of reframing thoughts. Only the workers who passed the onboarding tasks were qualified to work on the actual tasks. We calculated interannotator agreement using Krippendorf’s Alpha, which was 0.355 for the second task and 0.454 for the fourth task.4
2https://github.com/facebookresearch/Mephisto 3Our crowdsourcing tasks pay workers well above minimum wage. 4We compute Krippendorf’s Alpha for the binary patternlevel judgments from the the second task and the binary

heading: 4.3 Data Analysis

heading: 4.3.1 Dataset Statistics
text: PATTERNREFRAME contains 9,688 thoughts and 26,507 reframed versions of thoughts. We split the dataset into training, validation, and test sets of respective sizes 1,920 / 961 / 6,807 for thoughts, and 5,249 / 2,623 / 18,635 for reframed thoughts. One thought can have up to three reframed versions, with an average of 2.74 rewrites / thought after filtering out lower-quality rewrites. The average word lengths of thoughts and rewrites are 19.1 and 23.9, respectively.

heading: 4.3.2 Analysis of Unhelpful Thought Patterns
text: Figure 1 shows the distribution of thoughts across different patterns in our dataset, with initial conditioning pattern (1st task) in rows and annotator identified patterns (2nd task) in columns. As expected, there is a high overlap among some related patterns, e.g., Discounting the positive / Mental Filtering, Fortune Telling/ Catastrophizing, and Personalization / Labeling and Mislabeling. All or Nothing Thinking is difficult to distinguish, and shows high overlap with many categories. Mind Reading and Should Statement show the lowest amounts of overlap with other patterns.
reframe-level judgements from the fourth task.

heading: 4.3.3 Analysis of Reframing Strategies:
text: Figure 2 shows the distribution of reframing strategies used to reframe the unhelpful thoughts in our dataset, among the five strategies proposed by Ziems et al. (2022). Here, we use the strategies identified by the workers in the fourth task of evaluating reframed thoughts. Most rewritten thoughts make use of one of the five strategies, with very few being labeled as "None." Growth Mindset and Optimism are the most commonly used reframing strategies, followed by Neutralizing and Self-Affirmation. Optimism is especially common for patterns that focus on the negative aspects of the situation such as Discounting the positive and Mental Filtering.

heading: 5 Models to Generate, Recognize, and Reframe Unhelpful Thoughts
text: We train and evaluate different models using our PATTERNREFRAME dataset on three tasks: generating, identifying, and reframing unhelpful thoughts – all conditioned on a given persona.

heading: 5.1 Generating Unhelpful Thoughts

heading: 5.1.1 Task and Data
text: Given a persona and an unhelpful thought pattern, the goal is to generate a thought that exhibits the given pattern and aligns with the persona. We formulate the task as a standard conditioned generation problem and optimize the maximum likelihood loss during training. We use the train, validation, and test splits described in §4.3.1.

heading: 5.1.2 Methods
text: We evaluate methods based on fine-tuning and fewshot learning. We fine-tune BART-large (Lewis et al., 2020), T5-large (Raffel et al., 2020), and R2C2-3B (Shuster et al., 2022) (a BART-based language model specialized in dialogues). For the input, we concatenate the persona and the unhelpful thought pattern texts using a special delimiter token. We also generate responses with GPT3.5 (Ouyang et al., 2022), a state-of-the-art language model trained to follow human instructions, as a 1-shot method. We generated thoughts for only 100 random inputs in the PATTERNREFRAME test set, since we had limited access to the API5 to GPT3.5 (text-davinci-002)6. We provide implementation details and examples of input prompts in Appendix D and E, respectively.

heading: 5.1.3 Automatic Evaluation
text: Following previous work on text reframing (Ziems et al., 2022; Chen et al., 2021), we report BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020), which capture the semantic similarity between the generated thought and the human reference. We also report distinct-1, and distinct-2 metrics to measure the diversity of the generations. Distinct-n (Li et al., 2016) calculates the ratio between the number of unique n-grams and the total number of n-grams in a generation.
Table 2 shows the automatic evaluation results for the task. All the models perform close to each other in terms of BLEU, BERTScore, and ROUGE. GPT3.5 generates lexically diverse rewrites with the best Distinct-n scores. We provide examples of system outputs in Table 3.

heading: 5.1.4 Human Evaluation
text: As automatic metrics often fail to fully capture human preferences in text generation tasks, we also perform human evaluation. We collect human ratings of 100 random thoughts from the test set. Similar to previous style transfer works (Ziems et al., 2022; Briakou et al., 2021; Rao and Tetreault, 2018), we evaluate the generated rewrites along three dimensions through Yes/No binary ratings: (i) fluency, which evaluates the readability of the generation, (ii) meaning preservation, which here verifies if the rewrite aligns with the given persona
5https://openai.com/api/ 6In our experiments, we used text-davinci-002, since text-
davinci-003 had not been released yet.
and thought, and (iii) quality, which here evaluates if the generated thought exhibits the given unhelpful thought pattern. We collect 9 annotations for each system output and apply majority voting to extract the final annotation.7
Table 3 shows the percentage of outputs rated positively by at least five of the nine annotators. GPT3.5 outperforms all other approaches, including human references, in terms of fluency and quality. However, GPT3.5 shows the lowest (but still very high) meaning preservation score for generating thoughts. The other models have more difficulty including the unhelpful pattern (lower "thought quality" scores).

heading: 5.2 Classifying Unhelpful Thoughts

heading: 5.2.1 Task and Data
text: Given a persona and a thought, the goal is to classify them into one of the ten unhelpful thought patterns or “None”, which indicates that the input thought does not contain any of the ten unhelpful patterns, or the thought does not align with the persona. We formulate the task as a multiclass classification problem with eleven categories.
We once again use the same train, validation, and 7We also provide results using a more stringent threshold of 7 out of 9 annotators rating positively, in Appendix F. The pattern of results is similar.
test splits described in §4.3.1. Note that the dataset contains only positive examples for the classification task, i.e., thoughts that align with a specific thought pattern and persona. For every positive example, we construct a negative example by randomly choosing one of the following options: (i) a thought from our dataset that belongs to the same pattern but a different persona. (ii) a dialog text from PERSONA-CHAT belonging to the same persona (but presumably not containing any unhelpful pattern), (iii) a dialog text from PERSONA-CHAT belonging to a different persona (and again, presumably not containing any unhelpful pattern). Thus, negative examples encompass neutral texts and misaligned thoughts and personas. We assign the category “None” to these examples. We have 3,834 train, 1,915 validation, and 13,572 test instances after augmenting the dataset with these examples.

heading: 5.2.2 Methods
text: We finetune RoBERTa (Liu et al., 2019) using the soft-label distribution obtained through the second task of our data collection pipeline (§4.1), where we asked multiple annotators to identify the patterns exhibited in a thought, and then normalized the votes across the patterns. We use a soft label distribution instead of single label because of the high overlap across patterns. We also perform
11-way, 1-shot classification using GPT3.5. We construct the input prompt using one example from each category (examples in §E) and classify 100 random inputs in the test set. We include further implementation details in Appendix D.

heading: 5.2.3 Evaluation
text: Figure 4 shows the confusion matrices for RoBERTa and GPT3.5 on the augmented version of the PATTERNREFRAME test set. Given that several unhelpful thinking patterns are closely related (for example, All or Nothing Thinking and Catastrophizing), we cluster the patterns using the KMeans algorithm (Lloyd, 1982) to group together patterns that were deemed close by the model 8. RoBERTa performs well on all the categories (> 72%) except the Mislabeling category, which has a high overlap with the Polarized Thinking category. The None category has the highest performance, which shows that the classifier is able to differentiate neutral texts that do not contain any unhelpful pattern, or texts that are not aligned with the persona. 1-shot classification using GPT3.5 performs worse than fine-tuned RoBERTa. GPT3.5 has trouble distinguishing texts with and without unhelpful patterns and gets a low score for None. We also observed that 40% of the classification predictions changed for GPT3.5 after reordering the sequence of exam-
8We first constructed an aggregated vector for each pattern by averaging the 10-dimensional outputs of the classifier for the examples corresponding to each pattern on the validation set. We then clustered the ten 10-dimensional aggregated vectors into five clusters using the KMeans algorithm, with the number of clusters determined using the elbow method.
ples in the prompt, which shows that few-shot classification is not as reliable for this task, while still providing decent performance way above chance.

heading: 5.3 Reframing Unhelpful Thoughts

heading: 5.3.1 Task and Methods
text: Given a persona, an unhelpful thought pattern, and a thought exhibiting the given pattern, the goal is to reframe the thought in a way that still aligns with the persona and the context of the thought but does not contain the pattern. The reframing problem is similar to the generating one, except that the unhelpful thought is now a part of the input instead of the target. We use the same training, validation, and test splits for the reframing task (§4.3.1). We also evaluate the same approaches described in §5.1.2. For fine-tuned methods, we concatenate the persona, the pattern, and the thought texts with a special token. For few-shot methods, we construct a prompt similar to the one used for the generation task, but with the reframed statements (examples in Appendix E).

heading: 5.3.2 Automatic Evaluation
text: Table 2 shows the automatic evaluation results on the PATTERNREFRAME test set. We use the metrics described in §5.3.2 namely BLEU, ROUGE, BERTScore, and Distinct-n metrics. As each unhelpful thought can have up to 3 ground truth reframed versions, we take the maximum of the three scores and report the mean of these maxima. R2C2 performs the best in terms of BLEU and BERTScore. GPT3.5 again outperforms the
Generating Unhelpful Thoughts
Persona I like to chat with my friends. I like to read books that are in a series. I like to go to the movies. I love to go out to eat with my family. Pattern Catastrophizing T5 I love to go out to eat with my family, but I’m sure we’ll all get sick and die. BART I like to read books that are in a series, but I will never finish them. R2C2 I love to go out to eat with my family, but one day we will all get food poisoning and die. GPT3.5 I’m out of money to go out to eat with my family, maybe they won’t want to hang out with me anymore. Human My friends didn’t want to chat with me today. They must have all hate me and wish me ill.
Persona I write children’s books. I like to drink. I live alone. I shop online, too much. I don’t believe in love. Pattern Should statements T5 I write children’s books, everyone should read them. BART I write children’s books, everyone should read them. R2C2 I write children’s books. I should be published by now. GPT3.5 I should be able to write the perfect book, drink moderately, be in a relationship, and not shop online so much. Human I shop online and I should make enough money to be able to afford anything and everything I want.
Reframing Unhelpful Thoughts
other models and the human references in terms of Distinct-1 and Distinct-2 scores, which indicates that the generations are lexically diverse. Table 3 provides examples of system outputs.

heading: 5.3.3 Human Evaluation
text: Figure 3 shows human evaluation results on 100 reframed thoughts generated by different models given the persona, the pattern type, and the unhelpful thought from our test set. Similar to the generating thoughts task, we evaluate the reframed thoughts along fluency, meaning preservation, and quality, where we ask the annotators if the reframed thought removes the given unhelpful pattern while being consistent with the initial thought. All models perform close to human reference in terms of fluency and meaning preservation. In fact, all the outputs of R2C2 and GPT3.5 are fluent and preserve meaning (that is, they generate statements that are not contradictory with the initial thought). For reframing quality, that is, removing the un-
helpful pattern, all models perform over 70%, but GPT3.5 performs the best. GPT3.5’s superiority is even more marked when using the more stringent threshold of 7 out of 9 annotators rating positively in Appendix F.
Overall, the evaluation suggests that using modern models to produce reframing is a feasible approach, even with a small amount of data for finetuning. In particular, GPT3.5 performs remarkably well and very close to crowdsource worker performance, only based on prompting.

heading: 6 Conclusion
text: In this work, we introduced a novel dataset, PATTERNREFRAME, which contains (1) about 10k statements exhibiting unhelpful thought patterns, conditioned on a persona, and (2) multiple rewritten complementary thoughts that do not contain the initial unhelpful pattern, instead reframing the thought in a more constructive way.
Using this dataset to train or prompt various mod-
ern language models, we showed that this range of models can already be a powerful tool to generate, identify, and reframe unhelpful thoughts, conditioned on a persona. By releasing our dataset 9, we hope to help practitioners of CBT draw from a richer, more diverse set of examples of unhelpful thought patterns and reframings. This would help address the important limitation of a lack of personalized and specific examples in existing datasets, when teaching cognitive techniques.
Future work will evaluate whether leveraging models to produce richer training material results in more robust learning and understanding of the types of unhelpful thought patterns in humans.This may serve as the basis for future psychological validation studies of the materials and support future studies of low-intensity self-help interventions.

heading: 7 Limitations
text: This work relied on previously published datasets to source personas on which to anchor the generated unhelpful thoughts, and thus shares the limitations of those datasets. In particular, they use English-language responses, written by workers located in the United States.10. While these workers are reasonably diverse (Moss et al., 2020), the examples generated may not reflect the thought patterns and personas across cultures and diverse populations. This data is also generated by people who are being paid, as opposed to people genuinely engaging about situations that matter to them. Besides the substance of the thoughts themselves, a more direct limitation is that the models generate only English, so would not be directly usable for speakers of other languages.
In addition, the data collected reflects the understanding of lay people, rather than trained clinical psychologists. While this makes the material more immediately relatable to other lay people, it is possible that the data do not capture what clinical psychologists would consider adequate illustrations of unhelpful patterns. Our data has been spot-checked by a CBT-trained clinical psychologist and found generally sound, but the entire material should undergo further validation.
Another limitation is that the models that we have tested are resource-intensive. In particular, the
9https://github.com/facebookresearch/ ParlAI/tree/main/projects/reframe_ thoughts
10Our crowdsourcing tasks pay workers well above minimum wage.
best-performing model, GPT3.5, is only available through a paid API.

heading: 8 Ethical considerations
text: While our work was developed to generate abundant data supporting work towards improving wellbeing, the negative statements it generates could be misused. The parallel data of unhelpful thoughts and their reframed versions can also be used to generate negative texts from neutral ones, by training systems with reframed versions as the input and unhelpful thoughts as the output. This risk of generating negative content from positive/neutral texts aligns with the risks of toxicity reduction and sentiment style transfer tasks.
Conversely, a different risk stems from overeager use of our work. This work aims to examine the feasibility of generating ample practice material anchored on specific personas. We hope that releasing a large dataset of unhelpful thoughts and reframings will further research that will ultimately help practitioners, but there is a danger that people attempt to use the material as is, without the supervision of a trained professional, which could be harmful, as the material has not been tested with participants while monitoring adverse events such as increased anxiety or warped understanding of what unhelpful thoughts and useful reframings are.

heading: A Data Collection Details
text: A.1 Onboarding Tasks We introduce two onboarding tasks to ensure that the crowdsource workers understood the concept of unhelpful thoughts and how to reframe them. The onboarding tasks were reviewed by a CBT-trained psychologist. We use one onboarding task for tasks 1 and 2 and another onboarding task for tasks 3 and 4 of the data collection pipeline. For the first onboarding task, we display an unhelpful thought pattern, one positive example that contains the pattern, and one negative example that does not, and ask the workers to select the positive one. We only allowed the workers that were able to identify the correct example for three out of four such instances. For the second onboarding task, we display an unhelpful thought pattern, a thought containing the pattern, one positive example that reframes the thought, and one negative example that does not. We only allow the workers that were able to identity the positive example in three out of four such instances.

heading: B Data Collection Interface Snapshots

heading: C Evaluation Interface Snapshots
text: D Implementation details
D.1 Generation Models We finetuned the BART, T5, and R2C2 baselines using ParlAI11. We used the BARTlarge (400M parameters), T5large (770M parameters), and R2C2base (2.7b parameters)12 architectures. We used Adam optimizer (Kingma and Ba, 2014) and performed a hyperparameter search over learning rates 1e-05, 1e-06, 1e-07, and 1e-08. We used linear warmup of 100 steps and applied early stopping with a patience value of 5. We evaluated the validation set once in every 200 updates and truncated the input and the labels to 1000 tokens. We applied gradient clipping value of 1.0. We used a batch size of 32. During inference, we used beam search with beam size 10. We chose the best checkpoint during training based on the perplexity on the validation set. Each model takes around 1 hour to run on 8 NVIDIA Tesla V100 Volta 32GB GPUs.
D.2 Classification Models For classification experiments, we finetuned the RoBERTa-large checkpoint from Huggingface13. We used Adam optimizer (Kingma and Ba, 2014), learning rate of 1e-05, with linear warmup of 100 steps. We trained the model for a maximum of 10 epochs. We evaluated on the validation set every 200 updates. We used a batch size of 16. We chose the best checkpoint during training based on the weighted F1 value on the validation set. The model takes around 1 hour to run on 1 NVIDIA Tesla V100 Volta 32GB GPU.

heading: E GPT3.5 Prompt Examples
text: 11https://www.parl.ai/docs/index.html 12https://parl.ai/docs/zoo.htmlr2c2-base-2-7b 13https://github.com/huggingface/transformers

heading: F Results with 7 over 9 agreement
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract + introduction (1) + Conclusion (6)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4 describes the data we collect, 5 the models we use and finetune, with more details in the appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: all citations given across sections 4, 5, and the appendix
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? all the models and data we use have been open-sourced for academic use and are widely used with that intent

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 7 and 8 reiterate that the use of our data is intended as early research

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We specify in section 4 that we are not collecting any personal information.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We describe limitations of the dataset in Section 7, in particular language and culture limitations.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In section 4.
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. We use our own set-up for evaluation
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 and Appendix

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix B and C

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 4 + appendix B and C: crowdsource workers are not providing personal information
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? The data collection protocol is reviewed by internal reviewers but not subject to an IRB as there is no sensitive data or personal information

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Described in section 7

heading: Future Work
text: Future work will evaluate whether leveraging models to produce richer training material results in more robust learning and understanding of the types of unhelpful thought patterns in humans. This may serve as the basis for future psychological validation studies of the materials and support future studies of low-intensity self-help interventions