Are Experts Needed? On Human Evaluation of Counselling Reflection Generation
Year: 2023

Abstract:
Reflection is a crucial counselling skill where the therapist conveys to the client their interpretation of what the client said. Language models have recently been used to generate reflections automatically, but human evaluation is challenging, particularly due to the cost of hiring experts. Laypeople-based evaluation is less expensive and easier to scale, but its quality is unknown for reflections. Therefore, we explore whether laypeople can be an alternative to experts in evaluating a fundamental quality aspect: coherence and context-consistency. We do so by asking a group of laypeople and a group of experts to annotate both synthetic reflections and human reflections from actual therapists. We find that both laypeople and experts are reliable annotators and that they have moderateto-strong inter-group correlation, which shows that laypeople can be trusted for such evaluations. We also discover that GPT-3 mostly produces coherent and consistent reflections, and we explore changes in evaluation results when the source of synthetic reflections changes to GPT-3 from the less powerful GPT-2.

heading: 1 Introduction
text: Motivational Interviewing (MI, Miller and Rollnick, 2012) is a highly effective counselling practice in healthcare (Moyers et al., 2009), where the therapist focuses on evoking the client’s own motivation for behaviour change, such as smoking cessation and alcohol use reduction. In MI, reflective listening is a crucial strategy of showing empathy, where the therapist conveys a brief conversational summary of how they understand what the client said (Miller et al., 2003; Rollnick et al., 2008). An example is shown in Table 1.
Learning effective reflective listening requires considerable training time and expert supervision (Rautalinko and Lisper, 2004; Rautalinko et al., 2007). Therefore, recent studies used language models (LMs) as automatic reflection generators to aid training (Shen et al., 2020, 2022; Ahmed,
2022), where the LM receives a dialogue context as the input and outputs a reflection (Table 1).
Human evaluation of reflection generation is crucial, since automatic metrics are often not robust (Liu et al., 2016). For such evaluations, experts (professional therapists) are used due to their deep understanding of the complex and sensitive domain of counselling dialogue. However, expert evaluation is costly and difficult to scale, and previous human evaluations often adopted oversimplified annotation schemes (good vs bad reflection) or worked with short dialogue contexts (5 turns). Evaluation with laypeople (such as crowdworkers) tends to be less expensive (Iskender et al., 2020), but to the best of our knowledge its reliabil-
6906
ity for reflections is unknown. In this work, we investigate if laypeople are a viable alternative to experts for human evaluation of coherence and context-consistency (referred to as coherence for brevity). This is a weak point of recent generative models (Ji et al., 2022) and also a fundamental quality aspect of reflection generation, since a reflection has to first “make sense” in the context before it can be evaluated against counselling principles.
To this end, we recruit a group of MI experts and a group of laypeople as annotators and analyse their evaluation1 quality (Figure 1). The workload of
1Data available at https://github.com/uccollab/ expert_laypeople_reflection_annotation.
each annotator consists of mixed human reflections from actual therapists and synthetic reflections produced by language models (GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020)2), and the annotator is not informed of the source of any reflection. For each reflection, the annotator flags whether it is coherent as a Yes/No binary choice. If “No” is chosen, the annotator proceeds to select one or more applicable incoherence error categories. In doing so, our evaluation goes beyond a binary Yes/No scheme and sheds light on the types of inco-
2We also conducted human evaluation of reflections generated by BART (Lewis et al., 2020), but the results are not included in the main body as the model failed to generate sufficiently diverse reflections (Appendix B)
herence errors made by reflection generators. Notably, we adopt long dialogue contexts — 14 turns on average — to allow for more detailed conversational background to both the reflection generator and the annotator.
Based on the annotations, we conduct in-depth analysis of intra-group agreement among laypeople and among experts, as well as the inter-group correlation between laypeople and experts. We also explore whether more powerful LMs produce more coherent synthetic reflections and how they affect annotations of human reflections. We find that:
I Both laypeople and experts are reliable annotators based on their intra-group agreements on binary coherence evaluation. They also show moderate to strong inter-group correlation.
II Human reflections are more often annotated as coherent than GPT-2 reflections, but it is not the case with the more powerful GPT-3. Interestingly, both laypeople and experts are less likely to annotate a human reflection as coherent when its surrounding synthetic reflections come from GPT-3, though experts are relatively more consistent in this regard.
I represents the first evidence that laypeople are capable of coherence evaluation for reflection generation. II poses an interesting research question on whether synthetic reflections from large LMs can match or outperform human reflections on dimensions deeper than coherence, such as empathy.

heading: 2 Related Work

heading: 2.1 Human Evaluation for Response Generation
text: In most studies of response generation, human evaluation is considered the ultimate benchmark, since it can assess quality aspects like interestingness and safety (Deriu et al., 2021; Liu et al., 2016; Thoppilan et al., 2022) that may elude automatic metrics. Typically, the human evaluator rates modelgenerated responses in an interactive or static setup.
In an interactive setting, the human converses with the dialogue model and evaluates its responses as good/bad (e.g., Shuster et al., 2022) or selects applicable attributes like knowledgeable/engaging/... (e.g., Komeili et al., 2022). In a static setup, the human evaluates responses or entire dialogues on the Likert scale for an attribute (Rashkin et al., 2019; Li et al., 2020, inter alia) or compares responses from
different models through ranking or A/B testing (Xie and Pu, 2021; Kim et al., 2021, inter alia).
Despite their popularity, standard human evaluation protocols suffer from various issues. One such example is subjectivity (Li et al., 2019; Howcroft and Rieser, 2021), in particular in the context of Likert scales. Other issues include the lack of reproducibility across studies and the influence of evaluation instructions (Belz et al., 2023; Huynh et al., 2021; Smith et al., 2022).

heading: 2.2 Reflection Generation and Its Human Evaluation
text: Shen et al. (2020) developed the first LM-based reflection generator. Shen et al. (2022) leveraged commonsense and domain knowledge for reflection generation. Ahmed (2022) adopted a few-shot approach. All those studies used at most 5 turns as the dialogue context, in contrast to the 14 turns on average in our work. Therefore, our generation and evaluation is more context-aware.
For human evaluation, Shen et al. (2020, 2022) asked two experts to evaluate relevance, fluency and “reflection-like-ness” on Likert scales. Ahmed (2022) conducted expert evaluation of GPT-3 generated reflections in a good-vs.-bad setup. Wu et al. (2022a) proposed non-expert evaluation of coherence and context-consistency and developed an error annotation scheme accordingly. We adopt this annotation scheme in our work, but we focus on comparing laypeople- and experts-produced evaluations and investigating if laypeople can be a viable alternative to experts for coherence evaluation.

heading: 2.3 Expert and Non-Expert Evaluation for Natural Language Generation
text: Whether to use experts for NLG evaluation generally depends on the domain. For example, opendomain dialogue generation mostly involves nonexperts to assess attributes like engaging-ness and human-ness (e.g., Roller et al., 2021; Komeili et al., 2022), while response generation for specialised domains like mental health (Sharma et al., 2021) and clinical dialogue (Miehle et al., 2018) is largely evaluated by domain experts.
Some human evaluation studies have compared expert and non-expert NLG evaluations, such as for summarisation (Gillick and Liu, 2010; Fabbri et al., 2021), machine translation (Freitag et al., 2021), story generation (Karpinska et al., 2021) and others (e.g., Snow et al., 2008). Many of these works reveal considerable gaps between assessments from
experts and those from crowdworkers. In particular, Freitag et al. (2021) find that automatic metrics outperform crowdworkers in terms of correlation with expert judgement.

heading: 3 Methodology

heading: 3.1 Synthetic Reflection Generation
text: We leverage LMs to generate synthetic reflections through fine-tuning and prompting, both of which are based on AnnoMI (Wu et al., 2022b), an expertannotated dataset of transcribed MI sessions over various topics such as smoking cessation and alcohol use reduction. AnnoMI contains 110 conversations with 4441 therapist turns (utterances), 28% (1256) of which are reflections and we refer to those as “human reflections”.
For each human reflection, we concatenate its preceding utterances and keep the rightmost (i.e., temporally most recent) 384 tokens as the dialogue context, which contains 14 previous turns on average. Notably, this is 3 times the context size used in previous work (≤ 5 turns), as we assume richer context enables better reflection generation. Thus, we construct 1256 ⟨context, human reflection⟩ pairs based on AnnoMI.

heading: 3.1.1 Fine-Tuning
text: Following recent work on reflection generation (Shen et al., 2020), we fine-tune GPT2 (gpt2-medium, Radford et al., 2019) on ⟨context, human reflection⟩ pairs. At test time, we use greedy, beam and nucleus (Holtzman et al., 2020) (p ∈ {0.4, 0.6, 0.8, 0.95}) decoding to generate diverse synthetic reflections.

heading: 3.1.2 Prompting
text: We also prompt GPT-3 (text-davinci-002, Brown et al., 2020) to generate reflections, in light of the impressive generative capabilities of large LMs shown recently (Bhaskar et al., 2022; Goyal et al., 2022, inter alia) including for reflection generation (Ahmed, 2022). We use the default temperature (1.0) and p ∈ {0.4, 0.6, 0.8, 0.95} for decoding. We model our prompt as asking GPT-3 to read a series of ⟨context, human reflection⟩ pairs (learning examples) and then to complete a final dialogue context where the reflection is missing (test example).
The test example is always a dialogue context from AnnoMI, but we explore two sources of learning examples — AnnoMI and textbook — to diversify the generation. The former (Figure 2a) is
simply ⟨context, human reflection⟩ pairs we constructed previously, while textbook examples (Figure 2b) are taken from the Motivational Interviewing Treatment Integrity (MITI) coding manual (Moyers et al., 2014). Each textbook example consists of a client statement — which we use as dialogue context — along with a simple reflection and a complex one, where the complex reflection adds more meaning/emphasis to the client statement than the simple one (Miller et al., 2003).

heading: 3.2 Human Evaluation
text: We recruit 2 groups of annotators:
• 9 laypeople known to us and with no experience in MI;
• 9 experts found through professional networks, in particular the Motivational Interviewing Network of Trainers3, an international organisation of MI trainers and a widely recognised MI authority.

heading: 3.2.1 Workload
text: Table 2 presents the annotation workload overview. To create annotation materials, we randomly sample 15 ⟨context, human reflection⟩ pairs from 15 AnnoMI dialogues. For the context in
3https://motivationalinterviewing.org/
each pair, we generate 9 semantically diverse synthetic reflections4 with GPT-3 and 7.13 on average5 with GPT-2. Thus, for each ⟨context, human reflection⟩ pair, we create 2 annotation batches that each contain the context, the human reflection and synthetic reflections. The two batches differ in that the synthetic reflections in one batch come from GPT-2 while those in the other batch are from GPT-3.
Each annotator is first randomly assigned 5 batches where the synthetic reflections are from GPT-2 (GPT-2 stage). After completion of these batches and then a waiting period of at least 3 days (Appendix C), the annotator is randomly assigned 5 more batches where the synthetic reflections are from GPT-3 (GPT-3 stage). The task ends when the annotator has finished all 10 batches. Overall, each batch is randomly assigned to 3 laypeople and 3 experts, resulting in each reflection being evaluated 3 times by laypeople and 3 times by experts.

heading: 3.2.2 Annotating One Batch
text: When annotating a batch (Figure 1), the annotator first reads the context and then iteratively annotates all the reflections. The reflections in each batch are shuffled, and the annotator is not informed of the source of any reflection.
4There is one context with 7 instead of 9 GPT-3 reflections due to lack of semantic diversity among generated candidates.
5In practice, GPT-2 and BART reflections were evaluated together, and their combined size is the same as GPT-3 reflections’. Thus, there are fewer GPT-2 reflections than GPT-3 reflections. We exclude BART reflections from the GPT-2 stage for fairness considerations. More details in Appendix B.
For each reflection, the annotator chooses Yes/No regarding whether it is coherent. If the answer is No, the annotator selects one or more applicable error categories. We adopt the error annotation scheme developed by Wu et al. (2022a), since the categories were qualitatively extracted from free-text feedback provided by laypeople w.r.t. model-generated reflections. Therefore, those categories represent a good approximation of what errors our annotators may find in synthetic reflections. Those categories are:
• Malformed: suffers from unclear references, bad grammar, and/or confusing logic.
• Dialogue-contradicting: contradicts context partially or fully.
• Parroting: repeats a part of context unnaturally.
• Off-topic: little to no relevance to context.
• On-topic but unverifiable: relevant to context but including content that cannot be verified based on context alone.
Prior to annotation, the annotator reads a mandatory tutorial about coherence and consistency with examples for each error category, and it remains accessible throughout the annotation process.

heading: 3.2.3 Cross-Stage Human Reflection Recurrence
text: Due to random batch assignment, an annotator may annotate batch bm in the GPT-2 stage and bn in the GPT-3 stage where bm and bn share the same ⟨context, human reflection⟩. For the annotator in such cases, the shared human reflection is recurring across stages, and hence the annotator annotates it twice. To make it less likely that an annotator annotates a recurring human reflection in the GPT-3 stage based on how they recall annotating it in the GPT-2 stage, each annotator waits for at least 3 days6 between completing their last batch in the GPT-2 stage and starting their first batch in the GPT-3 stage.

heading: 4 Annotation Results & Analysis

heading: 4.1 Intra-Group Agreement
text: We measure intra-group agreement among laypeople and among experts, i.e., how much the annota-
6More details in Appendix C.
tors of the same group agree with each other, which gauges the internal consistency of the annotators.
4.1.1 Coherent and Incoherent We first analyse the global agreement on the binary Yes/No (Coherent/Incoherent) annotation. We adopt both the classical Fleiss’ kappa (Fleiss, 1971) and Randolph’s fixed-marginal kappa (Randolph, 2005), because 1) Fleiss’ kappa is known to be overly penalising when the marginal label distribution is imbalanced (Feinstein and Cicchetti, 1990) and 2) Randolph’s kappa is preferable when the annotators have no prior knowledge of the expected label distribution (Assimakopoulos et al., 2020).
As Table 3 shows, Fleiss’ kappa in the GPT-2 stage indicates moderate agreement (Landis and Koch, 1977) for both annotator groups, but in the GPT-3 stage it drops to fair agreement for laypeople and almost zero for experts. The drop may appear to suggest a drastic change in agreement, but deeper analysis reveals a considerable cross-stage change of marginal label distribution that may skew Fleiss’ kappa — for example, experts annotate GPT-3 reflections as Coherent 82% of the time (§4.3.2) as opposed to 38% for GPT2 reflections. As an evidence, Randolph’s kappa, which is not influenced by marginal label distribution, still shows (Table 3) fair agreement among the laypeople and moderate agreement among the experts in the GPT-3 stage.
Beyond global agreement, we conduct more granular analysis on which one of {Coherent, Incoherent} is easier to agree upon. Specifically, we follow Tsakalidis et al. (2022) to calculate the perlabel majority agreement ratio (referred to as “agreement ratio” for brevity) for Coherent and Incoherent separately. For a label l, its agreement ratio AM (l) is:
AM(l) = #(reflections assigned l by 2 annotators)
#(reflections assigned l by ≥ 1 annotators)
For example, the agreement ratio of Coherent is the number of reflections annotated as Coherent by 2 out of 3 annotators (hence majority agreement)
divided by the number of reflections annotated as Coherent by any annotator.
As Table 4 shows, the agreement ratio of Incoherent has a minor lead over that of Coherent in the GPT-2 stage. In the GPT-3 stage, however, Coherent shows substantially higher agreement ratio than Incoherent. Therefore, as the LM grows in power (GPT-2→GPT-3), it becomes easier for annotators to agree on what is Coherent than on what is not, and this applies to both groups.
We note that, in Tsakalidis et al. (2022), an example is given label l if the agreement ratio of l is above 0.3 and a majority of annotators assign l to the example. Our results show that both laypeople and experts have agreement ratios that are almost always comfortably higher than the 0.3 threshold, particularly w.r.t. Coherent (0.66∼0.90). Thus, also considering the global agreements (Table 3), both laypeople and experts appear to be reliable annotators, and a reflection should be considered Coherent if a majority of annotators deem it so.

heading: 4.1.2 Agreement on Error Categories
text: We also measure agreement ratio for each error category to inspect whether some errors are easier than others for annotators to agree upon.
Based on Tables 4 and 5, one can observe that agreement ratio is generally higher for Incoherent than for any error category. While it may be inherently more challenging to annotate an error category than to annotate Coherent/Incoherent due to the label space size difference (5 vs. 2), this is
still a strong indication that it is easier for annotators to agree that a reflection is Incoherent than to agree upon any specific incoherence problem.
Interestingly, Parroting has clearly higher agreement ratio among laypeople than among experts in both stages, which means some experts are more tolerant of Parroting than others but laypeople are similar to each other in this regard.

heading: 4.2 Inter-Group Correlation
text: We also investigate inter-group correlation, namely the correlation between laypeople and experts w.r.t. their annotations. We measure it based on coherence scores: given a reflection and the 3 annotators to whom it was assigned, its coherence score is the number of annotators that flagged it as Coherent. Thus, a coherence score has a range of {0, 1, 2, 3}, and each reflection has one score from laypeople and one from experts.
As Table 6 shows, inter-group correlation is strong in the GPT-2 stage and moderate in the GPT-3 stage (Prion and Haerling, 2014). Combined with our previous findings on the intra-group agreement on coherence (§4.1.1), this is further evidence that laypeople can be a viable alternative to experts for scaled-up reflection coherence evaluation. In particular, a binary Coherent/Incoherent setup may be more suitable, since per-label majority agreement ratios are clearly higher on Coherent and Incoherent than on the error categories (§4.1.2). Nevertheless, the weaker inter-group correlation in the GPT-3 stage does suggest experts-laypeople differences (we probe them further in §4.3), and it also shows that laypeople-based evaluation is relatively more challenging when the reflections come from powerful LLMs.

heading: 4.3 Cross-Stage Annotation Changes
text: We further investigate how reflections — both human and synthetic ones — are annotated differently in different stages. We focus on the distribution of Coherent/Incoherent labels and error labels based on the results in Figure 3.

heading: 4.3.1 Cross-Stage Shift on Human Reflections
text: Both laypeople and experts flag human reflections as Coherent less often in the GPT-3 stage than in the GPT-2 stage. Therefore, we analyse the distribution of Coherent and Incoherent labels given to human reflections and examine whether the crossstage distribution shift is significant. We do so with 2 settings: All and Recurrence-Free. All takes into account all the Coherent and Incoherent labels. Recurrence-Free removes the labels from an annotator for a reflection if the reflection is recurring (§3.2.3) for the annotator (i.e., the annotator annotated the reflection in both stages) and therefore removes recurrence-caused annotator bias. As shown in Table 7, under both All and Recurrence-Free, both laypeople and experts less often annotate human reflections as Coherent in the GPT-3 stage. Notably, the shift of laypeople is significant, while the shift of experts is not.
Beyond the global distribution of Coherent and Incoherent labels, we also inspect the cross-stage
shift w.r.t. coherence scores (defined in §4.2) of human reflections. With the paired Wilcoxon signed-rank test, we have a similar discovery: laypeople-based coherence scores are significantly (p < 0.05) lower in the GPT-3 stage than in the GPT-2 stage, while it is not the case for experts.
Also shown in Figure 3, human reflections are clearly more likely (∆ ≥ 11%) to be annotated by laypeople as Parroting and On-topic but unverifiable in the GPT-3 stage. In comparison, error annotations by experts for human reflections are more consistent across stages, with minor (∆ ≤ 4%) increases in On-topic but unverifiable, Malformed and Dialogue-contradicting.
Therefore, compared to experts, laypeople are overall more influenced by synthetic reflections when annotating human reflections. This annotation fluidity is a potential concern for laypeoplebased scaled-up coherence evaluation.

heading: 4.3.2 Cross-Stage Differences on Synthetic
text: Reflections
As Figure 3 shows, GPT-3 reflections are significantly (chi-squared test, p < 0.05) more often annotated as Coherent than GPT-2 ones by both laypeople and experts, which is not surprising given that GPT-3 is considerably more powerful. Interestingly, while laypeople and experts are similar in Coherent/Incoherent label distribution for GPT-2, experts are significantly more likely than laypeople to annotate GPT-3 reflections as Coherent.
Upon further analysis, we notice that much of the laypeople-experts divide on GPT-3 Coherent rate can be attributed to Parroting, which is used 19% of the time by laypeople but only 7% by experts. For the other 4 error categories, laypeople are experts behave similarly: the proportion of each category is substantially lower in the GPT-3 stage. This shows that GPT-3 makes most types of incoherence errors less often than GPT-2.
Overall, it is clear that experts are less strict about Parroting. This is likely because a reflection summarises what the client said, which may sometimes appear repetitive to a layperson when an expert may consider it good practice. As further evidence, we note that human reflections, which showcase good practice, are not annotated as Parroting by experts in either stage, while laypeople do so in the GPT-3 stage (§4.3.1).
4.3.3 Human vs. Synthetic in Coherent Rate We compare human and synthetic reflection w.r.t. the proportion of Coherent labels7. As shown in Figure 3, human reflections are annotated as Coherent significantly (chi-squared test, p < 0.05) more often than synthetic reflections by both laypeople and experts in the GPT-2 stage. This is not unexpected since human reflections are considered the gold standard. However, the trend is reversed in the GPT-3 stage, even though the lead of GPT3 over human reflections is not significant. This shows that GPT-3 is capable of producing coherent reflections, and it can even sometimes match or outperform human reflections. It also raises interesting research questions on whether GPT-3 can compete with human reflections on aspects deeper than coherence, such as empathy and adherence to counselling principles.

heading: 4.4 Case Study
text: To gain qualitative insights into the annotations, we show a case study in Table 8 which presents the annotations on the reflections shown in Table 1.
While the human reflection is annotated as Coherent by every layperson in the GPT-2 stage, it is flagged by 2 laypeople as Parroting in the GPT-3 stage, which may be because those 2 laypeople found the human reflection to be a rephrase of the last client utterance (e.g., “it’s your stuff” in the human reflection compared to “it’s my stuff” in the client utterance). Notably, this example echoes the overall trend that human reflections are more likely (0%→13%) to be flagged by laypeople as Parroting in the GPT-3 stage (§4.3.1).
On the other hand, the human reflection is annotated as Coherent by every expert in the GPT-2 stage, but it is flagged by 1 expert as Malformed in the GPT-3 stage. We postulate that the fluency of GPT-3 reflections may make the human reflection appear less fluent to some annotators. This may be particularly true when there are faithfully transcribed self-repetitions and midsentence changes (“it’s-that’s-it’s your stuff”) in the human reflection, even though we explicitly informed the annotators that those are normal.
For comparison, we also analyse the annotations on the examples of GPT-2 and GPT-3 synthetic reflections. The GPT-2 reflection roughly matches the mood of the client but is also generic,
7We do not compare at the granular error-category-level due to the different scales of human and synthetic reflections.
and it is annotated as Off-topic by 1 layperson and 2 experts. On the other hand, the GPT-3 reflection is fluent and more specific to the dialogue, and un-
surprisingly it is annotated as Coherent by all 6 annotators. While those two reflections cannot cover all of the variety of synthetic reflections, their qualitative difference w.r.t. the human reflection is a good example for showing why annotators may be influenced by the surrounding synthetic reflections when they are annotating a human reflection.

heading: 5 Conclusion
text: In this work, we probed whether laypeople can be used as an alternative to experts in evaluating coherence and context-consistency of counselling reflection generation. Accordingly, we asked both laypeople and experts to annotate synthetic reflections generated by LMs and human reflections from actual therapists. We found that both laypeople and experts are reliable annotators and that they also show moderate to strong inter-group correlation, which is the first concrete evidence that laypeople are capable of such annotations, although laypeople are relatively less aligned with experts on GPT-3 reflections. Furthermore, we found that GPT-3 is mostly able to generate coherent and consistent reflections, and we also explored the annotation shift on human reflections when the source of synthetic reflections changes from the smaller GPT-2 to the more powerful GPT-3.
For future work, we plan to mix, in each batch, synthetic reflections from models of different scales, and investigate how the resulting human evaluations might differ. Another direction worth exploring is alternative ways of coherence annotation, such as ranking, for more nuanced human evaluation results. Future work may also re-examine and modify the error categories to increase IAA on error annotations. We also leave potentially IAAimproving annotation procedures to future work, such as using a warm-up exercise task before actual annotation and allowing annotators to discuss with each other to resolve their differences.
Limitations
The main limitation of this work is the quantity of annotated human reflections. Overall, 15 human reflections are annotated, which are outnumbered more than 7:1 by GPT-2 reflections and 9:1 by GPT-3 reflections. If there were more human reflections annotated, we may be able to confirm, among other potential findings, that GPT-3 reflections were indeed significantly more often annotated as Coherent compared to
human reflections. We also note that the laypeople had a longer between-stage waiting period than the experts, because we could not enforce a similarly long waiting period for the experts due to practical reasons (Appendix C). While an ideal setup would keep the same waiting period duration, Appendices C and D show that the duration difference is not critical.
Furthermore, we adopted sequential annotation for reflections within a batch to make the interface easier to navigate for the human annotators, but this also means that the early samples in a batch might indirectly affect the annotation of the later samples. We leave more investigation on this to future work.
Ethics Statement
In this section, we briefly discuss the ethical aspects of our experiments. We do this with regard to our experiment as a whole.
Ethical Review
Prior to our experiment, materials and methodology underwent ethical review by our institution’s Ethics Board. The proposal was flagged as ethically compliant and accepted without major revisions.
Risks
Our work inspects the annotation differences between laypeople and experts in the counselling domain (MI and reflections in particular). With these premises, it could be seen as a message that therapy can be fully automated, laypeople can replace therapists in creating such systems and generative models could act as "virtual counsellors". We acknowledge that past work inspected similar options (Fiske et al., 2019; D’Alfonso, 2020; Saha et al., 2022), but we take distance from it. Our work is framed as modelling technological advancements that are solely directed at therapist training. We foresee the use of neural NLG as promising in counselling, but only for supporting trainees. We also point out previous work showing why replacing mental health practices with language models (or AI in general) should not be considered (Le Glaz et al., 2021).
Information and Consent
Prior to starting the annotation, both laypeople and experts received an electronic information sheet containing details on the task, purpose of research, workload and pay. This also included the fact that
data would be made available for future research, in accordance with data anonymisation requirements.
Upon starting the annotation, annotators were prompted with a mandatory consent form to confirm their understanding of the terms and conditions and their willingness to take part in the annotation. Annotators were also given an email contact in case of problems during the annotation or any other query. Annotators were automatically prevented from doing the annotation if they did not provide consent.
Demographic Information of Annotators All annotators were highly proficient in English, which is the language of the dialogues. 5 out of the 9 laypeople were based in the Netherlands while the other 4 resided in Italy. Among the experts, 4 were based in the UK, 1 in the Netherlands, 1 in Hungary, 1 in Italy and 2 in Sweden.
We recruited laypeople who were known to us, as this allowed active monitoring of the annotation task, hence ensuring high quality. While this approach is different from other standard ones (such as using crowdsourcing platforms), we argue that the focus of this work is to understand if fully committed laypeople can be valid annotators, which can be challenging considering the annotation quality issues that crowdsourcing platforms suffer from (Dennis et al., 2020).
We also note that the group of laypeople is diverse in demographics and educational backgrounds. Specifically, the group includes people of 5 nationalities in their 20s, 30s and 40s who range from bachelor’s student to professional with a PhD.
To verify the generalisability of our laypeoplebased evaluation, future work may replicate our setup on crowdworkers and compare the resulting annotations with ours.
Remuneration The annotation workload was made explicit in the task (a total of 5 annotation batches in each stage, with a detailed description of what a batch consists of). Annotators were given 30 minutes to complete each annotation batch: laypeople received 19.5 USD/h, while experts received 21.6 USD/hour. This difference is motivated by the generally higher hourly pay of experts. The remuneration is considerably (>50%) higher than the minimum wage levels of the countries of residence of the annotators. It also took most annotators much less than 30 minutes (e.g., 10 to 15 minutes) to complete
a batch, so the effective hourly remuneration was higher than 19.5/21.6 USD.
Data Anonymisation
No personal data about the annotators was kept stored at the end of the experiment. During the annotation process, no annotator ever got in touch with anyone involved in the experiments except for the researchers.

heading: Acknowledgements
text: This work has been funded by the EC in the H2020 Marie Skłodowska-Curie PhilHumans project (contract no. 812882) and the European Research Council (Grant agreement No. 101039303 NG-NLG). We also thank Craig Thomson and Vivek Kumar for their suggestions.

heading: A Modelling & Computation Details
text: A.1 Fine-Tuning We convert the input dialogue context into a plaintext sequence of utterances with interlocutor labels and utterance separators in between, while the output reflection is simply plain text without special preprocessing. An example is shown in Table 9, which formats the ⟨context, human reflection⟩ pair of Table 1 accordingly. In particular, the “~⟨listening⟩” is the cue for the LM to start generating a reflection.
For training, we first divide the 1265 ⟨context, human reflection⟩ pairs into 10 folds, and we then fine-tune the same pre-trained model 10 times independently to generate synthetic reflections for the pairs in each test fold. Each time when fine-tuning a model, we use 8 folds as the training data, 1 as validation data and 1 as test data. We allot pairs from the same dialogue to the same fold in order to avoid overlap between training/validation/test data.
Our experiments are based on the HuggingFace package8. We adopt the pre-trained gpt2-medium (345M parameters). We use 2e-5 as the learning rate for training, based on a hyperparameter search over different learning rates where the metric is perplexity. The other hyperparameters are fixed, including 8 as the batch size and 42 as the random seed. The fine-tuning stops when perplexity has not improved on the validation data for 3 epochs. We ran the fine-tuning on an NVIDIA V100 GPU (16GB). In total, the fine-tuning and inference took under 50 GPU hours.
A.2 Prompting We used text-davinci-002, the largest GPT-3 variant (175B parameters) at the time of experiment.
8https://huggingface.co/
The total cost of generation during the GPT-3 stage was 23.68 US Dollars.
A.2.1 Prompting with Textbook Examples As learning examples, textbook examples are different from AnnoMI examples in that 1) textbook examples are written texts instead of transcripts like AnnoMI, and 2) the context in a textbook example is considerably shorter than the average AnnoMI context which contains 14 utterances.
A simple reflection typically repeats or rephrases what the client has said, while a complex one adds substantial meaning or emphasis and communicates a deeper or richer picture of the client’s statement (Miller et al., 2003). An example is shown in Table 10.
A prompt (Figure 2b) begins with an instruction, followed by 8 textbook examples and the test example placed at the end. Thus, the model is prompted to generate 2 synthetic reflections, one simple and the other complex. Considering recent studies (e.g., Zhao et al., 2021) about the impact of few-shot example ordering on the output, we create 3 prompts to generate 3 different sets of {simple reflection, complex reflection}, where the textbook examples in each prompt are identical but with different random orders.
A.2.2 Prompting with AnnoMI Examples In this prompting method, we do not take simple/complex reflection into account, because human reflections in AnnoMI do not have such labels. Similar to prompting with textbook examples, we construct 3 prompts for each test example in order to obtain diverse GPT-3-generated reflections. The difference from prompting with textbook is that we create those 3 prompts by sampling 3 different sets of learning examples instead of shuffling.
Therefore, the learning example set in each of the 3 prompts is unique, and to ensure fairness the learning examples are not from the same dialogues as the test example.

heading: B Reflection Sampling for Annotation & Inadequacy of BART
text: As mentioned briefly in the main body (Footnotes 2 and 5), human evaluation in the GPT-2 stage included both GPT-2 reflections and BART reflections in practice, since we wanted to diversify synthetic reflections from smaller LMs in the GPT-2 stage. For BART, we fine-tuned the pretrained bart-large (406M parameters, similar in scale to gpt2-medium) in the exact same way we fine-tuned GPT-2, and we also used the same decoding methods for test-time generation.
Overall, for the context in each of the 15 sampled ⟨context, human reflection⟩ pair, we generated 26 synthetic reflections in total with GPT-2, 26 with BART and 36 with GPT-3. In order to ensure smaller LMs and large LMs were equally present in the human annotation of synthetic reflections, we randomly sampled (Appendix B.1) 9 semantically distinct reflections from the 52 GPT-2/BART reflections and also 9 from the 36 GPT-3 reflections for human annotation.
Thus, for each ⟨context, human reflection⟩ pair, we created 2 annotation batches that each contained the context, the human reflection and 9 synthetic reflections. The two batches differed in that the synthetic reflections in one batch came from GPT-2 and BART while those in the other batch were from GPT-3. Both batches were later annotated (§3.2). In other words, GPT-2 and BART reflections were annotated together in the GPT-2 stage. However, BART reflections were vastly outnumbered by GPT-2 and GPT-3 reflections because they were sampled less frequently due to a lack of diversity (Appendix B.2), so we reported only GPT-2 and GPT-3 in the main body for fairness.
Nevertheless, we analyse the annotations on BART-generated synthetic reflections in Appendix B.3, but we note that it is limited by the small quantity of BART reflections and therefore in particular should not be used to compare with the findings w.r.t. GPT-2 and GPT-3 reflections.
B.1 Reflection Sampling Procedure
We grouped reflections through semantic clustering based on their embeddings9, such that the reflections in each cluster were semantically almost identical. For example, if two reflections were identical except that one had a "Hmm." at the beginning while the other did not, they were grouped into the same cluster. Afterwards, we randomly sampled 9 clusters from all the GPT-2 and BART reflection clusters, and we similarly sampled 9 GPT-3 reflection clusters. Finally, we drew from each cluster the reflection with the most tokens, deeming it as the most semantically rich.
B.2 Lack of Diversity Among BART Reflections
While we generated the same number (26) of GPT2 and BART reflections for sampling, in practice there was a considerable lack of diversity among BART reflections that led to them being grouped into fewer clusters and therefore less frequently sampled. Specifically, GPT-2 reflections outnumbered BART reflections 4:1, which means the overall BART:GPT-2:GPT-3 reflection quantity ratio was 1:4:5. Therefore, to ensure fairness, we only reported GPT-2 and GPT-3 reflections in the main body, considering their similar quantities.
To illustrate the lack of diversity among BART reflections, we measure the lexical and semantic diversity of synthetic reflections from GPT2/BART/GPT-3 using Self-BLEU (Zhu et al., 2018) and average pairwise semantic similarity, respectively.
Self-BLEU is based on BLEU (Papineni et al., 2002) which measures the lexical similarity between two sentences at the n-gram level (n ∈ {1, 2, · · · }). Self-BLEU takes all pairs of generated texts (in our case, reflections for the same context), calculates the BLEU score for each pair, and averages the pairwise BLEU scores. Thus, lower Self-BLEU indicates higher diversity among the generated texts. We follow (Zhu et al., 2018) in reporting 2-, 3-, 4-, and 5-gram-level Self-BLEU10 for BART, GPT-2 and GPT-3 reflections in Table 11. Clearly, BART reflections are substantially more homogeneous than those from GPT-2 and GPT-3. For example, Self-BLEU-4 of BART is at 40.70,
9We used the SentenceTransformers package (https: //www.sbert.net/) and all-mpnet-base-v2 (Song et al., 2020) as the embedding model.
10We calculate Self-BLEU based on the NLTK (https: //www.nltk.org/) implementation of BLEU.
compared to the drastically lower 4.49 of GPT-2 and 12.02 of GPT-3.
To compute average pairwise cosine similarity, we 1) compute the cosine similarity between the embeddings (from the same embedding model used for clustering) of the two sequences in each pair of generated reflections for the same context, and then 2) average the similarities of all pairs. As shown in Table 11, the semantic similarity between BART reflections is also considerably higher compared to GPT-2 and GPT-3.
B.3 Label Distribution for BART Reflections
We show in Table 12 the distribution of labels given to BART reflections. Notably, laypeople and experts show considerable difference (∆ = 39%) in the proportion of Coherent labels, which is substantially higher compared to GPT-2 (∆ = 3%) and GPT-3 (∆ = 17%) shown in Figure 3.
Upon further analysis, it is clear that most of the laypeople-experts divide in coherence annotation can be attributed to Parroting, which is used considerably more (∆ = 29%) by laypeople than experts. This again echoes the observation in §4.3.2 that laypeople are more strict about Parroting than experts.
Qualitatively, Table 13 shows the BART reflection for the case study dialogue (Table 8), which
clearly mirrors the last client utterance. Matching the trend discussed above, the reflection is annotated by every layperson as Parroting but by every expert as Coherent.
This finding, together with the low diversity among BART reflections (Appendix B.2), shows that BART has a higher tendency to repeat or rephrase a part of the dialogue context and does not show considerable deviation from this pattern under different decoding parameters. Empirically, this is also our observation of BART reflections in general.

heading: C Waiting Period Between Stages
text: Initially, we conducted the ⟨Laypeople, GPT-2 stage⟩. We then collected GPT-3-generated reflections and invited the same laypeople for the GPT-3 stage annotation. As those two stages were not planned together, there was about a one-month period in between.
Upon discovering the shifting human reflection annotations (§4.3.1) from the laypeople’s results, we recruited the experts to investigate whether the phenomenon was limited to laypeople. Due to time constraint, we were only able to enforce a minimum waiting period of 3 days between the two stages for the experts.
The mean and standard deviation of the waiting period lengths of each annotator group are shown in Table 14. Overall, laypeople had a 39-day gap between the two stages while experts had 7 days.
To probe whether the waiting period difference had an effect, we requested the annotators to fill out a post-annotation questionnaire, where we asked the question “While you were annotating in Phase
2 (i.e., GPT-3 stage), did you remember seeing any response candidate that you had seen in Phase 1 (i.e., GPT-2 stage)?”. We received 7 valid responses from the 8 laypeople who had annotated recurring human reflections, and similarly 7 from the 8 experts that had had recurring human reflections in their workload. Their answers are shown in Table 15.
Clearly, the same number (3) of experts and laypeople remembered seeing recurring human reflections in the GPT-3 stage, but 3 experts answered “No” while 3 laypeople answered “Maybe”, which is not surprising since the longer waiting period may have caused more laypeople not to be able to recall exactly. Nevertheless, the fact that the same number of experts and laypeople are positive about seeing recurring human reflections shows that the waiting period for experts was not overly short and may have in fact been sufficient. This is further evidenced by the finding (Appendix D) that laypeople and experts are similarly consistent in annotating recurring human reflections.

heading: D Shifts of Individual Annotators
text: In §4.3.1, we showed that laypeople and experts as annotator groups are less likely to annotate human reflections as coherent in the GPT-3 stage. In this section, we further inspect whether each layperson/expert annotates human reflections consistently across stages. Since the workload of each annotator consists of non-recurring human reflections (appearing in
only one stage) and sometimes also recurring human reflections (appearing in both stages), we probe the shift of each annotator on these two types of human reflections separately.
We first examine how often each annotator flags recurring human reflections identically (namely choosing “Yes” in both stages or “No” in both) across stages. As shown in Table 16, 8 laypeople and 8 experts have recurring human reflections in their workload. Among those annotators, 3 laypeople and 4 experts fail to annotate all (100%) recurring human reflections identically across stages. Overall, laypeople and experts annotate recurring human reflections identically 71% and 73% of the time, respectively. Those similar numbers are evidence that the laypeople-experts difference in the between-phase waiting period duration (Appendix C) is not critical.
Then, we investigate whether each annotator flags non-recurring human reflections more, equally or less often as Coherent in the GPT-3 stage than in the GPT-2 stage. As table 17 shows, 5 laypeople less often annotate non-recurring human reflections as Coherent in the GPT-3 stage, 1 does so more often, while the other 3 stay at the same level across stages. Among the experts, 4 give Coherent annotations less often, 2 do so more often, while the remaining 3 do not show cross-stage frequency change, which is a similar distribution compared to laypeople. Considering that laypeople and experts have different levels of overall cross-stage shift on non-recurring reflections — 29% for laypeople and 6% for ex-
perts — we posit that laypeople and experts differ less in the proportion of “shifting” annotators but more in the magnitude of shifts displayed by individual annotators.

heading: E Label Distribution for Differently Generated Synthetic Reflections
text: Table 18 shows the distribution of Coherent and error labels for synthetic reflections from GPT-2 and GPT-3 under different generation settings.
For GPT-2 reflections, larger p values in nucleus decoding cause less coherent reflections, especially when p ∈ {0.8, 0.95}. This is unsurprising, since larger p’s give the model more freedom in generation and thus also make it more prone to errors.
For GPT-3, reflections generated through textbook-based in-context learning are overall less coherent than reflections generated through AnnoMI-based in-context learning. This is not surprising, since test examples themselves are from AnnoMI, which means examples from AnnoMI are more useful in helping the model learn to produce coherent reflections for long dialogue contexts.
Among reflections from GPT-3 (textbook), simple reflections are overall more often annotated as Parroting than complex ones, especially by laypeople. This is likely because simple reflections mostly repeat/rephrase what the client said, which may appear repetitive to a layperson when an expert would more likely consider it good practice (§4.3.2).
Finally, we note that the Coherent rates of GPT3 reflections can vary considerably under different nucleus decoding p’s but without a clear trend,
which we leave to future work to probe.

heading: F Data Use & Creation
text: We leveraged AnnoMI, a dataset available under the Public Domain license. We used it for research purposes, which is consistent with its intended use. While AnnoMI contains therapy dialogues, the data does not reveal personal information since the dialogues are transcripts of professionally produced MI demonstrations. The dataset does not reveal demographic information, but we observe that the dialogues seem to be set in English-speaking countries.
Based on AnnoMI, we created a dataset of human annotations w.r.t. coherence of reflections, and we release it11 under the CC BY-NC license, which is also compatible with the access conditions of AnnoMI. The human annotations do not reveal any information of the laypeople or experts, and we use L1∼9 to represent the 9 laypeople and E1∼9 to represent the 9 experts. We discussed the demographic information of the annotators in the Ethics Statement.

heading: G Annotation Flow
text: In practice, each annotation batch contained some parts that are not investigated in this study, which are therefore not shown in the main body. The complete annotation flow is detailed below.
As shown in Figure 4, a batch starts with the annotator reading the context. Then, the annotator reads one reflection and chooses Yes/No regarding whether it is coherent and context-consistent. If the answer is Yes, the annotator assesses the level of empathy displayed in the reflection. If the answer is No, the annotator selects one or more error categories that apply, and in the case of multiple selected errors the annotator further pinpoints the most evident one. Afterwards, the annotator proceeds to annotate the next reflection in the same steps, and the batch ends when all its reflections have been annotated.

heading: H Annotation Interface
text: The annotation process takes place in the Mechanical Turk Sandbox12. Details of the annotation interface are shown in Figures 5, 6 and 7. We note that there is a purposely off-topic reflection in each
11Available at https://github.com/uccollab/ expert_laypeople_reflection_annotation.
12https://workersandbox.mturk.com/
batch as an anti-scam mechanism, which is why there appear to be 11 reflections instead of 10 to annotate in those figures.

heading: I Colour-Blind-Safe and Greyscale-Safe Version of Figure 3
text: Figure 8 shows the colour-blind-safe and greyscalesafe version of Figure 3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section (unnumbered) after the conclusion

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement -> Risks

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract is at the beginning; Introduction is Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 1) We used a dataset (AnnoMI), and it is mentioned multiple times in the paper. The first mention is in the Introduction. 2) We also created a dataset of human annotations (evaluations), and it is mentioned multiple times in the paper. The first mention is in the Introduction. Section 3 describes our methodology.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: AnnoMI is first cited in the Introduction (Table 1), and formally introduced in Section 3.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix F

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix F

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix F

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3.2; Ethics Statement; Appendix F
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.1 and Appendix A
C 3 Did you run computational experiments? Section 3.1 and Appendix A

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3.1 and Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.1 and Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sections 3 & 4; Appendices B, D and E. We note that this is a human evaluation study, so some conventional descriptive statistics are not applicable, e.g., max/mean/single run. Nevertheless, we did use a variety of decoding parameters to generate diverse texts for the annotators to evaluate.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.1, Appendices A & B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.2 and Ethics Statement

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix H and Supplementary Material (.zip file uploaded)

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.2 and Ethics Statement

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethics Statement

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics Statement

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Ethics Statement

heading: Future Work
text: Finally, we note that the Coherent rates of GPT3 reflections can vary considerably under different nucleus decoding p’s but without a clear trend,
which we leave to future work to probe