An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives
Authors: Young Min Cho
Sunny Rai
Lyle Ungar
João Sedoc
Sharath Chandra Guntuku
Venue: EMNLP 2023 Main (2023)
Keywords: Conversational Agent
Chatbot
Mental health
TL;DR: A comprehensive literature review using the PRISMA framework from both Computer Science and Medical field

Abstract: Mental health conversational agents (a.k.a. chatbots) are widely studied for their potential to offer accessible support to those experiencing mental health challenges. Previous surveys on the topic primarily consider papers published in either computer science or medicine, leading to a divide in understanding and hindering the sharing of beneficial knowledge between both domains. To bridge this gap, we conduct a comprehensive literature review using the PRISMA framework, reviewing 534 papers published in both computer science and medicine. Our systematic review reveals 136 key papers on building mental health-related conversational agents with diverse characteristics of modeling and experimental design techniques. We find that computer science papers focus on LLM techniques and evaluating response quality using automated metrics with little attention to the application while medical papers use rule-based conversational agents and outcome metrics to measure the health outcomes of participants. Based on our findings on transparency, ethics, and cultural heterogeneity in this review, we provide a few recommendations to help bridge the disciplinary divide and enable the cross-disciplinary development of mental health conversational agents.

An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives
Young-Min Choi
Sumny Rai
Lyle Ungar
João Sedoe² Sharath Chandra Guntuku!
UUniversity of Pennsylvania "New York University
{jcho, sunnyrain sharathg)@seas. upenn. upenn.edu, jsedoc@stern.nyu edu
Abstract
individuals in high-income countries do not receive care and treatment (Kohn al., 2004; Arias et al., 2022). The sensitive (and often stigmatized) nature of mental health discussions further exacerbates this problem, as many individuals find it difficult to disclose their struggles openly (Corrigan and Matthews, 2003).
Mental health conversational agents (a.k.a. chatbots) widely studied for their poten- tial to offer accessible support to those experi- encing mental health challenges. Previous sur- veys on the topic primarily consider papers pub- lished either computer science or medicine, leading divide in understanding and hin- dering the sharing of beneficial knowledge be- tween both domains. bridge this gap, we conduct comprehensive literature review us- ing the PRISMA framework, reviewing 534 papers published in both computer science and medicine. Our systematic review reveals 136 key papers on building mental health-related conversational agents with diverse characteris- tics of modeling and experimental design tech- niques. We find that computer science papers focus LLM techniques and evaluating re- sponse quality using automated metrics with little attention to the application while medi- cal papers use rule-based conversational agents and outcome metrics to measure the health out- comes of participants. Based on our findings on transparency, ethics, and cultural heterogeneity this review, we provide a few recommenda- tions to help bridge the disciplinary divide and enable the cross cross- dissssiicppl development of mental health conversational agents.
Conversational agents like Woebot (Fitzpatrick et al., 2017) and Wysa (Inkster al., 2018) were some of the first mobile applications to address this issue. They provide an accessible and consider- ably less intimidating platform for mental health support, thereby assisting a substantial number of individuals. Their effectiveness highlights the po- tential of mental realth-focusse CAs one of the viable solutions to the mental health disclosure and treatment gap.
Despite the successful implementation of certain CAs in mental health, a significant disconnect per- sists between research in computer science (CS) and medicine. This disconnect is particularly ev- ident when consider the limited adoption of advanced NLP (e.g. large language models) mod- els in the research published in medicine. While CS researchers have made substantial strides in NLP, there lack of focus on the human evaluation and direct impacts these developments have on pa- tients. Furthermore, we observe that mental health CAs are drawing significant attention in medicine, yet remain underrepresented in health-applications- focused research in NLP. This imbalance calls for more integrated approach in future studies to op- timize the potential of these evolving technologies for mental health applications.
1 Introduction
The proliferation of conversational agents (CAs), also known as chatbots or dialog systems, has been spurred by advancements in Natural Language Processing (NLP) technologies. Their application spans diverse sectors, from education (Okonkwo and Ade-Ibijola, 2021; Durall and Kapros, 2020) to e-commerce (Shenoy et al., 2021), demonstrating their increasing ubiquity and potency.
In this paper, we present a comprehensive anal- ysis of academic research related to mental health conversational agents, conducted within the do- mains of CS and medicine'. Employing the Pre- ferred Reporting Items for Systematic Reviews
The utility of CAs within the mental health do- main has been gaining recognition. Over 30% of the world's population suffers from one or more mental health conditions; about 75% individuals in low and middle -ncole countries and about 50%
'Our data and papers are availahle on our GitHub: http:giuihh.oo////ecchhhhhhhaaae chatbot survey

---

and Meta-Analyses (PRISMA) framework (Mo- her al., 2010), we systematically reviewed 136 pertinent papers to discern the trends and research directions in the domain of mental health conversa- tional agents over the past five years. We find that there is a disparity in research focus and technol- ogy across communities, which is also shown in the differences in evaluation. Furthermore, out the issues that apply across domains, including transparency and language/cultural heterogeneity. The primary objective of our study is to con- duct a systematic and transparent review of mental health CA research papers across the domains of CS and medicine. This process aims not only to bridge the existing gap between these two broad disciplines but also to facilitate reciprocal learning and strengths sharing. In this paper, we aim to address the following key questions:
Idenciiicaioon
Records identified through database searching (n=082)
Additional records identifled through other sources (n=3)
Records duplicates removed (n=534)
Soreening
point
Records screened (n=534)
Records excluded (n=377)
Eligbility
Full-text articles assessed for eliibility (n=157)
Records excluded (n=21)
Inclused
Studies included in qualitative syntheais (36)
Figure I: Pipeline of our PRISMA framework.
1. What are the prevailing focus and direction of research in each of these domains?
tecture despite the growing popularity of end-to- end architectures in the NL. domain. A similar technical review by Safi et al. (2020) also reports a high reliance on static dialogue systems in CAs developed for medical applications. Task-oriented dialogue systems usually deploy a guided conver- sation style which fits well with rule-based sys- tems. However, Su et al. (2020); Abd-Alrazaq et al. (2021) pointed to the problem of robotic conversa- tion style in mental health apps where users prefer an unconstrained conversation style and may even want to lead the conversation (Abd-Alrazag et al., 2019). Huang (2022) further underlines the need for self-evolving CAs to keep up with evolving habits and topics during the course of app usage. Surveys from the rest of CS cover HCI (de Souza et al., 2022) and the system design of CAs (Dev et al., 2022; Narynov al., 2021a). de Souza al. (2022) analyzed 6 mental health mobile applica- tions from an HCI perspective and suggested 24 design considerations including emputhetic con- versation style, probing, and session duration for effective dialogue. Damij and Bhattacharya (2022) proposed three key dimensions namely people (cit- izen centric goals ), process (regulations and gover- nance) and Al technology to consider when design- ing public care CAs.
2. What key differences can be identified be- tween the research approaches taken by each domain?
3. How can we augment and improve mental health CA research methods?
2 Prior Survey Papers
Mental health conversational agents are discussed in several non-CS survey papers, with an emphasis on their usability in psychiatry (Vaidyam et al., 2019; Montenegro et al., 2019; Laranjo et al.,
2018), and users' acceptability (Koulouri et al.,
2022; Gaffney et al., 2019). These survey papers focus on underpinning theory (Martinengo et al., 2022), standardized psychological outcomes for evaluation (Vaidyam et al., 2019; Gaffney et al., 2019) in addition to accessibility (Su et al., 2020), safety (Parmar al., 2022) and validity (Pacheco- Lorenzo et al., 2021; Wilson and Marasoiu, 2022) of CAs.
Contrary to surveys for medical audiences, NLP studies mostly focus on the quality of the generated response from the standpoint of text generation. Valizadch and Parde (2022) in their latest survey, reviewed 70 articles and investigated task-oriented healthcare dialogue systems from a technical per- spective. The discussion focuses on the system architecture and design of CAs. The majority of healthcare CAs were found to have pipeline archi-
These survey papers independently provide an in-depth understanding advancements and chal- lenges in the CS and medical domains. However, there is a lack of studies that can provide a joint

---

appraisal of developments enable cross-learning across these domains. With this goal, consider research papers from medicine (PubMed), NLP (the ACL Anthology), and the rest of CS (ACM, AAAI, IEFE) to examine the disparities in goals, methods, and evaluations of research related to mental health conversational agents.
Screening Process
ACI.
AAAT
IEEE
ACM
PubMed
Database Search
68
30
52
280
104
Title
26
16
39
137
84
Screening
Abstract
9
4
31
45
68
Screening
3 Methods
Full-Text Screening
9
4
20
40
63
3.1 Paper Databases
Model Experiment
6
3
15
35
43
We souree papers from eminent databases in the fields of NLP, the rest of CS, and medicine, as these are integral knowledge areas in the study of men- tal health CA. These databases include the ACL Anthology (referred to as ACL. throughout this pa- per)?, AAAI², IEEE*, ACMS, and PubMed ACL is recognized a leading repository that highlights pioneering research in NLP. AAAI features cutting- edge studies in AI. IBEE. leading community, embodies the forefront of engineering and tech- nology research. ACM represents the latest trends in Human Computer Interaction (HCT) along with several other domains of CS. PubMed, the largest search engine for science and biomedical topics including psychology, psychiatry, and informatics among others provides extensive coverage of the medical spectrum.
Table I: Steps screening process and the number
of papers retained in each database.
ically, we include paper if it met the following conditions for a focused and relevant review the literature that aligns with the objectives of our study:
Primarily focused on CAs irrespective modality, such as text, speech, or embodied.
Related to mental health and well-being, These could be related to depression, PTSD, or other conditions defined in the DSM- IV (Bell, 1994) other emotion-related inter- vention targets such stress.
Contribute towards directly improving mental health CAs. This could be proposing novel models or conducting user studies.
Drawing on insights from prior literature reviews (Valizadeh and Parde, 2022; Montenegro et al., 2019; Laranjo et al., 2018) and discussion with experts from both the CS and medical domains, we opt for combination of specific keywords. These search terms represent both our areas of focus: con- versational agents ("conversational agent", "chat- bot") and mental health ("mental health", "depres- sion"). Furthermore, we limit our search criteria to the paper berween 2017 to 2022 to cover the most recent articles. We also apply the "research article" filter on ACM search, and "Free Full Text or Full Text" for PubMed search. Moreover, we manually add papers recommended by the domain experts (Fitzpatrick et al., 2017: Laranjo et al., 2018; Mon- tenegro et al., 2019). This results in 534 papers.
The initial step in our screening process is title screening, in which we examine all titles, retaining those that are related to either CA or mental health. Our approach is deliberately inclusive during this phase to maximize the recall. As a result, out of 534 papers, we keep 302 for the next step.
Following this, we proceed with abstract screen- ing. In this stage, we evaluate whether each paper meets our inclusion criteria. To enhance the accu- racy and efficiency of our decision-making process, we extract the ten most frequent words from the full text of each paper to serve as keywords. These key- words provide additional layer of verification, assisting our decision-mate makiin process. Following this step, we are left with a selection of 157 papers.
3.2 Screening Process
The final step is full-text screening. When we verify if paper meets the inclusion criteria, we extract key features (such as model techniques and evaluations) from the paper and summarize them in tables (see appendix). Simultaneously, we high- light and annotate the papers' PDF files to provide evidence supporting our claims about each feature
For subsequent steps in the screening process, we adhere to set of defined inclusion criteria. Specif-
hnttps://aclantthllee org/
Phttps://aaal http://aaa....gaaaaapaaaaaaa palai-ppllccaionnss
"https://ieeexplor" org/
chttps://dl.acc acm.///lllaam.rrg.g... org/
ncbi, nlm.nih.gov.

---

similar the methodology used in Howcroft et al. (2020). This process is independently conducted by two co-authors on a subset of 25 papers, and the annotations agree with each other. Furthermore, the two co-authors also agree upon the definition of features, following which all the remaining papers receive one annotation.? 7
Language
CS
Med
All
English
47
30
77
Chinese
1
5
6
Korean
4
5
German
1
2
Italian
1
1
2
Portuguese
0
2
2
The final corpus contains 136 papers: 9 from ACI., 4 from AAAI, 20 from IEEE, 40 from ACM, and 63 from PubMed. We categorize these papers into four distinct groups: 102 model/experiment pa- pers, 20 survey papers, and the remaining 14 papers are classified as 'other'. Model papers are articles whose primary focus is on the construction and explanation of theoretical model, while experi- mental papers are research studies that conduct spe- cific experiments the models to answer pertinent research questions. We combine experiment and model papers together because experimental pa- pers often involve testing on models, while model papers frequently incorporate evaluations through experiments. The 'other' papers include dataset papers, summary papers describing the proceed- ings of workshop, perspectives/viewppni papers, and design science research papers. In this paper, we focus analyzing the experiment/model and survey papers, which have more uniform set of features.
Other
5
8
Table 2: Distribution of predominant language of the data and/or participants recruited in mental health CA papers. Other languages include Bangla, Danish, Dutch, Japanese, Kazakh, Norwegian, Spanish, and Swedish.
Mental Health Category
CS
Med
All
Not Specified
32
21
53
Depression
9
10
19
Anxiety
8
16
Stress
0
4
Sexual Abuse
3
0
3
Social Isolation
3
3
Other
14
25
Table 3: Distribution of mental health category in men- tal health CA papers. A paper could have multiple focused targets. Other categories include affective disor- der, COVID-19, eating disorders, PTSD, substance use disorder, etc.
3.3 Feature Extraction
15 from IEEE, 35 from ACM, and 43 from PubMed. In this section, we briefly summarize the observa- tions from the different features we extracted.
We extract set of 24 features have a detailed and complete overview of the recent trend. They in- clude general features ("paper type", "language", "mental health category", "background", "target group", "targel demographic") techniques ("chat- bot name", "chatbot type", "model technique", "off the shelf", "outsourced model name", "train- ing data"), appearance ("interface", "embodi- ment", "platform", "public access") "), and exper- iment ("study design", "recruitment", "sample size", "duration", "automatic evaluation", "hu- man evaluation", "statistical test", "ethics"). Due to the limited space, we present a subset of the features in the main paper. Description of other features can be found in Appendix.
4.1 Language
We identify if there a predominant language as- sociated with either the data used for the models or there is a certain language proficiency that was a part of the inclusion criteria for participants. Our findings, summarized in Table 2, reveal that En- glish dominates these studies with over 71% % of the papers utilizing data and/or participants proficient in English. Despite a few (17%) papers emerging from East Asia and Europe, we notice that studies in low-resource languages are relatively rare.
4.2 Mental Health Category
4 Results
Most of the papers (43%) we reviewed do not deal with a specific mental health condition but work towards general mental health well-being (Saha et al., 2022a). The methods proposed in such pa- pers are applicable to the symptoms associated with a broad range of mental health issues (e.g. emo-
Under the category of model and experiment pa- pers, there are 6 papers from ACL, from AAAI,
Annotated PDF files with evidence each feature are available our GitHub.
FFull feature table is available in the supplemental material.

---

Target Demographic
CS
Med
All
Model Technique
CS
Med
All
General
43
26
69
Retrieval-Based
27
22
49
Young People
4
6
10
Rule-Based
23
19
42
Students
5
3
8
Generative
10
0
10
Women
3
4
7
Not Specified
3 3
6
Older adults
5
5 Distribution of model techniques used in mental health CA papers. A paper could use multiple modeling techniques. The Not Specified group includes papers without a model but employing surveys to ask people's ss opinions and suggestions towards mental health CA.
Other
4
5
Table Distribution of demographics focused by men- tal health CA papers. A paper could have multiple fo- cused target demographic groups. Other includes black American, the military community, and employee.
focus on the mental health care of women, for in- stance in prenatal and postpartum women (Green et al., 2019; Chung et al., 2021) and sexual abuse survivors (Maeng and Lee, 2022; Park and Lee, 2021). Papers targeting older adults are mainly de- signed for companionship and supporting isolated elders (Sidner et al., 2018; Razavi al., 2022).
tional dysregulation). Some papers, the other hand, more tailored address the characteris- ties of targeted mental health conditions. As shown in Table 3, depression and anxiety are two major mental health categories being dealt with, reflect- ing the prevalence of these conditions (Eagle et al., 2022). Other categories include stress management (Park et al., 2019; Gabrielli et al., 2021); sexual abuse, to help survivors of sexual abuse (Maeng and Lee, 2022; Park and Lee, 2021), and social isolation, mainly targeted toward older adults (Sid- ner al., 2018; Razavi et al., 2022). Less-studied categories include affective disorders (Maharjan mental health
4,4 Model Technique
Development of Large Language Models such GPT-series (Radford et al., 2019; Brown et al.. 2020) greatly enhanced the performance of gener- ative models, which in turn made significant im- pact on the development of CAs (Das al., 2022b; Nie et al., 2022). However, as shown in Table 5, LLMs are yet to be utilized in the development of mental health CAs (as of the papers reviewed in this study), especially in medicine. No paper from PubMed in our final list dealt with generative mod- cls, with the primary focus being rule-based and retrieval-based CAs.
et al., 20,2a,b), issues (Kim et al., 2022; Ludin et al., 2022), eating disorders (Beilharz et al., 2021), and PTSD (Han et al., 2021).
4.3 Target Demographic
Most of the papers (>65%) do not specify the tar- get demographic of users for their CAs. The target demographic distribution is shown in Table 4. An advantage of the models proposed in these papers is that they could potentially offer support to broad group of users irrespective of the underlying mental health condition. Papers without a target demographic and target mental health category focus on proposing methods such as using gen- erative language models for psychotherapy (Das et al., 2022a), or to address specific modules of the CAs such as leveraging reinforcement learn- ing for response generation (Saha et al., 2022b). spe-
Ruse-based models operate on predefined rules and patterns such as if-then statements or deci- sion trees to match user inputs with predefined responses. The execution of Rule-based CAs can be straightforward and inexpensive, but develop- ing and maintaining comprehensive set of rules can be challenging. Retrieval-base bassed models rely on a predefined database of responses to gener- ate replies. They use techniques like keyword matching (Daley et al., 2020), similarity measures (Collins et al., 2022), or information retrieval (Mor- ris et al., 2018) to select the most appropriate re- sponse from the database based on the user's input. Generative model-based CAs mostly developed using deep learning techniques such as recurrent neural networks (RNNs) or transformers, which learn from large amounts of text data and generate responses based on the learned patterns and struc-
On the other hand, 31% papers focus cific user group such as young individuals, stu- dents, women, older adults, etc, to give advanced assistance. Young individuals, including adoles- cents and teenagers, received the maximum atten- tion (Rahman et al., 2021). Several papers also

---

Outsourced Model
CS
Med
All
the leading mental health Al assistant that supports over 30M individuals with easy access.
Google Dialogflow
2
13
Rasa
5
5
10
Alexa
4
4.6
Evaluation
DialoGPT
3
0
3
Mental health CAs are evaluated with various methods and metries. Multiple factors, including user activity (total sessions, total time, days used, total word count), user utterance (sen- timent analysis, LIWC (Pennebaker et al., 2015)), CA response quality (BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), lexical diversity, perplex- ity), and performance of CA's sub-modules (classi- fication fl score, negative log-likelihood) are mea- sured and tested. We find that papers published in the CS domain focus more on technical evaluation, while the papers published in medicine are morc interested in user data.
Automatic:
GPT
3
0
3
X2AI
3
3
Other
17
6
23
Table 6: Distribution of outsourced models used for building models used in mental health CA papers. Other includes Manychat', Woebot (Fitzpatrick et al., 2017) and Eliza (Weizenbaum, 1966).
tures. While they can often generate more diverse and contextually relevant responses compared to rule-based or retrieval-basee models, they could suffer from hallucination and inaccuracies (A (Azaria and Mitchell, 2023).
Human outcomes: Human evaluation using sur- vey assessment is the most prevalent method to gauge mental health CAs' performance. Some sur- vey instruments measure the pre- and post-study status of participants and evaluate the impact of the CA by comparing mental health (e.g. PHQ- (Kroenke al., 2001), GAD-7 (Spitzer et al., 2006), BFI-10 (Rammstedt et al., 2013)) and mood scores (e.g. WHO-5 (Topp et al., 2015)), or collect- ing user feedback on CA models (usability, diffi- culty, appropriateness), or asking a group of indi- viduals to annotate user logs or utterances to collect passive feedbacks (self-disclosure level, compe- tence, motivational..
4.5 Outsourced Models
Building CA model from scratch could be chal- lenging for several reasons such as a lack of suffi- cient data, compute resources, or generalizability. Publicly available models and architectures have made building CAs uccessible. Google Dialogflow (Google, 2021) and Rasa (Bocklisch et al., 2017) are the two most used outsourced platforms and frameworks. Alexa, DialoGPT (Zhang et al., 2019),
GPT (2 and 3) (Radford al., 2019; Brown al.. 2020) and X2AT (now called Cass) (Cass, 2023)
are also frequently used for building CA models. A summary can be found Table 6.
Google Dialogflow is a conversational Al plat- form developed by Google that enables develop- ers to build and deploy chatbots and virtual assis- tants across various platforms. Rasa is open- source conversational AI framework that empow- ers developers to create and deploy contextual chatbots and virtual assistants with advanced natu- ral language understanding capabilities. Alexa is a voice-controlled virtual assistant developed by Amazon. It enables users to interact with a wide range of devices and services using voice com- mands, offering capabilities such playing music, answering questions, and providing personalized recommendations, DialoGPT is a large, pre-trained neural conversational response generation model that is trained on the GPT2 model with 147M conversation-lik exchanges from Reddit. X2AI is
4.7 Ethical Considerations
Mental health CAs inevitably work with sensitive data, including demographics, Personal Identifiable Information (PIT), and Personal Health Information (PHI). Thus, careful ethical consideration and a high standard of data privacy must be applied in the studies. Out of the 89 papers that include hu- man evaluations, approximately 70% (62 papers) indicate that they either have been granted approval by Institutional Review Boards (IRB) or ethics re- view committees or specified that ethical approval is not requirement based on local policy. On the other hand, there 24 papers that do not mention seeking ethical approval or consequent consider- ations in the paper. Out of these 24 papers that lack a statement on ethical concerns, 21 papers are published the field of CS.
"https://manychat...

---

5 Discussion
PubMed papers rather focus on retrieval-based and rule-based methods, which are, arguably, previous-generation CA models as far as the techni- cal complexity is concerned. This could be due to a variety of factors such as explainability, accuracy, and reliability which are crucial when dealing with patients.
5.1 Disparity in Research Focus
Mental health Conversational Agents require expert knowledge from different domains. However, the papers we reviewed, treat this task quite differently, evidenced by the base rates of the number of pa- pers matching our inclusion criteria. For instance, there are over 28,000 articles published in the ACL Anthology with the keywords "chatbot" or "conver- sational agent", which reveals the popularity of this topic in the NLP domain. However, there are only 9 papers related to both mental health and CA, which shows that the focus of NLP researchers is primar- ily concentrated on the technical development of CA models, less on its applications, including men- tal health. AAAI shares a similar trend as ACL. However, there are a lot of related papers to mental health CAs in IEEE and ACM, which show great interest from the engineering and HCT community. PubMed represents the latest trend of research in the medical domain, and it has the largest number of publications that fit our inclusion criteria. While CS papers mostly do not have a specific focus on the mental health category for which CAs are being built, papers published in the medical domain often tackle specific mental health categories.
5.3 Response Quality vs Health Outcome
The difference in evaluation also reveals the vary- ing focus across CS and medicine domains. From the CS domains, 30 (of 59 papers) applied auto- matic evaluation, which checks both model's per- formance (c.g. BLEU, ROUGE-L, perplexity) and participant's ' CA usage (total sessions, word count, interaction time). In contrast, only 13 out of 43 papers from PubMed used automatic evaluation, and none of them investigated the models' perfor- mance.
The difference is also spotted in human eval- uation. 40 (of 43 papers) from PubMed consist of human outcome evaluation, and they cover a wide range of questionnaires to determine partici- pants' status (e.g. PHQ-9, GAD-7, WHO-5). The focus is on users' psychological well-being and evaluating the chatbot's 's suitability in the clinical setup (Martinengo et al., 2022). Although these papers do not test the CA model's performance through automatic evaluation, they asked for par- ticipants' ratings oversee their model's 's quality (e.g. helpfulness, System Usability Scale (Brooke et al., 1996), WAI-SR (Munder et al., 2010)).
5.2 Technology Gap
CS and medical domains are also different in the technical aspects of the CA model. In the CS domain (ACL, AAAI, IEEE, ACM), 41 (of 73 papers) developed CA models, while 14 (out of 63) from the medical domain (PubMed) developed models. Among these papers, 8 from the CS do- main are based on generative methods, but no paper in PubMed uses this technology. The NLP com- munity is actively exploring the role of generative LLMs (e.g. GPT-4) in designing CAs including mental healthcare-related CAs (Das al., 2022a; Saha al., 2022b: Yan and Nakashole, 2021). With the advent of more sophisticated LLMs. fluency. repetitions and, ungrammatical formations are no longer concerns for dialogue generation. However, stochastic text generation coupled with black box architecture prevents wider adoption of these mod- cls in the health sector (Vaidyam et al., 2019). Un- like task-oriented dialogues, mental health domain CAs predominantly involve unconstrained conver- sation style for talk-therapy that can benefit from the advancements in LLMs (Abd-All Abbd-Alrazaq et al., 2021). I).
All 6 ACL papers that satisfied our search cri- teria, solely focus on dialogue quality (c.g. flu- ency, friendliis etc.) with no discussion on CA's effect on users' well-being through clinical mea- sures such as PHQ-9. CAs that aim be the first point of contact for users seeking mental health sup- port, should have clinically validated mechanisms to monitor the well-being of their users (Pacheco- Lorenzo et al., 2021; Wilson and Marasoiu, 2022). Moreover, the mental health CAs we review are designed without any underlying theory for psy- chotherapy behavior change that puts the utility of CAs providing emotional support to those suf- fering from mental health challenges in doubt.
5.4 Transparency
None of the ACL papers that we reviewed released their model or API. Additionally, baseline or com- parison with the existing slate-ofthhe-n the-art model is often missing in the papers. There is no standard-

---

ized outcome reporting procedure in both medicine and CS domains (Vaidyam al.. 2019). For in- stance, Valizadeh and Parde (2022) raised concerns about the replicability of evaluation results and transparency for healthcare CAs. We acknowledge the restrictions posed to making the models public due to the sensitive nature of the data. However, providing APIs could be a possible alternative to enable comparison for future studies. To gauge the true advantage of mental health CAs in a clinical setup, randomized control trials are an important consideration that is not observed in NLP papers. Further, standardized benchmark datasets for evalu- ating mental health CAs could be useful in increas- ing transparency.
studies. They vary in research focus, technology. and evaluation purposes. We also identify com- mon issues that lie between domains, including transparency and language/cultural heterogeneity.
Potential Recommendations
We systematically study the difference between do- mains and show that learning from each other is highly beneficial. Since interdisciplinary works consist of a small portion of our final list (20 over 102 based author affiliations on papers; 7 from ACM, 2 from IEEE, and 11 from PubMed), we suggest more collaborations help bridge the gap between the two communities. For instance, NLP (and broadly CS) papers on mental health CAs would benefit from adding pre-post analysis on hu- man feedback and considering ethical challenges by requesting review of an ethics committee. Fur- ther, studies in medicine could benefit by tapping into the latest developments in generative meth- ods in addition to the commonly used rule-based methods. terms of evaluation, both the qual- ity of response by the CAs (in terms of automatic metrics such as BLEU, ROUGE-L, perplexity, and measures of dialogue quality) as well as the effect of CA on users' mental states (in terms of mental health-specific survey inventories) could be used to assess the performance of mental health CAs. Moreover, increasing the language coverage in- clude non- non-English data/participants and adding cul- tural heterogeneity while providing APIs to com- pare against current mental health CAs would help in addressing the challenge of mental health care support with a cross cooss-disciplinary effort.
5.5 Language and Cultural Heterogeneity
Over 75% of the research papers in our review cater English-speaking participants struggling with de- pression and anxiety. Chinese and Korean are the two languages with the highest number of research papers following English, even though Chinese is the most populous language in the world. Fu- ture works could consider tapping into a diverse set of languages that also have a lot of data avail- able for instance, Hindi, Arabic, French, Rus- sian, and Japanese. which are among the top 10 most spoken languages in the world. The growing prowess of multilingual LLMs could be an incred- ible opportunity to transfer universally applicable development in mental health CAs to low-resource languages while being mindful of the racial and cultural heterogeneity which several multilingual models might miss due to being trained on largely English data (Bang al., 2023).
Limitations
6 Conclusion
This survey paper has several limitations. Our search criteria are between January 2017 to De- cember 2022, which likely did not reflect the devel- opment of advanced CA and large language models like ChatGPT and GPT4 (Sanderson, 2023). We couldn't 't include more recent publications to meet the EMNLP submission date. Nonetheless, we have included relevant comments across the differ- ent sections on the applicability of more sophisti- cated models.
In this paper, we used the PRISMA framework systematically review the recent studies about mental health CA across both CS and medical do- mains. From the well-represented databases both domains, we begin with 865 papers based on a key- word search to identify mental health--rlated con- versational agent papers and use title, abstract, and full-text screening to retain 136 papers that fit our inclusion criteria. Furthermore, we extract wide range of features from model and experiment pa- pers, summarizing attributes in the fields of general features, techniques, appearance, and experiment. Based on this information, we find that there is a gap between CS and medicine in mental health CA
Further, search engines (e.g. Google Scholar) are not deterministic. Our search keywords, filters, and chosen databases do not guarantee the exact same search results. However, we have tested mul- tiple times on database searching and they returned

---

consistent results. We have downloaded PDF's of all the papers and have saved the annotated them to reflect the different steps used in this review paper. These annotations will be made public. For some databases, the number of papers in the final list may be (surprisingly!) small to represent the general research trends in the respective do- mains. However, it also indicates the lack of focus on mental health CA from these domains, which also proposes further attention is required in the field.
manuscript: and decision submit the manuscript for publication.
References
Tahir Abbas, Vassilis-Javed Khan, Ujwal Gadiraju, and Panos Markopoulos, 2020. Trainbot: A conversa- tional interface to train crowd workers for delivering on-demand therapy. In Proceedings of the AAAI Con- ference on Human Computation and Crowdsourcing. volume 8, pages 3-12.
Alaa Abd-AIrazaq, Mohannad Alajlani, Ali Abdal- lah Alalwan, Bridgette M Bewick, Peter Gardner, and Mowafa Housch. 2019, An overview of the features of chatbots in mental health: A scoping re- view. International Journal of Medical Informatics, 132: 103:703
Ethics Statement
Mental Health CAs, despite their accessibility, po- tential ability, and anonymity, cannot replace hu- man therapists in providing mental health care. There are a lot of ongoing discussions about the range of availability of mental health CAs, and many raise several challenges and suspicions about automated conversations. Rule-based and retrieval- based models can be controlled for content gener- ation, but cannot answer out-of-domain questions. Generative models are still a developing field, their non-deterministic nature raises concerns about the safety and reliability of the content. Thus at the cur- rent stage, CA could play a great supporting com- plementary role in mental healthcare identify individuals who potentially need more immediate care in an already burdened healtheare system.
Alaa A Abd-AIrazaq, Mohannad Alajlani, Nashva Ali, Kerstin Denecke, Bridgette M Bewick, and Mowafa Househ. 2021. Perceptions and opinions of patients about mental health chatbots: scoping review. Jour- nol of medical Internet research, 23( I
Mohammad Rafayet Ali, Seyedch Zahra Razavi, Raina Langevin, Abdullah Al Mamun, Benjamin Kane, Reza Rawassizadeh, Lenhart K Schubert, and Ehsan Hoque. 2020. A virtual conversational agent for teens with autism spectrum disorder: Experimental results and design lessons. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pages 1-8.
Daniel Arias, Shekhar Saxena, and Stéphane Verguet. 2022. Quantifying the global burden of mental disor- ders and their economic value. ECltnicalMedicine, 54:101675.
Since the patient's personal information and medical status are extremely sensitive, we highly encourage researchers and developers to pay extra attention to data security and ethics Arias et al. (2022). The development, validation, and deploy- ment of mental health CAs should involve multi- ple diverse stakeholders to determine how, when, and which data is being used to train and infer participants' mental health. This effort requires a multidisciplinary effort to address the complex challenges of mental health care (Chancellor et al., 2019).
Amos Azaria and Tom Mitchell. 2023. The internal state of an Ilm knows when its lying, arXiv preprint arXiv:2304.1 137344
Petter Bae Bae Brandtzæg, Marita Skjuve, Kim Kristof- fer Kristoffer Dysthe, and Asbjorn Følstad. 2021. When the social becomes non-human: young peo- ple's perception of social support chatbots. Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1-13.
Acknowledgements
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.0400.
We would like to thank the reviewers for their fruit- ful discussion with us. This work was partly sup- ported by grant NIMHD: R01MD018340 from the National Institutes of Health and Penn Global Re- search Engagement Fund. The funders had no role in the design and conduct of the study; col- lection, management, analysis, and interpretation of the data; preparation, review, or approval of the
Clare Beatty, Tanya Malik, Saha Meheli, and Chaitali Sinha. 2022. Evaluating the therapeutic alliance with free-text cbt conversational agent (wysa): A methoss study. Frontiers in Digital Health, 4:847991.

---

Francesca Beilharz, Suku Sukunesan, Susan Rossell, Jayashri Kulkarni, Gemma Sharp, et al. 2021. Devel- opment of positive body image chatbot (kit) with young people and parents/carers: qualitative focus group study. Journal of Medical Internet Research, 23(6):27807.
Stevie Chancellor, Michael L Birnbaum, Eric Caine. Vincent MB Silenzio, and Munmun De Choudhury. 2019. A taxonomy of ethical tensions in inferring mental health states from social media. In Proceed- ings of the conference on fairness, accountability, and transparency, pages 79-88.
Carl C Bell. 1994. Dsm-iv: diagnostic and statistical manual of mental disorders. Jama, 272(10):828-8 829.
Kyungmi Chung, Hee Young Cho, and Jin Young Park. 2021. A chatbot for perinatal women's 's and partners' obstetric and mental health care: Development and usability evaluation study. JMIR Medical Informat- ics, x1)xc186077
Matthew Russell Bennion, Gillian E Hardy, Roger K Moore. Stephen Kellett, and Abigail Millings. 2020. Usability, acceptability, and effectiveness of web- based conversational agents to facilitate problem solv- ing in older adults: controlled study. Journal of Med- ical Internet Research, 22679444
Christopher Collins, Simone Arbour, Nathan Beals, Shawn Yama, Jennifer Laffier, and Zixin Zhao. 2022. Covid connect: Chat-driven anonymous story- sharing for peer support. In Designing Interactive Systems Conference, pages 301-318.
Yashwardhan Bhangdia, Rashi Bhansali, Ninad Chaud- hari, Dimple Chandnani, and ML Dhore. 2021. Speech emotion recognition and sentiment analysis based therapist bot. In 2021 Third International Con- ference Inventive Research in Computing Applica- tions (ICIRCA). pages 96100. IEEE
Patrick Corrigan and Alicia Matthews. 2003. Stigma and disclosure: Implications for coming out of the closet. Journal of mental health, 12(3):235-244 248.
Samuel Rhys Cox and Wei Tsang Ooi. 2022. Does chat- bot language formality affect users' self-disclosure? In Proceedings of the 4th Conference Conversa- tional User Interfaces, pages 1-13.
Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. 2017. Rasa: Open source language understanding and dialogue management. arXiv preprint 1712::1712205181
Reuben Crasto, Lance Dias, Dominic Miranda, and Deepali Kayande. 2021. Carebot: A mental health chatbot. In 2021 2nd International Conference for Emerging Technology (INCET), pages 1-5. IEEE.
Kyle Boyd, Courtney Polts. Raymond Bond, Maurice Mulvenna, Thomas Broderick, Con Burns, Andrea Bickerdike, Mike Mctear, Catrine Kostenius, Alex Vakaloudis, et al. 2022. Usability testing and trust analysis of mental health and wellbeing chatbot. In Proceedings of the 33rd European Conference on Cognitive Ergonomics, pages 1-8.
Kate Daley, Ines Hungerbuehler, Kate Cavanagh, Heloísa Garcia Claro. Paul Alan Swinton, and Michael Kapps. 2020. Preliminary evaluation of the engagement and effectiveness of mental health chat- bot. Frontiers in digital health, 2:576361.
John Brooke et al. 1996. Sus-a quick and dirty usability scale. Usability evaluation in indusiry. 189(194):4- 7.
Nadja Damij and Suman Bhattacharya. 2022. The role of ai chatbots in mental health related public ser- vices in a (post) pandemic world: review and fu- ture research agenda. In 2022 IEEE Technology and Engineering Management Conference (TEMSCON EUROPE), pages 152-159, IEEE
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 18:78771
Alison Darcy, Jade Daniels, David Salinger, Paul Wicks, and Athena Robinson. 2021. Evidence of human- level bonds established with digital conversational agent: cross-sectional, retrospective observational study. JMIR Formative Research, 577)
Franziska Burger, Mark A Neerinex, and Willem-Paul Brinkman. 2022. Using a conversational agent for thought recording as a cognitive therapy task: Fea- sibility, content, and feedback. Frontiers in Digital Health, page 125.
Avisha Das, Salih Selek, Alia R Warner, Xu Zuo. Yan Hu, Vipina Kuttichi Keloth, Jianfu Li, W Jim Zheng, and Hua Xu. 2022a. Conversational bots for psy- chotherapy: A study of generative transformer mod- els using domain-spectt of the 2/ Workshop on Biomedical Language Pro- cessing, pages 285-297.
Cass, 2023. Cass: The leading mental health ai assistant. Cass website.
dialogues. In Proceedings
William W Chan, Ellen E Fitzsimmons-Crafi, Arielle C Smith, Marre-Laure Firebaugh. Lauren A Fowler, Bianca DePietro. Naira Topooco, Denise Wilfley, C Barr Taylor, and Nicholas C Jacobson. 2022. The challenges in designing prevention chatbot for cat- ing disorders: observational study. JMIR Formative Research, 6( 2800033.
Avisha Das. Salih Selek. Alia R. Warner, Xu Zuo, Yan Hu, Vipina Kuttichi Keloth, Jianfu Li, W. Jim Zheng, and Hua Xu. 2022b. Conversational bots for psy- chotherapy: A study of generative transformer mod- els using domain-specine dialogues. In Proceedings

---

the 21st Workshop on Biomedical Language Pro- cessing, pages 285- 295, Dublin, Ireland. Association for Computational I Linguistics.
Gilly Dosovitsky, Blanca S Pineda, Nicholas C. Jacob- son, Cyrus Chang. Eduardo L Bunge, et al. 2020. Artificial intelligence chatbot for depression: de- scriptive study of usage. JMIR Formative Research, 4(1 x)ee6665
Mauro De Gennaro, Eva G Krumhuber, and Gale Lucas.
2020. Effectiveness of an empathic chatbot in com- bating adverse effects of social exclusion on mood. Frontiers in psychology. page 3061.
Eva Durall and Evangelos Kapros. 2020. Co-design for competency self assessment chatbot and survey in science education. In Learning and Collaboration Technologies. Human and Technology Ecosystems: 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCHI 2020, Copenhagen, Denmark, July 19-24, 2020, Pro- ceedings, Part II 22, pages 13-24. Springer.
Johan Oswin De Nieva, Jose Andres Joaquin, Chaste Bernard Tan, Ruzel Khyvin Marc Te, and Ethel Ong. 2020. Investigating students' use of a mental health chatbot to alleviate academic stress. In 6th International ACM In- no-Cooperation HCI and UX Conference, pages 1-10.
Tessa Eagle, Conrad Blau, Sophie Bales, Noopur Desai, Victor Li. and Steve Whittaker. 2022. "i don't 't know what you mean byi am anxious"". for evaluating conversational agent responses to stan- dardized mental health inputs for anxiety and depres- sion. ACM Transactions on Interactive Intelligent Systems (TiiS), 1-(3):1-23.
Paula Maia de Souza, Isabella da Costa Pires, Vi- vian Genaro Motti, Helena Medeiros Caseli, Jair Bar- bosa Neto, Larissa C Martini, and Vânia Paula de Almeida Neris. 2022. Design recommendations for chatbots to support people with depression. In Proceedings of the 21st Brazilian Symposium on Hu- man Factors in Computing Systems, pages 1-11.
new method
Ahmed Fadhil, Gianluca Schiavo, Yunlong Wang, and Bereket A Yilma. 2018 The effect of emojis when in- teracting with conversational interface assisted health coaching system. In Proceedings of the 12th EAI in- ternational conference on pervasive computing tech- nologies for healthcare, pages 378-383.
Ashlin Deepa, Prathyusha Karlapati, Mrunhaal- hini Reddy Mulagondla. Pavitra Amaranayani, and Anika Pranavi Toram. 2022. An innovative emotion recognition and solution recommendation chatbot. In 2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS), volume I, pages 1100-1105 105. IEEE
Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior ther- apy to young adults with symptoms of depression and anxiety using fully automated conversational agent (woebot): randomized controlled trial. JMIR mental health, 4(2)cc7785.
Orianna Demasi, Yu Li, and Zhou Yu. 2020. multi- persona chatbot for hotline counselor training. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 3623-3636.
Silvia Gabrielli, Silvia Rizzi, Giulia Bassi, Sara Car- bone, Rosa Maimone, Michele Marchesoni, and Ste- fano Forti. 2021. Engagement and effectiveness of a healthy--opingg intervention via chatbot for univer- sity students during the covid-19 pandemic: mixed methods orfo--of-coneept study. JMIR mHealth and uHealth, (57x227.
Kerstin Denecke, Sayan Vaaheesan, and Aaganya Arul- nathan. 2020. A mental health chatbot for regu- lating emotions (sermo)-concep and usability test. IEEE Transactions on Emerging Topics in Comput- ing, 170-1177-
Pranto Dev. Sameeha Haque, Asmita Noor, Abir Alam Srabon, Mashruk Mohammed Wasik. Sumaiya Mim, Shadman Bin Sharife, Fariha Rahman, Syeda Rifa Syara, Shadab Iqbal, et al. 2022. A comparative study of chatbot catered toward mental health. In 2022 IEEE International Conference on Artificial In- telligence in Engineering and Technology (IICAIET), pages 1-6. IEEE.
Hannah Gaffney, Warren Mansell, and Sara Tai. 2019. Conversational agents in the treatment of mental health problems: mixed-method systematic review. 14166.
JMIR mental health,
Hannah Gaffney, Warren Mansell, and Sara Tai. 2020. Agents of change: Understanding the therapeutic pro- cesses associated with the helpfulness of therapy for mental health problems with relational agent mylo. Digital Health, 6:205520762091 1580.
Varshaa Dhanasekar, Yenugu Preethi, S Vishali, Praveen Joe IR, et al. 2021. A chatbot to promote stu- dents mental health through emotion recognition. In 2021 Third International Conference on Inventive Re- search Computing Applications (ICIRCA), pages 1412-1416. IEBE.
Sahil Garg, Irina Rish, Guillermo Cecchi, Palash Goyal, Sarik Ghazarian, Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. 2020. Modeling dialogues with hash- code representations: A nomparametric approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3970-3979.
Gilly Dosowitsky, Erick Kim, and Eduardo I. Bunge. 2021. Psychometric properties of a chatbot version of the phq-9 with adults and older adults. Frontiers in Digital Health, 3:645805.
Asma Ghandeharioun, Daniel McDuff, Mary Czerwin- ski, and Kael Rowan. 2019a. Emma: An emotion- aware wellbeing chatbot. In 2019 8th International

---

Conference on Affective Computing and Intelligent Interaction (ACII), pages 1-7. LEEE.
Abid Hassan, MD Ali, Rifat Ahammed, Sami Bourouis, Mohammad Monirujjaman Khan, et al. 2021. Devel- opment of nip-integrated intelligent web system for e-mental health. Computational and Mathematical Methods in Medicine, 2021.
Asma Ghandcharioun, Damiel McDuff, Mary Czerwin- ski, and Kael Rowan. 2019b. Towards understanding emotional intelligence for behavior change chatbots. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 8-14. IEEE
Yuhao He, Li Yang, Xiaokun Zhu, Bin Wu, Shuo Zhang. Chunlian Qian, and Tian Tian. 2022. Mental health chatbot for young adults with depressive symptoms during the covid-19 pandemic: Single-blind. three- arm randomized controlled trial. Journal of Medical Internet Research, 24(1 1)e40719.
Raman Goel, Sachin Vashisht, Armaan Dhanda, and Seba Susan. 2021. An empathetic conversational agent with attentional mechanism. In 2021 Interna- tional Conference on Computer Communication and Informatics (ICCCI), pages 1 1-4. IEEE.
Chester Holt-Quick and Jim Warren. 2021. Establish- ing dialog agent policy using deep reinforeement learning in the psychotherapy domain. In 2021 Aus- tralasian Computer Science Week Multiconferrnce, pages 1-9.
Google. 2021. Google dialogflow: A conversational ai platform. Google Cloud Platform Documentation.
Yenushka Goonesekera and Liesje Donkin. 2022. cognitive behavioral therapy chatbot (otis) for health anxicty management: Mixed methods pilot study. JMIR Formative Rexearch, 6( (10):737777
David M. Howcroft, Anya Belz. Miruna-Adriana
Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computational Lin- guistics.
Eric P Green, Yihuan Lai, Nicholas Pearson, Sathyamath Rajasekharan, Michiel Rauws, Angela Joerin, Edith Kwobah, Christine Musyimi, Rachel M Jones, Chaya Bhat, et al. 2020. Expanding access to perinatal depression treatment in kenya through automated psychological support: Development and usability study, JMIR Formative Research, 4(10):e1795...
Xiao Huang. 2022. Ideal construction chatbot based on intelligent depression detection techniques. In 2022 IEEF International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), pages 511-515. IEEE.
Eric P Green, Nicholas Pearson, Sathyanath Rajasekha- ran, Michiel Rauws, Angela Joerin, Edith Kwobah, Christine Musyimi, Chaya Bhat, Rachel M Jones, and Yihuan Lai. 2019. Expanding access to depres- sion treatment in kenya through automated psycho- logical support: protocol for single-case experi- mental design pilot study. JMIR research protocoss, 8(4):el 180.
Ines Hungerbuehler, Kate Daley, Kate Cavanagh, Heloísa Gareia Claro, and Michael Kapps. 2021, Chatbot-based assessment of employees' mental health: Design process and pilot implementation. JMIR Formative Research, 5(4):22
Stephanie Greer, Danielle Ramo, Yin-Juei Chang, Michael Fu, Judith Moskowitz, Jana Haritatos. et al. 2019. Use of the chatbot "vivibot" to deliver posi- tive psychology skills and promote well-being among young people after cancer treatment: randomized con- trolled feasibility trial. JMIR mHlaalth and uHealth, 1518).
Becky Inkster, Shubhankar Sarda, Vinod Subramanian, et al. 2018. An empathy-driven, conversational artifi- cial intelligence agent (wysa) for digital mental well- being: real-world data evaluation mixed mexhhds study. JMIR mHealth and uHealth, 6(11)=12200
Christine Grové. 2021. Co-developing mental health and wellbeing chatbot with and for young people. Frontiers in psychiatry. 11:606041.
Etsuko Ishii, Genta Indra Winata, Samuel Cahyawi- jaya, Divesh Lala, Tatsuya Kawahara, and Pascale Fung. 2021. Erica: an empathetic android com- panion for covid-19 19 quarantine. arXiv preprint arXiv:2106.0333
Megha Gupta, Tanya Malik, Chaitali Sinha, et al. 2022. Delivery of a mental health intervention for chronic app
pain through artificial intelligence-enablle (wysa): Protocol for a prospective pilot study. JMIR Research Protocols, 11(3)x33610.
Shashank Jaiswal, Michel Valstar, Keerthy Kusumam. and Chris Greenhalgh. 2019. Virtual human question- naire for analysis of depression, anxiety and person- ality. In Proceedings of the 19th acm international conference on intelligent virtual agents, pages 81-87,
Hee Jeong Han, Sanjana Mendu, Beth K Jaworski, Ja- son E Owen, and Saced Abdullah. 2021. Ptsdialogue: Designing conversational agent to support individ- uals with post- traumatic stress disorder. In Adjunet Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Comput- ing and Proceedings of the 2021 ACM International Symposium on Wearable Computers, pages 198-203.
Qiaolei Jiang. Yadi Zhang, and Wenjing Pian. 2022. Chatbot as an emergency exist: Mediated empathy for resilience via human-ai interaction during the covid-19 19 pandemic. Information Processing & Man- agement, 59(6):103774

---

Masamune Kawasaki, Naomi Yamashita, Yi-Chich Lee, and Kayoko Nohara. 2020. Assessing users' mental status from their journaling behavior through chat- bots. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pages 1-8.
Brooke Linden, Linna Tam-Seto, and Heather Stuart. 2020. Adherence of the# here4u app- poiimmittrry ver- sion to criteria for the development of rigorous mental health apps. JMIR Formative Research, 4(6);
Hao Liu, Huaming Peng. Xingyu Song, Chenzi Xu, and Meng Zhang. 2022. Using ai chatbots to provide self-help depression interventions for university stu- dents: A randomized trial of effectiveness. Internet Interventions, 27:100495.
Junhan Kim, Jana Muhic, Lionel Peter Robert, and Sun Young Park. 2022. Designing chatbots with black americans with chronic conditions: Overcom- ing challenges against covid-19. In Proceedings of the 2022 CHI Conference on Human Factors in Com- puting Systems, pages 1-17.
Nicola Ludin, Chester Holt-Quick, Sarah Hopkins, Karolina Stasiak, Sarah Hetrick, Jim Warren, and Tania Cargo. 2022. A chatbot to support young peo- ple during the covid- 19 pandemic in new zealand: Evaluation of the real-world rollout of an open trial. Journal of Medical Internet Research, 24(1 :):387433
Maria Carolina Klos, Milagros Escoredo, Angela Jo- erin, Viviana Noemi Lemos. Michiel Rauws, and Ed- uardo L Bunge. 2021. Artificial intelligence- based chatbot for anxiety and depression in university stu- dents: pilot randomized controlled trial. JMTR for- mative research, 5(8):2067
Martin Luerssen and Tim Hawke. 2018. Virtual agents as a service: Applications in healthcare. In Proceedings of the 18th International Conference Intelligent Virtual Agents, pages 107-112.
Robert Kohn, Shekhar Saxena, Itzhak Levav, and
Benedetto Saraceno. 2004. The ueatment gap in
mental health care. Bulletin of the World health Or-
ganization, 82(11)885-886
Kien Hoa Ly, Ann-Marie Ly, and Gerhard Andersson. 2017. A fully automated conversational agent for promoting mental well-being: A pilot ret using mixed methods. Internet interventions, 10:39-46.
Theodora Koulouri, Robert D Macredie, and David Olakitan. 2022. Chatbots to support young adults'
mental health: An exploratory study of acceptability. ACM Transactions on Interactive Intelligent Systems (TiiS), 12(2):1-33
Wookjac Maeng and Joonhwan Lee. 2022. Designing and evaluating chatbot for survivors of image-based sexual abuse. In Proceedings of the 2022 CIII Con- ference Human Factors in Computing Systems, pages 1-21.
Kurt Kroenke, Robert I. Spitzer, and Janet BW Williams. 2001. The phq-9: validity of a brief depression sever- ity measure. Journal of general internal medicine, 16(9):606-613.
Raju Maharjan, Kevin Doherty, Darius Adam Rohani, Per Bækgaard, and Jakob E Bardram. 2022a. Ex- periences of speech-enabled conversational agent for the self-report of well-being among people living with affective disorders: An in-lhe-wild study. ACM Transactions on Interactive Intelligent Systems (TiiS), 12(2):1-29.
Liliana Laranjo, Adam G Dunn, Huong Ly Tong. Ah- met Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie YS Lau, et al. 2018. Conversational agents in health- care: a systematic review. Journal of the American Medical Informatics Association, 1000:1248-12555
Minha Lee, Sander Ackermans, Nena Van As, Hanwen Chang, Enzo Lucas, and Wijnand IJsselsteijn. 2019, Caring for vincent: a chatbot for self-compassion. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Raju Maharjan, Darius A Rohani, Kevin Doherty, Per Bækgaard, and Jakob E Bardram. 2022b. What is the difference? investigating the self-report of well- being via conversational agent and web app. IEEE Pervasive Computing. 21 (1():60066.
Yi-Chieh Lee, Naomi Yamashita, and Yun Huang. 2020a. Designing a chatbot as mediator for pro- moting deep self-disclosure to real mental health professional. Proceedings of the ACM on Human- Computer Interaction, (CCSCWI):1221
Raju Maharjan, Darius Adam Rohani, Per Bæeggard, Jakob Bardram, and Kevin Doherty. 2021. Can we talk? design implications for the questionnare-drrven self-report of health and wellbeing via conversational agent. In Proceedings of the 3rd Conference on Con- versational User Interfaces, pages 1-11.
Yi-Chieh Lee, Naomi Yamashita. Yun Huang, and Wai Fu. 2020b. "i hear you, feel you": encouraging deep self-disclosur through a chatbot. In Proceedings of the 2020 CHi conference on human factors in computing systems, pages 1-12.
Audrey Mariamo, Caroline Elizabeth Temcheff, Pierre- Majorique Léger. Sylvain Senecal, Marianne Alexan- dra Lau, al. 2021. Emotional reactions and like- lihood of response to questions designed for men- tal health chatbot among adolescents: Experimental study. JMIR human factors, 8(1):22443.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Laura Martinengo, Ahmad Ishqi Jabir, Westin Wei Tin Goh, Nicholas Yong Wai Lo, Moon-Ho Ringo Ho,

---

Tobias Kowatsch, Rifat Atun, Susan Michie, and Lo- rainne Tudor Car. 2022. Conversational agents in health care: Scoping review of their behavior change techniques and underpinning theory. Journal of Med- ical Internet Research, 24( 10):c39243.
home environments. In Proceedings of the 1st ACM International Workshop on Intelligent Acoustic Sys- tems and Applications, pages 3 31-36.
Jasmine M Noble, Ali Zamani, Mohamad Gharaat,
Dylan Merrick, Nathanial Maeda, Alex Lambe Fos- ter, Isabella Nikolaidis, Rachel Goud, Eleni Stroulia, Vincent 10 Agyapong, et al. 2022. Developing, im- plementing. and evaluating an artificial intelligence- guided mental health resource navigation chatbot for health care workers and their families during and following the covid-19 pandemie: Protocol for cross-sectional study. JMIR Research Protocols, I(7):c33717.
Matthew Louis Mauriello, Nantanick Tantivasadakarn, Marco Antonio Mora-Mendoza, Emmanuel Thierry Lincoln, Grace Hon, Parsa Nowruzi, Dorien Simon, Luke Hansen, Nathaniel H Goenawan, Joshua Kim, al. 2021. A suite of mobile conversational agents for daily stress management (popbots); Mixed meth- ods exploratory study. JMIR formative research, 5(9):2252
Saha Meheli, Chaitali Sinha, and Madhura Kadaba. 2022. Understanding people with chronic pain who cognitive behavioral seerapy-baeed artificial intelligence mental health app (wysa): Mixed meth- ods retrospective observational study. JMIR Human Factors, 9())e33677.
Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. 2021. Chatbots applications in education: A system- atic review. Computers and Education: Artificial Intelligence, 2:100033.
Moisés Pacheco- Lorenzo, Somia M Valladares- Rodríguez, Luis E Anido-Rifón, and Manuel J FermándezzIglesiis, 2021. Smart conversational agents for the detection of neuropsychiatric disor- ders: A systematic review. Journal of Biomedical Informatics, 113:103632.
David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, Prisma Group, et 2010. Pre- ferred reporting items for systematic reviews and meta-anall vetaa-anayysss the prisma statement. International journal of surgery, 8(5):36-341.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Joao Luis Zeni Montenegro, Cristiano André da Costa, and Rodrigo da Rosa Righi. 2019, Survey of con- versational agents in health. Expert Systems with Applications, 129:56-67. 67.
Robert R Morris, Kareem Kouddous, Rohan Kshir- sagar, and Stephen M Schueller. 2018 Towards an artificially empathic conversational agent for men- tal health applications: system design and user per- ceptions. Journal of medical Internet research, 1016),
Gain Park, Jiyun Chung, and Seyoung Lee. 2022. Effect of ai chatbot emotional disclosure on user satisfaction and reuse intention for mental health counseling: a serial mediation model. Current Psychology, pages 1-11.
Thomas Munder, Fabian Wilmers, Rainer Leonhart, Hans Wolfgang Linster, and Jürgen Barth. 2010. Working alliance inventory-short revised (wai-sr): psychometric properties in outpatients and inpatients. Clinical Psychology & Psychotherapy: An Interna- lional Journal Theory & Practice, (7(3)2311233)
Hyanghee Park and Joonhwan Lee. 2021. Designing conversational agent for sexual assault survivors: defining burden of self-disclosure and envisioning survivor-centered solutions. In Proceedings of the 2021 CHI Conference on Human Factors in Comput- ing Systems, pages 1-17.
Sergazy Narynov. Zhandos Zhumanov, Aidana Gu- mar, Mariyam Khassanova, and Batyrkhan Omarov. 2021a. Chatbots and conversational agents in mental health: A literature review. In 2021 21st Interna- tional Conference on Control, Automation and Sys-
SoHyun Park, Jeewon Choi, Sungwoo Lee, Changhoon Oh, Changdai Kim, Soohyun La, Joonhwan Lee, and Bongwon Suh. 2019. Designing a chatbot for a brief motivational interview on stress management: Qual- itative case study. Journal of medical Internet re- search, 2224)
tems (ICCAS), pages 353-358, IEEE.
Sergazy Narynov. Zhandos Zhumanov, Aidana Gu- mar, Mariyam Khassanova, and Batyrkhan Omarov. 2021b. Development of chatbot psychologist ap- plying natural language understanding techniques. In 2021 21st International Conference on Control, Automation and Systems (ICCAS), pages 636-641. IBEE.
SoHyun Park, Anja Thieme, Jeongyun Han, Sungwoo Lee, Wonjong Rhee, and Bongwon Suh. 2021. "i wrote as if were telling story to someone knew.": Designing chatbot interactions for expressive writing in mental health. In Designing Interactive Systems Conference 2021, pages 926-941.
Pritika Parmar, Jina Ryu, Shivani Pandya, João Sedoc, and Smisha Agarwal. 2022. Health-focused conver- sational agents in person-centered care: a review of apps. NPJ digital medicine, 5(1):11.
Jingping Nie, Hanya Shao, Minghui Zhao, Stephen Xia, Matthias Preindl, and Xiaofan Jiang. 2022. Conver- sational ai therapist for daily function screening in

---

James Pennebaker, Roger Booth, Ryan Boyd, and Martha Francis. 2015. Linguistic inquiry and word count: Liwe2015.
for Health (H3): Ktaman-Habite Multimodal Inter- action for Promoting Health and Well-Being in the Internet of Things Era, pages 1-6.
Courtney Potts, Raymond Bond, Maurice D Mulvenna, Edel Ennis, Andrea Bickerdike, Edward K Coughlan, Thomas Broderick, Con Burns, Michael F McTear, Lauri Kuosmanen, et al. 2021. Insights and lessons learned from trialligg a mental health chatbot in the wild. In 2021 IEEE'S Symposium on Computers and Communications (ISCC), pages 1-6. IEEE
Prabod Rathnayaka, Nishan Mills, Donna Burnett, Daswin De Silva, Damminda Alahakoon, and Richard Gray. 2022. A mental health chatbot with cognitive skills for personalised behavioural acti- vation and remote health monitoring. Sensors, 22(10):3653.
S Zahra Razavi, Lenhart K Schubert, Kimberly van Orden, Mohammad Rafayet Ali, Benjamin Kane, and Ehsan Hoque. 2022. Discourse behavior of older adults interacting with dialogue agent competent in multiple topics. ACM Transactions on Interactive Intelligent Systems (TiS), 12(2):1-21.
Judith Prochaska, Erin A Vogel, Amy Chieng, Michael Baiocchi, Dale Dagar Maglalang, Sarah Pajarito, Kenneth R Weingardt, Alison Darcy, and Athena Robinson, 202 randomized controlled trial of a therapeutic relational agent for reducing substance misuse during the co pandemie. Drug and Alcohol Dependence, 227:108986.
Hyeyoung Ryu, Soyeon Kim, Dain Kim, Sooan Han, Keeheon Lee, and Younah Kang. 2020. Simple and steady interactions win the healthy mentality: de- signing chatbot service for the elderly. Proceed- ings of the ACM on Hhman-Comppun Interaction, 4(CSCW2):1-25.
Judith J Prochaska, Erin A Vogel. Amy Chieng, Matthew Kendra, Michael Baiocchi, Sarah Pajarito, and Athena Robinson. 2021b. therapeutic rela- tional agent for reducing problematic substance use (woebot): development and usability study. Journal of medical Internet research. 23(3):
Zeineb Safi, Alaa Abd-AIrazaq. Mohamed Khalifa, Mowafa Househ, al. 2020. Technical aspects of developing chatbots for medical applications: scop- ing review. Journal of medical Internet research. 12(12):11
Simon Provoost, Annet Kleiboer, José Ornelas, Tibor Bosse, Jeroen Ruwaard, Artur Rocha, Pim Cuijpers. and Heleen Riper. 2020. Improving adherence an online intervention for low mood with virtual coach: study protocol of pilot randomized controlled trial. Trials, 21:1-12.
Tulika Saha, Saichethan Reddy, Anindya Das, Sriparna Saha, and Pushpak Bhattacharyya. 2022a. A shoul- der to cry on: Towards motivational virtual assis- tant for assuaging mental agony. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 2436-2449. Seattle, United States. Association for Computational Linguistics.
Juan Quiroz, Tristan Bongolan, and Kiran Ijaz. 2020. Alexa depression and anxiety self-tests: a prelimi- nary analysis of user experience and trust. In Adjanct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Comput- ing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, pages 494-496.
Tulika Saha, Saichethan Reddy, Anindya Das, Sriparna Saha, and Pushpak Bhattacharyya. 2022b. A shoul- der to cry on: Towards motivational virtual assistant for assuaging mental agony. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 2436-2449
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188::
Rifat Rahman, Md Rishadur Rahman, Nafis Irtiza Tripto, Mohammed Eunus Ali. Sajid Hasan Apon, and Rifat Shahriyar. 2021. Adolescentbot: Understanding op- portunities for chathots in combating adolescent sex- ual and reproductive health problems bangladesh. In Proceedings of the 2021 CHI Conference on Hu- mun Factors in Computing Systems, pages 1-15.
Katharine Sanderson. 2023. Gpt-4 is here: what scien- tists think. Nature, 6155(754))773.
Anita Schick, Jasper Feine, Stefan Morana, Alexan- der Maedche, and Ulrich Reininghaus. 2022. Va- lidity of chatbot use for mental health assessment: Experimental study. JMIR mHealth and uHealth. :0880)228088
Beatrice Rammstedt, Christoph Kemper, Mira Céline Klein, Constanze Beierlein, and Anastassiya Koval- eva. 2013. A short scale for assessing the big five dimensions of personality: 10 item big five inventory 10) methods, data, analyses, 7(2):17.
Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Ar- turo Toledo, Ann Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M Linehan. 2018. Pocket skills: A conversational mobile web app to support dialec- tical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-15.
Neelesh Rastogi, Fazel Keshtkar, and Md Suruz Miah. 2018. A multi-modal human robot interaction frame- work based on cognitive behavioral therapy model. In Proceedings of the Workshop on Human-Haabtat

---

RX Schwartz, Apama Ramanan, Disha Patel, Annabel Lynch, Sonia Bace, and Laura Barnes. 2022. Dara: Development of a chathot support system for an anxiety reduction digital intervention. In 2022 Sys- tems and Information Engineering Design Sympo- sium (SIEDS), pages 139-144, IEEE
Lu Sun, Yuhan Liu, Grace Joseph, Zhou Yu, Haiyi Zhu, and Steven Dow. 2022. Comparing experts and novices for ai data work: Insights on allocating hu- man intelligence to design a conversational agent. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 10, pages 195-206.
Ashish Shenoy, Sravan Bodapati, and Katrin Kirchhoff. 2021. ASR adaptation for E-commerce chatbots us- ing cross-utterance context and multi-task language modeling. In Proceedings of the 4th Workshop on e-Commerce and NLP, pages 18-25, Online. Associ- ation for Computational Linguisties.
Colm Sweeney, Courtney Potts, Edel Ennis, Raymond Bond, Maurice D Mulvenna, Siobhan O' neill, Martin Malcolm, Lauri Kuosmanen, Catrine Kostenius, Alex Vakaloudis, et al. 2021. Can chatbots help support a person's 's mental health? perceptions and views from mental healthcare professionals and experts. ACM Transactions on Computing for Heulthcare, 2(3):1- 15.
Ji Youn Shin and Jina Huh-Yoo. 2020. Designing ev- eryday conversational agents for managing health and wellness: A study of alexa skills reviews. In Pro- ceedings the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare, pages 50-61.
Christian Winther Topp, Spren Dinesen Østergaard, Su- san Søndergaard, and Per Bech. 2015. The who-5 well-being index: systematic review of the litera- ture. Psychotherapy and psychosomatics, 84(3):167- 176.
Dominic Ethan Sia, Marco Jalen Yu, Justine Leo Daliva, Jaycee Montenegro, and Ethel Ong. 2021. Investigat- ing the acceptability and perceived effectiveness of a chatbot in helping students assess their well-being In Asian CHI Symposium 2021, pages 34 40.
Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous. 2019. Chathots and conversa- tional agents in mental health: a review of the psychi- atric landscape. The Canadian Journal of Psychiatry, 64(7):456-466
Sayed Abu Noman Siddik, BM Arifuzzaman, and Abul Kalam, 2022. Psyche conversa-a deep learning based chatbot framework to detect mental health state. In 2022 10th International Conference Information and Communication Technology (ICoICT), pages 146-151. IEEF
Mina Valizadeh and Natalie Parde. 2022. The ai doctor is in: A survey of task-orienter dialogue systems for healthcare applications. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: LLnng Papers), pages 6638- 6660.
Candace L Sidner, Timothy Bickmore, Bahador Nooraie, Charles Rich, Lazlo Ring. Mahni Shay- ganfar, and Laura Vardoulakis. 2018. Creating new technologies for companionable agents to support iso- lated older adults. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(3):1-27.
Stefano Valtolina and Liliana Hu. 2021. Charlie: A chatbot to improve the eiderly quality of life and to make them more active to fight their sense of loneli- ness. In Clfialy 2021: 14th Biannual Conference of the Italian SIGCHI Chapter, pages 1-5.
Chaitali Sinha, Abby L Cheng, and Madhura Kadaba. 2022. Adherence and engagement with a cogni- tive behavioral therapy-based conversational agent (wysa for chronic pain) among adults with chronic pain: Survival analysis. JMIR Formative Research, 6(5):e3330..
Hashmaryne C van Cuylenburg and TNDS Ginige. 2021. Emotion guru: A smart emotion tracking application with ai conversational agent for exploring and pre- venting depression. In 2021 International Confer- ence on UK-Chine Emerging Technologies (UCET), pages 1-6. IEEE.
Robert L. Spitzer, Kurt Kroenke, Janet BW Williams, and Bernd Löwe. 2006. A brief measure for assessing generalized anxiety disorder: the gad-7. 7. Archives of internal medicine, (66(10):1092-199
Jelte van Waterschoot, Iris Hendrickx, Mohammed Arif Khan, Esther Klabbers, Marcel de Korte, Helmer Strik, Catia Cucchiarini, and Mariët Theune. 2020. Bliss: An agent for collecting spoken dialogue data about health and well-being. In Proceedings of the 12th language resources and evaluation conference. pages 449-458.
Zhaoyuan Su, Mayara Costa Figuciredo, Jucun Jo, Kai Zheng, and Yunan Chen. 2020. Analyzing descrip- tion, user understanding and expectations of in mo- bile health applications. In AMIA Annual Symposium Proceedings, volume 2020, page 1170. American Medical Informatics Association.
Shinichiro Suganuma, Daisuke Sakamoto, Haruhiko Shimoyama, et al. 2018 An embodied conversa- tional agent for unguided internet-hased cognitive behavior therapy in preventative mental health: fea- sibility and acceptability pilot trial. JMIR mental health, 5(3)e 10454.
Jinping Wang. Hyun Yang, Ruosi Shao, Saeed Abdul- lah, and S Shyam Sundar. 2020a. Alexa as coach: Leveraging smart speakers to build social agents that reduce public speaking anxiety. In Proceedings of the 2020 CHI conference on human factors computing systems, pages 1-13.

---

Liuping Wang, Dakuo Wang, Feng Tian, Zhenhui Peng. Xiangmin Fan, Zhan Zhang, Mo Yu, Xiaojuan Ma. and Hongan Wang. 2021. Cass: Towards building a social-support chatbot for online health commu- nity. Proceedings of the ACM on Computerr Interaction, ([CSCW1)11311..
interested in mental health conversational agents. In Table 7, we show the distribution of venues under each database for the papers that are selected for the final list.
B Full Table Explanation
Ruyi Wang, Jiankun Wang, Yuan Liao, and Jinyu Wang. 2020b. Supervised machine learning chat- bots for perinatal mental healtheare. In 2020 Inter- national Conference on Intelligent Computing and Human-Com Computer Interaction (ICHCI), pages 378- 383. IEEE
We show our final list of model/experiment papers in Table 8, Table and Table 10. Due to the limited size of the paper, some columns ("background") are removed and long values are truncated. The full table is available on our GitHub.
Joseph Weizenbaum. 1966. Eliza- a computer program for study natural language communication be- tween man and machine. Communications of the ACM, 91):36--55
For an easier understanding of our full table, we briefly introduce each feature we extracted below.
Paper: The citation of the selected paper.
Lee Wilson and Mariana Marasoiu. 2022. The develop- ment and use of chatbots in public health: Scoping review. JMIR human factors, 94)ee35882.
Database: The source of the paper.
Paper Type: The type of the paper. We here only show model or experiment papers.
Xinxin Yan and Ndapa Nakashole. 2021. A grounded well-being conversational agent with multiple inter- action modes: Preliminary results. arXiv preprint arXiv:2111.1408.
Language: Target language used in this paper. Mental Health Category: Target mental health category in this paper.
Target Group: Target group of this paper. Could be patients, caregivers, or clinicians. Target Demographic: Target demographic of this paper. If it not specified can be used by anyone, mark it as General.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre- training for conversational response generation, arXiv preprint arXiv:191..00536.
Yonghan Zhu, Rui Wang, and Chengyan Pu. 2022. "i am chathot, your virtual mental health adviser." what drives citizens' satisfaction and continuance inten- tion toward mental health chatbots during the covid- 19 pandemic? an empirical study in china. Digital Health, 8: 8:20552076221990003..
Chatbot Name: The name of the chatbot model used this paper.
Chatbot Type: Type of the mental health CA. Could be QA, open domain, or task-oriented. Model Technique: Type of technique used to build the model. Could be rule-based, retrieval-based, or generative.
A Venues of Selected Papers
Off the Shelf: Information about the usage of off-the-shelf models in the system. We limit Off-the-shelf models to pre-trained models or
In this paper, we searched all venues indexed under
5 databases to cover most of the venues that are
AAAI
ACL
ACM
IEER
Pubied
Venue
Count
Venue
Count
Venue
Count
Venue
Count
Venue
Count
HCOMM
EMNNP
CIII
ICIRCA
IMIR Form Res
AAAI
SIGDIAL
ACM-TIS
ACII
2
Med Internet Res
ICJCC
Digit Health
NAACL
ACM IICI
UCET
I
IMIR Mheulth Uhealth
NLP4PE
UbiComp
ICCCI
Res Protoc
LREC
CUI
ICHCI
Digit Health
PervasiveHealth
ICACOS
IMIR Ment Health
Cilialy
ISCC
IMIR Hum Factors
ACSW
IEEB Trans. Emerg
Interner Interv
Curr Psychol
Asian CHI
IEEE Pervasive Comput.
Comput Math Methods Med
ICCAS
Process Manag
CHIoXID
INCET
Fronr Paychol
ACM HEALTH
Trials Front Psychiatry
IASA
ROCR
Drig Alcohol Depend
Sensors (Busel)
JMIR Med Inform
Table 7: Venues in each database that have at least one paper in our final list and the corresponding number of model/experiment papers.

---

applications. Could be yes (directly used), used as part (off-the-shelf model part of the pipeline), or finetuned.
Outsourced Model Name: The name of the off-the-shelf model, if any.
Training Data: The name source of the training data, if any.
Interface: Type of input the model takes. Could be text, voice, visual, or button.
Embodiment: Embodiment of the model.
Could be physical or visual.
The platform the model run on. Could be Web, Mobile, PC, or other devices. Public Access: If the availability of the model is disclosed in the paper. Could be fully open (parameter level) API (able to use).
Platform:
Study Design: Type of user study performed in the paper. Could be RCT (Randomized Controlled Trial), user study (ask participants to use and evaluate), or comparative analy- sis (divide users with different conditions and compare the results).
Recruitment: How participants are recruited.
Sample Size: Size of the participants.
Duration: Duration of the user study,
Automatic Evaluation: List of automatic eval- uation metries used in this paper.
Human Evaluation: List of parame-
ters/metrics derived from Human Evaluation used in this paper.
Statistical Test: List of statistical tests used for measuring significance in this paper.
Ethics: Whether the paper mentioned ethical consideration. Could be IRB (Institutional Review Board), or yes (ethical consideration is mentioned in the paper).

---

Fully Open
Open
Pubk
APE
AP
Al
APg
APe
AP
AP
AP1
bdoblle Sian Speckerr
Mobile Smart Spealerr
Spenter
Mobill Smar Sponker
Mobile PC Mobbiee Other Devices
Mobill Smar Speater
Speake boobile PC
Mobile
bobblee PC
Mobile, PC
S Speaker
Mobile. PC
Mobile, Web
S Spooter
Vel Kotoor
Mobile
Mobile
Mobile
Modile
adobile
Mobile
Mobile
Modile
Mobile
Mobile
Mocelle
Mobiicc
Mobile
Mikile
Table 8: All method/exxpriment papers the final list of this survey. This table only shows general and appearance features.
Wib
Web
Weh
PC
Web
Wih
Web
Pky yycc
Vinual
Vinual
Payscca
J. Phyyyia
Payscua
Piynical
Vintel
Virual
Vinual
Physical
Vinial
#tysscal
Phyyscal
Paysical
Mosso Vini
Maine Visua
Test. Vince
Teu. Miree
Zext. Voice
Tess Miiie
Texi
Jour
Jest
Test
Teut
Jext
Ters
Teu
Teu
Teut
Jexr
Teut
Teur
Next
Teut
Teu
Mosce
Text
Texi
Jou
Teu
Jeu
Teu
Teur
Text
Text
Teu
Jeu
Teu
Mooce
Tour
High High-Sani Sdhoek Siukenns
High School Studeets
Univeriity Studentss
Youg People
Black Black-Asseie Amerion
Older Malls
Older Older-Auuls Mulls
Older Addis Adalls
Adblecan
Yong Yongg Woomm
Women
Adatt AdattWime
Temayss
Goocrall
Genoral
General
Goneral
Ceneral
Genen
Gentrll
Gonnal
Gentral
Gosna
General
Geoenall
General
Geozral
General
Gener
Geosral
Gonzall
Central
General
Womsn
Grosial
Getrral
Genirall
Gentr
Goneral
General
Gener
Central
Goona
Gentral
Gooeral
General
Genonal
General
Stedmns
Genmr
Poiens, Cargateers
Cliniians
Patents
Palkms
Chincias
Paicmn
Pawess
Pakins
Prieens
Palkon
Pritens
Panion
Poienn
Prikas
Patous
Priken
Palkon
Poneon
Paiens
Pakes
Palkos
Palkens
Pankon
Palcin
Patsoss
Prikens
Pations
Poncmn
Pakess
Patims
Parens
Palkms
Puntens
Paikms
Pakkns
Pintes
Palkos
Phllens
Patkmn
Panens
Pakiss
Paikms
Donder, Depession Bipolar Diooder
Deprosioo Ansicy, Rosouiity
Dakesical Buhonier Thrrey
Socal, Reyodacthe Seatth
Petinaad Menta aeatbberee
Mentel Hoalth Calcagrry
Solaiin Sodid Ansooy
Pullc Pullc Spaakng Austey
Depreviion Aniey
Depressonn Anieyy
Depressonn Ansiey
Im Raved Seond
Degrension Amdidy
Aution Spectum
Secal Apant
Shoting Thoma
Affectre Deunder
Cancall
pepression
Sucide
Cameral
General
Guscrll
Genenl
Centrll
Ceneral
Deprescoo,
Geicral
Cenenl
Conid 19
Gentral
Cenuall
Central
Londines
kolazon
Gancall
Central
Depressoon
Genent
Outeall
Geteral
Ganual
General
Central
Cenenl
Goccal
Ceneal
Ansidty
Gestrl
Eaiing
Cenen
Norweginn
Englos
English
Erglish
Figlish
Engilish
Eiglish
Enggiish
English
English
Jndiin
English
Engliish
Eiglish
Danish
lingiish
English
English
Esglish
English
English
English
Enplish
Feglish
Eiglish
English
English
English
Kerean
English
Engglish
English
English
English
English
Eiglish
Korean
English
Korcan
Engiiihh
Chinoor
Kemon
Engilsh
Figlish
Engllish
Engiish
Engllshh
English
Bnglish
English
English
Cloman
English
Engiish
Engiish
Kankh
English
English
Chineerr
Expeutment
Experiment
Experiment
Expeiieent
Expertmett
Expesiment
Experiment
Exyesiment
Experimen
Expelinett
Experment
Expeiientt
Experiment
Expesiment
Expesiment
Expesiment
Expezimen
Expertment
Expertment
Expesiment
Expeciment
Model
Moddel
Modiel
Model
Model
Model
Model
Model
Model
Model
Moald
Model
Mode
Model
Model
Model
Model
Model
Model
Modle
Model
Model
Model
Model
Model
Modial
Model
Model
Model
bddded
Model
Mddee
Modld
Moddel
Mode
Model
Model
Datahhs
AAAI
AAAI
ACT:
ACL
ACT.
ACL
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
ACM
Biba
IREE
IEEE
100B
IFFE
BEE
van lovora and Giagee (2023)
Halt-uask and Waren (2021)
Sun al (0022)
Carg ctal (2200)
téii al (3011)
Demos al (20230)
De al (312bb)
(20220)
Nakalldde (222))
van Nateodtoon (2020)
Fodhil al 2205))
Inooaal al (3319)
Mihomten at (202))
Eagk al (2022)
the trrnnnaggg etall (2021)
Quirce (2122)
Kowaukiera too (2000)
Shin and 1302))
De Niew et (2009)
Lee (2030))
a al (1021)
et (302))
Sahvoller (201)
aaliiiaann (200))
Sidner at (2018)
Lacvenand Heoke Honk (3008)
Ry at (20200
Razavi at al (2022)
Locecal (2000)
Aliet al, (200))
Sia al (3021)
Park and Ler (022))
Wang d. (2020))
Park (002))
(2022)
Wing (3001)
Ralman (312))
Maing and (202))
Chundehatton al (20190
Siilk oi (3023)
Geci at (2021)
Wang (20000)
uhongls al (200))
Daspa al. (3022)
Pont etal at (001)
Deroskc ctal CR020)
Chankehmenn al (201 9h)
Schwotte al (2IZ2)
Mahapan (300220
Sayyor (21210)
et (302)
Mahajjar
Clunetal
et
Gradi
Abhus
Kin

---

Pubk
AR
AR
APE
AP
AP9
AP
AM
APS
Enobted Device
Platorm
Mobile Web
Mobile RC
aoobile PC
Mohiln PC
Mobile
Mobile
Mobile
bdoellee
Mobile
Mobile,
Mobile
aoobile
Mobble
Table 8: All method//epperinee methoodexxprrinen papers the final list of this survey. This table only shows general and appearance features.
Web
Web
Weh
Weh
Weh
Weh
Meb.
Immbetinest
Virmaa
Vintaa
Vinal
Nin
Nan
Test Balton
Burbo
Munoo
Toxu
Tou
Teu
Test
Jext
Tou
Test
Teur
Text
Tour
Teu
Text
Tru
Text
Text
Tex
Teur
Text
Tour
Teu
Test
Text
Women, New Molles
Perauna Wonenn Plstooss
Denograp
Uniosiy Stalents
Miltary Communzy
Addovesee, Young
Universtyy Studtntts
Univesity Statents
Hoald Cuse Worke
Older Adols.
Adbiessn
Aails
Youg Adilis
People
Young Perple
Ocooral
Goonrall
Ceneral
General
Goonaal
General
Geneinll
Genrral
Genaral
General
Genoonll
Employee
Gentr
Gonnall
General
Genenal
Ceneral
General
Oenrr
General
Patlons Cuagaees
Chiiiiass
Palkos
Patens
Palkms
Piilens
Panion
Pakia
Palkos
Pritens
Panios
Poients
Prikss
Pattus
Paikens
Paikens
Pontens
Patess
Pakes
Palkos
Paikens
Pankon
Palons
Prlkes
Punlanic Relate Ausidy
Dlanders
Anborty Expresion FISO
Ausiett Depression, Steess
Calcaurry
Deversion
Subsante Use Diocoter
Suhausce Use DieDDoi Disoler
Deveesson Antey
Depression Andegy
Social Sacial Exiluioo
Depresson
Diprcsion
Depression
Depressonn
Mentel
Cancall
Contral
Ansidty
Gentral
Canural
Goscall
Ceneni
Cottral
Geteal
Geneaal
Canzai
Gentrll
Gatcal
Central
Chinaal
Geatall
Central
Aßia
Stress
Body
Pemugres
Chirese
English
Heginese
Eeglish
English
Eeglash
English
Spants
Chineso
English
Engiis
English
Eiglish
Engilsh
English
Engiish
Genman
Engiish
Esglish
English
Eiglish
Swotissh
Chinsos
English
English
English
English
Engilihh
English
English
English
English
Eiglish
English
Korean
Enginsh
Chinee
Expeuteett
Experiment
Expesiment
Expesiment
Expestment
Experiment
Experiment
Expeimen
Expeinent
Experteet
Expesiment
Experiment
Exyesiment
Experimen
Expeiinet
Expermentt
Expeiment
Experiment
Expestment
Expesiment
Expesiment
Expeziment
Expetiment
Expertmnt
Experiment
Model
Moald
Moddel
Modd
Model
Model
Model
Model
Model
Moale
Model
Model
Model
Pubbled
Puhlied
PubMee
PubhMal
PubMee
PunNted
Pullied
PutMded
PubMad
PuhMdd
PulMaa
Publked
PuhMked
PubMee
PohMde
PubMed
PolMall
Publked
Pubikca
Puttded
PulMdd
PulMal
ruNNed
Punkeed
PubMed
PooMed
PoWMed
Casnesaee and D-okin (1n22)
Hungebrellec at (001))
leager et al (700))
Rennio Suganimi stal. (20U8)
Guller at (2000)
Marimmra (2011)
Plooost (2020)
Linkon ctal. (2000)
Cuppa ct al 2202)
Proheeia eta (2021
Pochooka at (20210))
hung al (2011)
(2122)
Heate atl (1022)
Mahel al. 2202
Posovinlyccta cral (2000)
Dososiall at (2021)
(3120)
Park ot al. 22021
Haacn al (390))
Burger at (0220)
De Gonao etal (3020)
Barhngalaet (209))
al Htrprrick etall (207))
Noble at 2202)
Maunelo al (2001)
Chong al (2011)
Morns (20.8)
Belhur (202))
et al (3022)
Pakk (102))
Lines Kis
Gaboelli

---

Table 9: All method/e exprrmmmeeeeimeetttt papers the final list of this survey, This table only shows technique features, Long values are truncated due to limited space.
(Now) heaps Conversation Data
Depression Thempy Sevsinos, I....
Realistie Hotline Training Con...
(New)) Reddin, Traneripo Of A....
(Now) Medline Datn, MeoiDilog
(Nw) Use Clses Of Profession
Dr. Manhha Linelan DRT DRFSSkilL Skill...
Post Rol--Boopnne Pairs From...
(Now) Knowledge Base By Data F..
(New) Emocional Suppon. Infor...
Reddit Mental Health Ditestt
Fucehook All Empurettcc Dialogu...
Content From PTSD Couch
(New)) Colleaze From
(New) User Responses
(Now) ModVAue
Training Data
Kaggle
Google Assitian, Amazon Alexa...
Googls Dialogtoow
Manycha clat Google Dialoghoo
Outsonireed Model Namuo
GPT-2, DIALOGP!
Google Dialogtlow
Googla Dialogfoo
Google Dialogfiow
Google Dialogtlon
Google Dialogfoow
Google Dialogtiow
Google DinlogFlow
Google DialogElow
Woehor, Ungghe
DialeGPP
Manychan Skill Skill
Alexa Skill
OpEnNMT
Rasa
Nors
GPT
Flipper
Alesa
Woebor
Rusa
Rasa
Alexa
GPTS
Wit.As
BEXT
Rasa
Finctuned. Used An Part Finettuedd, Used As Part Uiadd
Pinctuned, Used As a Part Finetuned.
Pari Used As Pait
Used As Part, Finetuned
Used As Part
Lved Asia Part Yes
Part
Used As Port
Uied Asa Part
Used As a Part
Uxod As-a Part
Lied Asi Part Used a Part
Uxed Asa Part Used As Pott
Used As a Purt
Used As Part
Used As Phrt
OfT the Shelt
Finetused
Lxed
Yes
Yes
Yes
Nea
Yes
Genenoiiee Retrival--ood Boodd
Retteee Rettivva-Booed Baved, Gescentive
Rule Hosed Ketrivall Bused
Rate Rosed, Remeral
Model Technique
Berricol-BBodd
Retrienal Baxed
Restienal Based
Rerrieal--Bood
Retrieal Baved
Retricaal Bassee Bosed
Retrival- sarieoa--Badd Baodd
Retrieal Baxed
Rearieval Bared
Retrevll Beved
Retrival- Bared
Retrieall Baceal-Bedd Baxedl
Retreva--Bon Bisdd
Restieaal Based
Boodd
Rettieval Rettieva-BBeed Beved
Rearieval Resrieval-Busedd Based
Role Role-Bose Bosed
Bused
Generative
Generative
Rale Bosed
Role- Kalc-Hosed
Rale- Role-Bood
Kuie- Kole-Boses
Boord
Retrional
Generative
Generaiive
Role Hale-Bossdd
Role Rosdd
Rale Dosed
Role- Role-Basc Besed
Role Rosed
Rale Blsed
Booed
Rale Rale-Bood Blodd
Rale-Hosed
Generative
Generzaivee
Retricaa-
Rale Based
Retrica
Genercaive
Rule- Bule-Bosd
Role- Rale-Bosed Boodd
Rule Based
Rule- Helle-Hosed
Cheilel Type
Task Ohiented
Open Domain
Orienue
Tosk Ohienee
Open Domain
Open Domain
Open Donain
Open Domain
Tosk Oniented
Taok Onienee
Tosk Oniented
Task Onemed
Took Oriented
Open Domain
Tosk Oniented
Took Orionee
Task Oriented
Tosk Ouiented
Open Domain
Tosk Oriented
Thok Oniened
Took Oriented
Took Diiented
Oriented
Tisk Oniene
Oriened
lask Oniene
Task Oniened
Took Oniented
Took Oriented
Oriened
Tisk Oniented
Onioned
Task Onented
Task Oniened
Taok Oniented
Open Domain
Open Domain
Tosk Oriented
Task Oniened
Task Oriented
Baok Ohiented
Taakk Oriented
Open Domain
Oriened
Tosk Oniented
Tlak Oniened
Tosk Oniented
Book Oniented
QA
Gloogl Assisunt, Amazonn Alesa...
ARIA-VALA SALUSPA Marform
GPT2, DIALOGP LOGPTTT
SociaBBo,, Chalbot
Charbee Natio
Weebor, Unghox
Trainbot
Membertix
Unmamed
ERICA, Nora
Chiabbot
MIC MICM Modal
BLISS
Unnamedd
Couchii
Soliu
Sofia
Alexa Skill
Unnomeal
Alexa Skills
Weebox
Unnamed
Chuttal
Pocker Skills
PTSDialogue
Chariic
AlwaysOn
Clevertar
Yeunbetbbx
LISSA
Vincent
Unnamedd
Unmondd
LISSA
Unnamedd
Abox
NamalBex
Unnumed
DIAKYBOT
Unmamed
AddtssonBBll
Unnumed
EMMA
Unmamedd
Unnamed
Unnamed
Unnamed
Maxx
Unnumedd
Unnamed
ChitPal
Cuylanhuug and Gioige (2021)
(20220)
Bie brandtzzgg al (2021)
Larrssen and Hawke (2018)
Quilk-Qucck and Wareen (3021)
dhhandeuriuunc. et (201 9a)
Albous al (2020)
Sun et al (2022)
Garg et (2020)
ct (2021)
Demasi et al (2020)
Das et al. (2022a)
Saha ot al (202b)
Yas and Nokashole (2021)
van Watencooot
Cox and Oui (2022)
Fudhil at (2018)
Jniswal al (2019)
Mahurjan an al. (2021)
Eagle at at. (2022)
Mahurjan at. (2022))
Quitez et al (2020)
Kawasaki (2020)
Shin and Huh. You (2020)
Kin et (2122)
De Nica et al (2020)
Lee es al (20200
Boyd at al. (2022)
Schmader er al. (2018)
Ian al. (202))
Valtolins and Hu (2023)
Sidner et al. (2018)
et (2020)
Kizawi ot al. (2003)
Lee (3019)
Ristogi et (2018)
at (2020)
Les et al (20206)
(2021)
Park and es (2021)
Wang (2020t)
Park ot al (2001)
Nie et al (2022)
Wang et (2021)
Robman et al. (2021)
Macng and Lee (2022)
Slkkkk et (2022)
Godl et (202))
ot al. (20211s)
Dianasckar al (2021)
Bhanglia al. (2021)
Despa al. (2022)
Potts ctal. (20221)
Sweeney et al.
Paper

---

(New) Data Scraped GrapedFrrmm From Counse...
(New) Marked Emtiies in The D...
Table 9: All methoee weethod/experimenn papers the final list of this survey, This table only shows technique features, Long values are truncated due to limited space.
(ew Professinals In Psychol...
Pychollogie Clin.
CDC's Mental Healld Resaurced
(Now) Workslay With Designens
(Nioe) Obsterric QA. consledgee
Body Pusitive Conerratios
(New) Study Team Members
Authors
(New) Psychologier
(New) Kola Carpus
Training Data
Outkourredd Model Naino
IBM'' Warson Assitant
Google Dialogllow
Google Diatog
Google DislogElow
OSCOVA
MindTrais
DialeGPT
Test
Rasa
X2A1
Charfell
X2AI
X2AI
Rasa
ELIZA
Rasa
Kakaoi
Liadd As Pair, Finerueed
Lived As Fart, Finetuned
the Shelt
Uxed As Part
Fart
Used As Part
Used Asia Part
Used As Part
Lied Fart Used a Part
Used Asa Part
Used As Part
Part
Used As a Pait
Used Asia Part
Yes
Yes
Yes
Yes
Rale Bosed, Recrioaal Baoed
Model Technlquee
Retticna- Sattieal-BBeeed Baved
Retrieaal Baved
Restival Buved
Retrival- -aededd--Beddd Bood
Restienal Baved
Beerieval-Buee
Retrieaal Buxed
Retrical- Bi----dddd
Retrieal Bared
Retrival- Baree-BBBreedd Bared
Resrieal Bared
Retrival- Seteer--Beoedd
Bexed
Retrieal Bared
Restieal- Buved
Retricvd- sarreedl-Berdd Besdd
Retrieal Baved
Retrical- -loeeaa-BBedd Bood
Retrieval Based
Rettieel Beoeee Baxed
Retrieal Based
Kale-Hosed
Generative
Rale Bused
Role- Rale-Hoses
Rale: Hoodd
Kole- Bosed
Role Tule-Bosed
Rale: Rale-Bose Blood
Rale- -oood
Rale-Mtod
Rale Dused
Rale Boxed
Raie Rale-Bose Bosed
Restivval
Rale Rale-Bosc Boodd
Rettieal-Bon
Rale Roood
Rale Bused
Rule Kule-Hosed
Resrival
Retrieall
Rearioaal
Retiecal
Rale-
Rale
Rale
Chelbel Type
Task Ohienee
Jask Oriente
Took Oriented
Orienue
Tosk Chieneed
Open Domain
Task Oriented
Took Oriented
Oniened
Task Oniented
Task Onemed
Took Oriented
Oriented
Took Oniented
Orieneed
Task Oriented
Taok Onienee
Took Quiented
Tiok Oneemd
Tosk Oriened
Tosk Oniented
Thok Onianed
Task Oriented
Book Oitented
Orienedd
Tiok Onimme
Oniened
Jask Oniened
Task Oniened
Took Oniented
Took Oriented
Oriented
Tosk Oniented
Onioned
Task Onented
Oriented
Took Orienee
Oriened
Tosk Oniented
Task Oniened
Task Oriented
Task Ohiented
Took Oriented
Onlenned
QA
Here4l App Milinry Version
W-SUD, (Woobet SUDs)
Natio
MYLO, ELIZA
Moodhoster Lite
Healthy Mons
Chartee
SERMIO
Unnamed
DARA
Unianned
Carefox
Unnumed
Xioole
Repiika
SABORI
Olii
MYLO
Viciber
Toss
XinoNan
Wyss
Woebor
Woebor
Wysa
Microfont
Wyss
Wyst
Teis
Viki
Viall
Siam
Atona
Xioni
Unnamedd
Uniomed
Unnamed
Rose
Ash
Bonolox
Dunji
Anola
Weehot
MIRA
Pophooss
loy
Unnamed
KIT
Gromesskem and Daniin (2002))
Ghandehrrson et (2019))
Hungemoccherr et al (2021)
Deneoke al (2020)
Schwarte or al. (3r22)
(20226)
Narynov ot (202100)
Crssoetall (2021)
Chan al. (2022)
Zou etall (2022)
liang at (2122)
Bennion et (2022)
Sugamuno al. (2018)
Gaffiey et (2020)
Maniamo at al, (2121)
Provoosi at (2020)
Greer Kios al (2021)
(2UZ))
Linden al. (2020)
Gupra et al. (2122)
Proddnaka at (2021a)
Prochaska et (202116)
Greein (2021)
Darcy et al (20220)
Sinta (2022)
Schick al. (2022)
Boilly at. (20Z2)
Mebell al. (2022)
Doswiskky al. (2020)
Douvviaky (2021)
Daley et (2020)
Ly et (2017)
Gobricllier cral. eral. (2021)
(2022)
Park et al. (3022)
Hossi (2021)
Burger et al (2022)
De Genmuru al. (2020)
Park et aaaal (2019)
Rahmuyaka et (2n02)
Lodin al, (2002)
Fitparrck al (2017)
Noble at. (2022)
Maseiollo al (2021)
Chong al Morris al. 12018))
Beilbarz (2021)
Grove (2021)
Malurjan
Paper

---

Nen- Men.PPrnnriic Man Whinney U.... Mixed Mossed-Moddil ANOWA Tukey ISSD ......
Table 10: All method/experiment papers in the final list of this survey. This table only shows experiment features. Long values truncated due to limited space.
Indd midependonnn amplec Eoto, De...
Mann-WWiins U Test Lince Re...
Piets Kappo, www..ooppeeeeecccc M..
Shapieo Will Tis, Lovere's E...
Mixdd Model ANOVA, Tuley
Spetrnn's Bonk lonk Commiation Co.
Krukal Walls Parssn
Noc Punneric Mann Whitery le...
One Valed independent Teas...
Painel Sumple e T-st Medition.. One- One-Wey May- ANOWA Hoe Thkey
Peurson PearsonnCoreee Coreetation Coetiicte...
Pearson Cormeaatinn Coefiicien..
Parial Leas Sqaare Stretur
Wilcones Signed Rank Tei
Cohen'' Krigponteel Welal's Tes
Mixed troign ANOMA
Mixed boade MMM.EAANOO ANOWA
Tho Tattel Tatee Pained I-Ted
Ove- lad Pared -Tebss
Wetch Two Sampto T-Tee
The Ohe Suded Siked -To
Inolepoddon Sample
Ove- The-Saall TTa--a -To
Two lailed i Vest
Tukey' HSD
Stulk
Eihiss
DR3
Yes
Yes
1R3
100
IRB
083
Yis
Yes
Yoi
DR3
DR0
1R09
Yes
IR0
BRB
Yes
No.
URU
IRB
Yes
DRS
Eayoyssot, Presucc Holping S...
Oorail Exprione Fmmpulyy A...
Cohenssyy Conteency Farnec...
inmmutattion Back Baxi:Pagaatn Peydmolit... Furnyy, Mayeaaiir Motval...
Lictloo Didiane, Enjepun... Indiidaal Sal Conddros And...
GAD-T, EEND-,Tiemmee Tremment, Enguh.
Arpreaa af Sirpon, Emontonl
Expelienc Qucdinntaiee Syeem Unbilligg...
Kosla Poycholopicaa Dia aisreo....
(1) Kosoas for Revwees Use...
Rales, Ponuss, And Challeuge.. Paycholrpial Diries Seitess.sssan Asscrmn.. Seif Dielzzu Level Level Cosari... Usgr Of Casbol, Boneits Ch. Sysem Usbilin Sale, Chabo... OASIS, PHQ-9, DRT
Prappies Accpptoa Pore.. Sociecenngpppic Quezonalss.
Koolan Psyshologio D Datraao Center for Eiddemskki biikkgi Stud...
Seif Compustion Stale, Incell Leam
ticiuiiss Peoepheees Ro...
Cangamte And Levek. Iraa, t Pe Patonnus Hunalss, A Afteci...
Bundon Hocdd By Chaibol Stat Public Speaking Ansiev.... Sctwant Ontcome Saals, Cirnic.. Overall Scaiing MiLingees..... Oammsrr Ceneetrs. Relvomorr itteciieees Consareccy 1) Assudiiity 2) Appupei.... Use Pheferates
the Experiess ao Quedimnaiee
Big live Persmally Thus P...
PSS in Addiooal Addiiral Quaslist......
User Experience Quetonaars...
Weigtt Concems Stals, Sunfor.
Peromiization Miker Intemt.
PRO GAD-? BRL-1
Iaman Brahuubot
Dilically Expoyment
Appootiun Diene
SASSI Scotes, WHO PHi
LPDS WENWES Reformnce
mIQ-0 GADS3
P9Q-9 GAD-7
WLO
Number Monaggs Length ... Algnnei
Negniie Lox L.telhood, Emnm.
Lesiel Divendy, hormuss On...
BILBU Peplesiy, Roteee,,,,uui ROUGE L,
Aucerayy Negative Likdnno...
Completion Time, Commintion D..
Word OrPostivnN...
Completion Time, Success Phopo...
Sona Seasion lond time, Di...
Monetryy, DeosttrrDDDeeeeee Demenna Infommtioon....
Elleeracctece Settinctt And y.. Lnror Rore, Tool Word Cont
Domeniion Classiicatio aiion Accrc..
Roppase Lateryy Froquency OK... Clasallstion Accumery Prectsion Recall F Score. S..
Accency Precision Recall, E.. Aserncy, Prosicce, Rocall, F..
Uver Vesure, Lniqu Days. loa.
Chailei Spaton Lostle Length And Chi... Ahe.
Word Count. Wond Leughh Camplition Ruae
Iraltattonn
Esperittee SepretteSSeeliie Sampling
Sentmeen Anagas
Word Anteaction line
Accuaay
BLBB
Day, Weeks
Doys, Weetss
Duratbon
Weeks
Weeks
Weaks
Wooks
Wiek
Month
Weske
S Wooks
Weeks
Week
Days
Week
Wesks
Week
Weeks
100
156
30, 5
58
35
59
4.2
16
10
30
18
25
47
100
10
12
44
163
24, 25
30
07
47.9
07
25
19
53
50
25
39
24
211
21
39
12
23
210
371
AfTick Werken Domain Exper...
Google Fashhakk a Netwok Of..
Community adheniiemeei Oupa...
Throug Dovioopentttl Behh....
Cunventens Sunplag Lig Email in.
Crondvvtteos And Coundlers
Recurdd from Local Usiven....
Trained Recardhes Meral li...
Laiver..
Ulavesity is Soaath Clais Eis...
Setier (ligh Ifigh Scchen Schnol Stukkms Urivessity Studers
Action Menta Health And Ulie....
Rossital DRT Lisuerre
Pots Helss Pr...
Cliii Eiderty Coner, Walar...
Sectal Media Weteiles Univers.
Soctal Media, Pemoraal Coneat... Roaebal Our To Shodenss in PMo..
Schois Calegs, Universtt Recmite
Nationa Reenknent Site tin....
Sodid Molia, Reconal...
Pysisana, Peychologist
Amnzn Modicasioal Tark
Rectrted In Linoersittes
Expert In Monal Hoali Hellt
Far Te Bigger Pigjer Project
Per of The Digger D
Project
Salyst Mallie Malter Malie--Maerssseerr Esyests
Suukeas Relaivos
Partcipml Danbuse
Univenity Stuforrs
Chipliar Poss
From Collage
WeCCor CeCCorComip Grips
Recraiiment
Rachal Out
Socid Modia
Recuted
Recuied
Recured
Voluncers
Rocuted
Rasmall
Rocured
Pholide
Uses
Uses
Nin
Conquaatvv Amayss
Comporatios Andlyii
Compuativ Analysi
Comparatvv Amaystis
Comquralie Andlysis
Compinatvv Anaayyy Amaayti
Compusatve..oo Asayyi
Conquattvv Anayysi
Compunativ Anayyis
Comparatvv Anayyis
Comparitvv Anaaysis
Comparative Amalysis
Compantter Analysis
Compuration Amalysi
Compunaivve Analysis
Comparaive Analyis
Compantive Analysis
Compuniive Analyii
Stods Dedggn
User Stedy
Lier Stady
Ehar Stadyy
Stary
Compuse
User Stady
User
User Stadyy
Usei Stody
Uher Stndy
Study
Stady
Daen Stedy
User Stndy Stady
User Study
Use Stadyy
User Strdy
Usei Stady
Study
SterStody
Daer Stadyy
User Stadyy
Ther Stely
Esen Studyy
User Study
User Study
RCT
User
SCT
RCT
yan cun Covkerargg and Gionge (2021)
Smetal (2022)
Cury at (2000)
tess ai (3821)
Demos etal (2000))
ot (2022a)
Sala al (21220)
Nakatolk (2001)
Whttenoooon al (2220)
and Out
Fodhil at (2018)
heowal ctal. (5019)
Sbttrraine eral (202))
Eigl at, (2H22)
Urrndrrogg al (2011)
Quiree eral (2n222)
(2000)
Kawaukii al (2000)
Skin and Hob-Yow Yow 20000)
Kin al (202))
De Niew et (2000)
Lee al (2130))
Bops Awoonyettl etal (202))
(2E2)
Scheoller al al (2018)
Ban (312))
and Ho (201)
skiner at (2013)
and Hooke (3008)
Ryu (2130)
Robavteral ( (3002)
Lee al clolly
Hol Qokkk-okkknn and Warmn (3001)
Aliet 20200)
Lee al (20200)
Sio at (2221)
Bark and Ler (001))
Wing dt 1210000
Park (2021)
(202))
Wing (3011)
Ralmin elal. (2021)
Mang mnl Lar (100))
Chanddhution al (2019))
Saiik ai (2022)
Godl al (20211
Wing (20000)
Wastebee at (200))
Doga at (3023)
Port etal etal. (201)
Donske al (8000)
Chandelmmau at (201 Shj
Schwatrr ctaa (2IC2))
Mabrranneaa (0220)
etyyoret al (2021))
Chan (202))
2hr etai. (282))
thangdia eral
Gradi
Restg

---

Wilorron Ie...
Table 10: All method/experimeen papers in the final list of this survey. This table only shows experiment features. Long valuee truncated due to limited space.
ANOVA Inkepnient Tosss
Tho Flater Pistor Mixed Model ANOWA
Painal Sumples TToas And 1-30-
Power Fover-Andyy Andlye,, Paired Sanple..
Pasel anghti Regrosiion
Inkkenknii Taatt And Chi-Sg-
Wileooos Signol- Raai--akk Tou Pa.
Puire! cimper e lests And Chi
Puinel Painel Sunples TTass And AndMMN MeN.
Speamnn Order Conclnits
Nompunmers Bayesian
ANOVA, Ropesndd Meavure The Wonoonn Sigted lank Tea
Mann Whasey U Eas, Placed I..
Coobuah's Alpla. Speirmants R...
Coten's ' Sunnaaaa Coem.... indepmment
32-Te Stapio Tot, Fured Sumples t..
Pourr Andyyss Off Cocatinn.. Coabad''s And Andmttattioo
Speamin's sp Indkyendnn SeandennSaaggiee Sangles I-Tos, AN...
Colen's ANCOVA, ANOWA
Wilcones Signed Signel-Rank Rank Test
Syeamun Corottton Stagin....
Chi-Sgune General
Ott Chi-Squat Amaysss
Staptro Wik
Stalkdeal
Pean
Mann
Nos
Yes
No.
Neo
Yox
DR0
DR3
D000
DR0
Yes
Yes
Yes
Yox
IRB
000
Yoi
Nex
DR3
Yio
Pewsol Problems Helptiness
WHO 5.1, K.R. K. BADS AC. BADS SHAL Ix. GAA
IUS-112 ONSA Helpt bress Key Mechamens OF.. Ressival sensivellleee Emoooonk la Vilenxx I... Motrition beedheck LtC.
NPRS PROMSS-2. PRD CIAD-T....
Change ia Past Sheth Stbatinn Substioce... The Abatial the Diooders Mden.. Pil Working Allunce
Experiones Baluned Inventory.
Working A Aliine Inventrg Sho..
Phini- GAD-7 DASS-21, Insemnt.
PHO-9 Daynnek AniSSudiii...
Challer Uniniona Dicioous.
WiQ lnpagement In Selt Stet.. And And Negaivee Aflan.... S... Pollis
Pettippnn Intress And Th. (PSS.Ini.
PHDD,, GAD-7 PANAS, Accepiati.
Clinical Outonnss Is Rouise E..
Stnoe Stre Levess Leee, Sleep Quality.... USE Queskouniic Percived 8
Usbiiiy, Sugrorenns Idenn mag
Roundd Socta Modia Isto
(Sytouss
PRD u, GAD- DASS-ZI
Fasiiiiity Asuquabiliy
Roniding Scal, The
Contents Strucnne And
Pag Uiebibe
PRO GAD-7
Engagencnn
Use hainng
Pecesved
Chaibo
Toul Meoegrs Som F FrumToo Us...
Seainns
App Log., Number
Textud Snippets From Users
Tonual Suppers, Tox Usgao,
Emeesbanccs Engagoment
Eraalationn
Frequency, Duraton
Schetoling
User's Untmmnoss
Adhamauu
Adtermnc
Atteveation
Trme
1.2 Weoks
Mouth
Wookk
Weeks
Weeks
Wooks
Weeks
Warks
Days
Dogs
Moath
Dips
Wooks
Weeks
Week
Weat
Weeks
Nua
191. 203
29
15
10
25,
91
130
101
350700
10
1205
2094
354
5005
77
9629
14,
308
338
64.
60
30
70
49
13
$71
Adveriseal Dver The Moh Fosd...
Luee net Ro mescccch Compuny Fazibak bocllakl.aaagg fasigam Toilic....
Flyer And Rachook Advert Sem..........
divertseeens Digitl Meal...
Quiirics Sunceed Linsee Linerrs...
Lies Social Media, Medi,, Criggtie....
Onhopelic Ch..
Email lurnnt Bannn, Lea...
Univemitios Wabslee Social
Soctal Mudia Online
Pollcc Crowe Crowdd--orccii Souring Pla..
Suoweel Suig Suigling Socid Mesi...
Word Or Mouth And And University.
Roo Clasc. Suowiak Sungling
Socid Medua Online
Fazibaak Suniveoliig
Recured Univertty
Anaoon Mechanticl/// York
Depustmnt Subjea Pook
Bateinn
US Univenity Students
Tatcred Commanicss
Univesit's Ressach
Emmal Teleytone
Storral Sampling
Resemaatios
Osline Festor
Hosphl
Factbok
Recureed
Univesity
US
Now
Uaes
Uses
Uses
User
Compuratvv Anaysis
Compastio Amalys
Comquativv Anaaysis
Comparativ Anayyis
Compariive Anayyii
Compusatee Asayyii
Stedy
Dhen Stedyy
Lier Stedy Stady
User Stedy
Uhes Siddy
Eler Study
User Siddy
User Stadyy
Usei Study
User Strdy
Ller Stady
User Sindy
User Stody
Stedy
Study
User Stedyy
User Stady
Eses Stady
User Stody
RCT
SCT
RCT
User
User
RCT
Geeendemon and D-okin (20nt))
Hongehhoheer at (801)
Rennion et al (300))
Seganima at (3018))
Gamteey at (20200
erooost at (2020)
etal (2019)
Linden al (3000)
Guplo al (2022)
Procheilee esa (202
Pochoaks at (312
Grend (212))
Beatt ata (1822)
Melali at 13022)
Dosovislyy at (2000)
Dossitaky a. at.. (302))
Daley (207))
(3020)
Gahnell c.d.cao (2121)
al (302)
Puk at all (2022)
faaanni al (3021
Burger etal (202)
De Gomnmmeeaa ctal. (2020)
Pakk (5,2
Rohhapaka et
ikraaiike at (2013)
st (2022)
Mtunelo al (201)
Chang at (221))
(10.5)
Beltuz (2001)
Grove (000))
lang


Reviews:
[{"review_id": "wQsNOSBcMj", "reviewer": "Reviewer_e2C5", "summary": "", "questions": "How will the recommendations of this study be adapted to accommodate the current virtual mental health assistants, particularly in light of safety (safeAI), changing user preferences/behavior/symptoms, and the dynamic ethical considerations associated with mental health support?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and suggestions. It seems that you have specific venues in mind that we are missing, can you please give us specific examples that we are missing which would cause bias?\n\nThe scope of our paper is to review studies focused on conversational agents published across the CS and Medical fields and we believe the current databases (​​ACM, IEEE, ACL, AAAI, PubMed) cover most of the edge-cutting studies, if not all, in both CS and Medical fields. ACL Anthology (*CL) indexes over 42 ACL and 21 Non-ACL conferences, journals, and workshop proceedings in the computational linguistics and natural language processing domain (ACL, EMNLP, NAACL, EACL, COLING, among others), ACM Digital Library publishes proceedings of over 40 journals, 270 conferences, and 9 magazines spanning several CS domains including HCI and computational linguistics research, IEEE indexes over 200 journals and 1700 conference proceedings specializing in engineering disciplines, AAAI covers over 26 proceedings such as ICML, AAAI, IAAI among others in the AI field, and PubMed includes journal articles published in 5600 journals with a specific emphasis on biomedical studies. We believe our selection criteria cover most papers dealing with conversational agents for mental health. We examined psychology-specific databases, like APA PsycNet, for keyword searching. However, we found that the results were minimal (4 in total) and existed in PubMed search results. We request the reviewer to highlight the specific venues that we missed leading to a bias in the study, and we are excited to add them to our list. We followed and expanded on the venue selection of the CS field from a previous survey paper ([Valizadeh and Parde, 2022](https://aclanthology.org/2022.acl-long.458.pdf)), which also has a similar range of coverage in the final list. However, our paper differs in the following ways: 1. We encompass all categories of conversational agents, not just those that are task-oriented, 2. We incorporated PubMed in our database selection to provide an integrative view of two domains, 3. Our focus is on mental health conversational agents, not all healthcare applications.\n\nEthical considerations are one of the most important focus in our feature selection. Among our full list of 102 experiment papers, 70 papers mention ethical considerations, and 26 papers are IRB-approved. While data privacy and mentions of violence in language are the main concerns among researchers, we found that previous studies, especially in the medical community, choose not to use large language models because of wanting complete control over dialogue generation. From our study of over 100 papers, we belive addressing challenges such as biases, non-deterministic nature of generative models, and input/output data security should be the main focus for the NLP community to help LLMs to be broadly accepted in real-world situations. \n\nFor further explanation of line 663 and 665, we found that CS databases had a large number of papers on the database search stage, but passed a smaller number of papers at last, which is significantly different compared to the inclusion ratio in PubMed (17.0% vs 60.6%). This is due to many papers from CS databases mentioning mental health as an important application while introducing background and application of conversational agents, but did not design a mental health-specific solution. This phenomenon shows that the CS community understands the importance of conversational agents as a support to help with mental health issues, but did not give an equivalent amount of effort in model design. Our paper aims to reveal this gap and raise attention to CS communities. \n\nAlthough changing the paper's main focus from conversational agents to general AI could include more papers in the final list, that diverges from the focus of this paper on conversational agents, given the rise of focus and technology in large language model driven chatbots. Furthermore, our paper aims to provide key takeaways to the *CL community by summarizing and comparing how two different fields, CS and Medicine, approach this task differently.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors targeted an essential space to review with a very specific set of research questions to answer/understand (line 100 - 106). \n\n2. Diligent selection of papers through the PRISMA framework.", "reasons_to_reject": "While the paper displays a diligent collation and intelligently observed takeaways, here are a few concerns pertaining to this review.\n\n1. Although the PRISMA framework filtered a significant number of papers from the 534 papers initially recorded. It is easy to see that authors clearly missed not just a few but a significant number of papers from other potential venues. Nonetheless, I would like to raise an important point regarding the potential inclusion of additional papers from alternative sources. Drawing from my experience in both medical and AI domains, I believe that the selection of papers from various venues (9 from ACL, 4 from AAAI, 20 from IEEE, 40 from ACM, and 63 from PubMed) might benefit from a more diverse and inclusive approach, thus reducing bias and including fairness in the study. The reasoning to this in line 665 does not realistically comply with the statement in line 663. With this, I am not concerned about the frameworks' selection but would like to know the author's views on opting for selective venues. \n\n2. The findings are interesting; however, I find it inclined toward a limited selection of papers and hence carry a bias toward particular needs in the medical domain to improve. Emphasis should be on collaborative working in both fields (AI and Medical) and how it can be further nurtured. However, I would like to highlight the potential inclination of the paper toward addressing specific needs within the CA domain, leading to bias. To enhance the overall impact and applicability of the findings, I recommend placing greater emphasis on the importance of collaborative efforts between the AI and Medical fields, not just in CAs but in peer and multi-party conversational analysis as well. It will be a more holistic approach to addressing the challenges and opportunities identified in the paper.\n\n3. While the paper attempts to bridge the gap between computer science and medical studies, specifically focusing on CAs, the pool selection raises concerns about the generalizability of the findings. As it stands, the paper's conclusions and recommendations could be seen as disproportionately influenced by the chosen subset, diminishing their applicability and relevance to the wider research community.", "questions_to_authors": "How will the recommendations of this study be adapted to accommodate the current virtual mental health assistants, particularly in light of safety (safeAI), changing user preferences/behavior/symptoms, and the dynamic ethical considerations associated with mental health support?", "paper_topic_and_main_contributions": "The paper is a comprehensive survey of 136/534 papers in the space of conversational agents pertaining to mental health (or virtual mental health assistants) published in both computer science and medical venues. In the attempt to summarize the paper's takeaways, the authors presented some interesting findings on transparency, ethics, and cultural heterogeneity, along with essential recommendations to help bridge the cross-domain gap.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Authors are advised to dive a little deeper into the available study and opt for a holistic approach to self-assess the selection of venues for PRISMA.", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "IMjBw5ziTc", "reviewer": "Reviewer_5Bp6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful suggestions. It is an excellent suggestion for the study to be inclusive of proceedings such as COLING, CHI, and INTERSPEECH. COLING is already included in ACL Anthology (referred to as ACL in the paper) and CHI is included in ACM sections reviewed in our paper.  Following the reviewer’s suggestion, we searched for publications from INTERSPEECH in the DBLP and the ISCA Archive, but unfortunately there is no paper in INTERSPEECH that has (“chatbot” or “conversational agent” ) and (“mental health” or “depression”) at the same time. We have also created an additional table summarizing the paper counts from each of the venues indexed in the 5 databases we surveyed. ACL Anthology (referred to as ACL in the paper) indexes 63 venues (ACL, NAACL, EMNLP, COLING, etc.), ACM covers over 40 journals and 270 conferences (including CHI related venues), IEEE over 200 journals and 2700 conferences, AAAI over 26 conferences, and PubMed covers over 5600 journals. The table below shows venues in each database, and corresponding experiment papers in our final list. \n\n\n\n| AAAI  |       | ACL     |       | ACM             |       | IEEE   |       | PubMed               |       |\n|-------|-------|---------|-------|-----------------|-------|--------|-------|----------------------|-------|\n| Venue | Count | Venue   | Count | Venue           | Count | Venue  | Count | Venue                | Count |\n| HCOMP |     2 | EMNLP   |     1 | CHI             |     9 | ICIRCA |     2 | JMIR Form Res        |     9 |\n| AAAI  |     1 | SIGDIAL |     1 | ACM-TiiS        |     4 | ACII   |     2 | J Med Internet Res   |     7 |\n|       |       | BioNLP  |     1 | IVA             |     4 | Others |    11 | Front Digit Health   |     4 |\n|       |       | NAACL   |     1 | ACM-HCI         |     3 |        |       | JMIR Mhealth Uhealth |     3 |\n|       |       | NLP4PI  |     1 | UbiComp-ISWC    |     2 |        |       | JMIR Res Protoc      |     3 |\n|       |       | LREC    |     1 | CUI             |     2 |        |       | Digit Health         |     2 |\n|       |       |         |       | PervasiveHealth |     2 |        |       | JMIR Ment Health     |     2 |\n|       |       |         |       | Others          |     9 |        |       | JMIR Hum Factors     |     2 |\n|       |       |         |       |                 |       |        |       | Internet Interv      |     2 |\n|       |       |         |       |                 |       |        |       | Others               |     9 |\n\n'Others' in the table refers venues that only have one paper. They are:\n* ACM: CHItaly, ACSW, H3, Asian CHI, DIS, CHIuXiD, ACM-HEALTH, IASA, ECCE\n* IEEE: ICoICT, UCET, ICCCI, ICHCI, ICACCS, ISCC, IEEE Trans. Emerg., SIEDS, IEEE Pervasive Comput., ICCAS, INCET\n* PubMed: Curr Psychol , Comput Math Methods Med, Inf Process Manag, Front Psychol, Trials, Front Psychiatry, Drug Alcohol Depend, Sensors (Basel), JMIR Med Inform \n\nWe are very grateful for your recommendation about the keywords. We included \"dialog system\" in the initial set of keywords in our review, but found that it only includes a small number of non-duplicated publications (7 from ACM, 0 from PubMed, 0 from IEEE, 7 from ACL, 3 from AAAI) compared to our current set of keywords and many of them did not satisfy our criteria for paper selection. Consequently, we excluded this keyword from our search. We will highlight this in the revised manuscript, given an opportunity. Below is our search result using \"dialog system\"\n\n|                     | AAAI | ACL | ACM | IEEE | PubMed |\n|---------------------|------|-----|-----|------|--------|\n| Search Result       |   10 |  13 |  79 |    0 |      1 |\n| After Deduplication |    3 |   7 |   7 |    0 |      1 |\n| Full-Text Screening |    1 |   0 |   1 |    0 |      1 |\n\nIt is true that many of the papers we reviewed use traditional approaches for conversational agents. However, we want to state that our search criteria are between January 2017 to December 2022, which is before ChatGPT became famous. It is also worth noting that Medical domain papers implement rule-based systems based on psychotherapy or behavior change which are easy to program for controlled text generation, which we found not to be the case for CS papers. Our paper acts as an integrative review of the pre-LLM era studies in conversational agents, and provides insights for future studies in both CS and medicine fields. We found that there were specific reasons why previous studies and applications hesitated to use generative models (See Section 5.2). For instance, uncontrolled responses from LLM could possibly raise safety concerns in addition to ethical concerns. Technical requirements also were reported to act as a barrier for the medical community. Addressing these issues could be one of the directions that the NLP community can take.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough overview of different aspects of conversational agent research in mental health and computer science.\n2. Identify the gap that mental health CA has not benefited from most recent development of NLP methodologies.", "reasons_to_reject": "1. The inclusion criteria is not inclusive enough. Consider i) other proceedings such as COLING, CHI and INTERSPEECH that also includes research work about conversational agent. ii) Expand the keyword list to include 'dialogue system', which may increase the completeness of the paper list.\n\n2. As pointed out in the paper by the authors,  the intersection of mental health and conversational agent is still limited. The technical part of the paper is mostly about RNN, Transformers or rule-based approaches, so I feel like this topic may not be of great interest to the general audience of EMNLP.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper conducts a systematic review of mental health conversational agent(CA) papers and discuss the gap between the usage of conversational agent in mental health and the advance of language model in general AI research. They authors curated paper from multiple databases and conducted a multi-step filtering procedure to collect literature that match the topic. Distribution of multiple features from the finalist including languages, model techniques and target users were shown in the paper. The authors conclude there is a disparity in research focus and evaluation measurement between mental health and CS CA research but also have some common weakness in transparency. The authors call for evolving more recent technologies for mental health applications to bridge the gap.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W7IrIn8huw", "reviewer": "Reviewer_2fQL", "summary": "", "questions": "1) Were there any interdisciplinary work reviewed between CS and medical field?\n2) Did you examine any psychology or psychiatric specific databases?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We aimed to give a thorough summary and comparison between the CS and medical fields about the work on mental health conversational agents differently. By comparison, we were able to suggest possible enhancements to both communities and provide future directions. Since interdisciplinary works consists a smaller portion of our final list (20 over 102 based on author affiliations on papers; 7 from ACM, 2 from IEEE, and 11 from PubMed) and there is a large disconnect between CS and the medical field, we hope this paper can act as the bridge between two communities that help to learn from each other. We will add affiliations in the feature list on the camera-ready version.\n\nWe examined psychology-specific databases, like APA PsycNet, for keyword searching. However, we found that the results were minimal (4 in total) and existed in PubMed search results. So we did not include other medical databases in our criteria.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is extremely well-written and highlights integral issues regarding research on mental health conversational agents. The authors objectively summarize and discuss how the two different fields are approaching the matter and make good recommendations to improve the matter. It is interesting and incredibly relevant in the modern day, especially with the rise of AI companion chatbots being distributed and used almost everywhere.", "reasons_to_reject": "The authors could provide more detailed recommendations for the two fields moving forward, as well as some suggestions for interdisciplinary work on this matter. It may also have been beneficial to examine psychology or psychiatric-specific research databases (in addition to PubMed) considering they were looking at the use of CAs with mental health issues.", "questions_to_authors": "1) Were there any interdisciplinary work reviewed between CS and medical field?\n2) Did you examine any psychology or psychiatric specific databases?", "paper_topic_and_main_contributions": "The paper is a literature review comparing research on mental health conversation agents in the computer science field and the medical field. This paper highlights the disparity between the two, especially including the differential focuses in the fields when researching conversation agents - CS focuses more on the technical aspects of the model while medical focuses more on the user. Moreover, the paper highlights the technological gap between the two fields, with CS using more advanced models compared to the medical field. This paper does well to discuss the gap between the fields which should collaborate on this matter", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a survey on mental health chatbots, paying particular attention to "bridging the gap" between medical and computer science perspectives.

**Pros**: Reviewers all agree on the importance of the problem. Most reviewers highlight the paper as well-written with clear takeaways, bringing light to an understudied area of health-oriented conversational agents (i.e., for mental health). Reviewers find the survey to be "thorough" and "diligent". 

**Cons**: Regarding the last "pro", two reviewers agree the survey is comprehensive, but express concerns about potential selection bias (w.r.t the papers chosen) due to an apparently limited number of papers from important venues. Authors provide a detailed rebuttal with exact counts from mentioned venues, but some reviewers still express minor concerns about this. One reviewer would have liked to see even more venues, while another reviewer was still struggling to make sense of the numbers (based on their existing experience). In any case, these concerns may be more minor as evidenced by the scores.
Decision: Accept-Main