Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper presents a conversational system, Expressive Interviewing, that is tailored to assist people in coping with the impact of COVID-19 on mental health. The system asks users a variety of questions in an attempt to make them write about their feelings and emotions. Before and/or after interacting with the system, users are asked various questions about their emotional states and their interaction experience (e.g., stress levels, life satisfaction, meaningfulness of interaction with the system). The system uses word occurrence frequency statistics from user inputs to select appropriate questions for specific topics or word categories. To validate the system’s functionality and usefulness, the authors collected interactions with 174 users and report a thorough analysis on the impact of using the system based on the collected data. 
Furthermore, the system’s usefulness for user interaction related to COVID-19 is compared Woebot, a related general-purpose mental health application offering a conversational agent. Based on data from 12 participants discussing topics related to COVID-19, the authors show that using Expressive Interviewing, participants report lower stress levels as compared to using Woebot, and that it is easier to use, motivated the users to engage in the conversation more strongly and led to more meaningful interactions.  

The paper is well-structured and easy to read, the interface is straightforward and easy to use, and the idea of providing such an online tool is interesting. The quantitative and qualitative analyses are thorough and provide insights into how users interact with the system. The system’s internals are only described in the text, a figure or table to illustrate its functionality would be very helpful to better understand the underlying processes.

The comparison against Woebot furthermore suggests that Expressive Interviewing is better suited for users discussing their concerns related to COVID-19. However, the authors do not provide a justification for why Woebot was selected for comparison and why their system was not compared against the Wysa tool. Is there a reason for this? Furthermore, the sample size is very small and statistical analyses would support the findings of this experiment.

In sum, I believe that this is an interesting tool which has the potential to find applications in in the context of COVID-19, but I would encourage the authors to better illustrate the system’s design and to conduct more detailed analyses on the comparison to the Wysa/Woebot conversational agents.

Other comments:
- Table 1: there’s a typo in the caption (“ratinga”).
- Table 5: in the caption, there’s a missing “of” after “Comparative evaluation” and a typo “...with scores > 3 in a 7-point…”.

review_title: Interesting tool and thorough analysis; a better illustration would be helpful

final_score: rating: 4: Ok but not good enough - rejection
review_content: Overall - I like the idea and the paper is well-written. The online system is great - I did it and I much appreciate the authors' effort on making it available.

I feel that some comments need addressing;

- intro paragraph: references are missing. It would also be good to look at some of thee other work on well-being, mental health, etc. done on COVID-19 (e.g. in the COVID ACL workshop)
- p1 (last sentence): you say multiple meta-analyses but mention only one. Please add more.
- p2 (paragraph on MI): maybe add example questions used in MI sessions
- general aspects about the chat interaction:
- - how long was the interaction?
- - who ended it?
- - how often does the system follow up?
- - what are the rules/criteria for following up (or not)?
- - what were the criteria for detecting emotions?
- - how was the detection of topics and emotions validated? And what was the performance?
- - how often were responses flagged for the reasons given on p3 (right column, middle)
- - all in all: the comments above all require more detail on the system
- related: the sample is quite small: why not use it in an online study with a few hundreds? This would allow for a more representative view on the system and its comparison tools (see below)
- another big issue relates to the statistical reporting:
- - since you present multiple correlations, you should correct the significance alpha level by the number of comparisons (e.g. 0.05 for 1, 0.025 for 2 comparisons, 0.005 for ten comparisons, etc.)
- - absent that Bonferroni correction the findings capitalise on chance and might contain FP statistical artifacts.
- - using p < .05 is already very liberal (many argue that we need to move to stricter values --> https://www.nature.com/articles/s41562-017-0189-z)
- - an interpretation of "just-not" significant relationships is inappropriate as these fail to meet your own statistical criterion (even in its uncorrected and liberal form)
- - the majority of the results section would thus need rewriting
- please add detail on the participants (who were they, where was the sample obtained from, etc.)
- a comparison to a non-chat-based tool would be interesting (e.g. how does your system compare to simple writing?)
- p7 (left col, bottom): "more associated with the group [...]" than what?
- the comparison with Woebot is interesting but incomplete - I'd like to see a proper sample that meets criteria of experimental research (e.g. either they get A or B and then you assess which one was better).
- - the comparison uses Likert scales but you report a binarisation based on an arbitrary cut-off (> 3) - please report means and SDs so we can assess differences between the users' ratings of either tool
- - both points above will be solvable with a proper, large sample
- I appreciate the scope of this paper is on introducing the tool but I'd really like to see a section on linguistic analyses of the responses (what did they write about, did emotions change during the coversation or depending on the topic?)

Overall: I'd like to see this paper reworked with proper analysis. The tool is great but it requires more adequate statistical tests and more analyses to fully introduce the system (and its limitations).
review_title: Interesting but needs more detail, proper analyses and method

final_score: rating: 2: Strong rejection
review_content: 
**Recommendation:** Authors must include system description that would allow for reproduction. 
#### Summary
**Strengths**
1. Novel approach to chat system
2. Comparison to Woebot is promising, but not thorough
3. Authors engage with ethical nature of their work
**Weaknesses**
1. Missing methods section is glaring; system is not explained or reproducible
2. Authors should prefer interval or effect size over significance tests
3. Authors should discuss how they intend to assess their system earlier in the paper

#### Review
The authors introduce a COVID-19 specific mental-health support chatbot that focuses on motivational interviewing and expressive writing. The assess the correlations of various mental health statuses against the time users spent using this bot in a study. Additionally, they compare user-ratings of satisfaction against a standard system: Woebot. The paper has major shortcomings in terms of system description and analysis.

The paper's chief flaw is the omission of a meaningful system description. The author's have an obligation to describe the system in such a way that a skilled practitioner (or team thereof) would be able to reproduce the system. The only hints to how this system works are that it uses motivational interviewing and expressive writing as influences and that it is built using Python and Javascript. The authors should describe which NLP techniques are used for the various parts of their chat system and how these pieces interact with one another.

Additionally, the paper's analysis is lacking. The author's use significance testing for their analysis, which makes it hard to gauge the true meaningfulness of their findings. They should prefer effect size or confidence intervals. Additionally, no statistical testing is done for the comparison with Woebot; however, this does not stop the authors from making claims about differences between the systems' performances. The authors must perform statistical tests if they wish to make claims about the differences in performance. 

Authors should review [Yan, Song and Wu (2016)](http://www.ruiyan.me/pubs/SIGIR2016.pdf) as example, noting 
1. Well defined system methods 
2. Clear distinction between system description and system evaluation

#### Reproducability
This research is not reproducible.
review_title: Lacking methods and approach