Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
the article describes the process and end result of creating a chatbot aimed at providing vetted COVID-related information to the public.

General impression:
Jennifer clearly represents a considerable effort, undertaken by a talented group under great time pressure, using vast amounts of information that is highly uncertain and in flux. 

Now for the submission: to put it perhaps soberingly, it is 60% helicopter-view engineering white paper, 20% sales pitch, 10% incidental observations, 8% future work, and 2% science. It is the virtual absence of scientific substance that is my main concern. 
As one illustration: the manuscript has 24 citations, 11 of those look peer-reviewed, 5 of these are provided in the future work (‘ongoing challenges’) section, therefore not supporting the past activities but rather motivating future ones. Several of the other ones are referred to outside of their scope (e.g. the Vargas reference). Related research on chatbox design, the spine of this work, is all but absent. (2 citations, from 30 and 15 years ago). 

Note that the submission has no ‘Results’ section. No systematically collected observations or other evaluation materials are introduced. At the very least, end-user usage statistics would have been informative to assess the impact of this work.
The engineering decisions that are reported make sense, but are largely not substantiated with supporting literature or first-hand evaluation. 

While it is claimed, it is not clearly demonstrated that this project addresses misinformation (or disinformation). It is an access point (and repeater of) trustworthy information but not one that specifically points out misinformation, or hunts it down. It is not fully clear how accountability is ensured for the automated provision of information to the general public – can users easily vet the sources of such information, and is there a path of recourse when a user spots incorrect answers? Who is responsible for the answer in such a case, especially so if it led to harm?  Are there mechanisms to prevent deliberate infiltration of the expert-crowd or other efforts to sabotage Jennifer (think of Tay AI)? 

More detailed points:
The Abstract refers to COQB, but the main article never does.

Section 2.4: "Answers to questions should be verified against reliable sources in the language of question."
--> That is a debatable statement: why would an English language source suddenly be necessarily inadequate when a person asks a question in French!?

Discussion: "Jennifer has successfully demonstrated that, with the right combination of technology and human experts, information from reputable sources can be more quickly and effectively organized and shared at scale."
--> This is a statement that is not supported by anything reported in the submission.

"Given the evolving tasks and the large number of volunteers with diverse background, putting the right process around tasks, workflow, and sequencing (Norheim-Hagtun and Meier, 2010) is key to ensuring efficient use of the volunteers’ time to the advantage of the project."
--> Unfortunately, full documentation of tasks, workflow, and sequencing is not provided in this manuscript, and use of volunteer time is not assessed.

Conclusions: 
"Jennifer leverages cutting-edge chatbot technology, as well as a diverse network of volunteers from around the globe, to combat misinformation related to the pandemic." 
--> Only one of these three claims is supported by the presented material.

At best, this submission could be included as a discussion article, certainly not a research article. Accepting it as a full peer-reviewed research contribution would lend it more credibility than it deserves.

Minor points, stylistic points:
"to best prevention and management practices." -->?? "to prevention and management best-practices."

Section 2.3: " following guidelines and recommendations in (National Academies of Sciences, Engineering, and Medicine, 2017)". The citation appears to point to a research agenda document, rather than a set of guidelines and recommendations.

The sentence "Much of the recent research focusing on automating the task of fact checking (e.g., Adair et al. (2017); Pathak and Srihari (2019))." is incomplete.

Several citations are incomplete re. sources, most notably the Jurkowitz (Mark, not Marl) citation. 

review_title: Engineering white-paper, but regrettably lacking in science

final_score: rating: 7: Good paper, accept
review_content: The authors describe a chatbot and a dataset of frequently asked questions regarding the SARS-CoV-2 disease. They enumerate the challenges in building an effective health chatbot. They introduce a dataset of SARS-CoV2 and COVID19 questions.

I would like to congratulate the authors on having an actual deployed system. I have many concerns about the paper and the datasets themselves that hopefully the authors can address.

Pros:
1. The authors collect what might be a high quality dataset harvested from many QA sources
2. They provide compelling examples of good bot behavior.
3. They provide a reasonable roadmap forward.
4. They are displaying great care in deployment of their system, particularly beyond their initial input domain.
5. I am a huge fan of the ongoing curation of the answers in their dataset.

Concerns:
1. The authors should provide a brief overview of the capabilities from Juji: Does the question/QA grouping come from Juji itself, or do the authors have some other approach? (From a brief skim of the Juji paper it seems that the authors must have taken some alternative approach)
2. There is no measure of question/QA grouping. Presumably this should be a very high precision grouping since incorrectly including a question could mean it has a wrong answer.
3. Similarly, there is no measure of inter-rater agreement on the evaluation of the answers in the crowd sourced context. A measure such as Krippendorf's alpha (or many others) would be appropriate here and essential to knowing how well the system performs overall. A measure like this becomes doubly important when the authors are attempting to fight the spread of misinformation.
4. How were the automatically generated question generated? If you used Juji templates, what are some examples? Given the constraints of available structured data (e.g. CDC data), what types of questions did you choose to answer, and which ones did you not? 
5. For the questions gathered from multiple sources, how are conflicts resolved? Presumably two sources do not always have consistent answers.
6. How many people evaluate each question/answer pair?
7. How is the translation performed? I can't tell if it's automatic with manual curation or entirely manual. If there was an automatic component, what system did this?
8. I am concerned about the use of "engagement" as a metric of success. This becomes even more concerning to me when more time spent interacting with a system is seen as a measure of its success - while I agree that when answering questions one might need further clarification or to know about related questions and answers - encouraging people to spend more time interacting with a system could lead to a perverse incentive where it keeps users "engaged" without answering their questions. 

The authors have built an impressive system, but I am reluctant to recommend it for acceptance without further details in data generation and evaluation. If these details were added (even for the system/questions/answers at a particular point in time), I would gladly recommend this paper for acceptance.
review_title: Impressive system, lacking crucial details

final_score: rating: 7: Good paper, accept
review_content: The paper provides a good description of the COVID-19 Q&A chatbot and the process of creating the crowd-sourced Q/A pairs that the bot uses for question answering. The authors also outline the opportunities and challenges of implementing COVID-19 chatbots in the real world.
While the study authors list examples of lessons learned as well as open challenges of using chatbots in a pandemic setting in which information changes quickly, it would be very helpful to better understand the use of a Jennifer-like chatbot in a situation in which the evidence-base is more stable. It would also be very helpful to understand the authors' perspectives on how chatbot technologies and their use could be effectively evaluated, and how such an evaluation could be better integrated into the chatbot development and validation of the provided responses. In addition, it might be helpful to provide examples of other chatbot technologies that have been used successfully during and prior to the pandemic, as this could put better into perspective the issues and imitations that are being raised in the manuscript. 
Another important aspect regarding the use of chatbot technologies is their acceptance both by public health authorities as well as the intended users. In using novel technologies, governments and public health organizations typically tend to take a more conservative approach, and accept them only following rigorous testing and evaluation. Equally, there may be also hesitancy from certain population groups with formed/strong opinions to engage with chatbots, particularly when provided answers originate from just one or several sources (i.e. when the whole body of evidence is not provided). It remains unclear who is (are) the intended target audience(s) of the chatbot and, ultimately, its objective. It would be very helpful to have that information stated and discussed up front. Lastly, it is important to recognize the significant limitations of the use of chatbot technologies for "fact-checking" in the absence of a robust knowledge-base. What has been presented in the paper may be more applicable to situation in which there is a need to help users identify and navigate through the relevant content, particularly when information is not well structured or ambiguous. It would also be great if the authors could reflect in the sustainability of initiatives that have a very high reliance on volunteers 
I would recommend that the manuscript be accepted, as it provides a good example on the potential usefulness of chatbot technologies in supporting the general public to efficiently find credible information on the web.  
review_title: N/A

Meta-review: This paper introduces a system which leverages a chatbot as a tool for public dissemination of health information related to COVID-19, and describes the use of human experts to craft responses to questions. 

However, the paper itself is somewhat ambiguous about the specific role of the chatbot and the NLP aspects of the system in "combatting misinformation". Specifically although there is significant work in the NLP community to leverage automated analysis methods for fact checking and flagging incorrect information, that is not what the system does -- ultimately it is the human curators and admins that do all of the combatting of misinformation, while the chat bot is simply the means for accessing and disseminating that information. From an NLP perspective, the contributions of the work are primarily the QA pairs that are produced (the data set), although we are not told specifically how those pairs are used, i.e. how are user questions mapped into questions in the Question bank (beyond "The  Dialog  Manager relies on pre-trained machine learning models" -- what models? relies how? how do they work?). 

From a broader application perspective, as the reviewers have identified, a nice example of how NLP can be combined with a public health activity, and points to the value of chat bots for supporting dissemination of information to the public, but there are unaddressed questions about the actual quality of the information that results from this process, and the acceptability of this sort of approach to public health officials.

In short, the system represents a potentially valuable public health contribution based on NLP and a nice example of crowd-sourcing, but the presentation in this manuscript does not sufficiently address the role of NLP in achieving the stated objectives.

We nevertheless think it is interesting to hear about this work in the context of the workshop and would therefore like to invite you to present your system at the workshop (10 minute presentation), and to provide a 1-page summary of the system and the data set for inclusion in the post-proceedings; instructions will follow after the workshop.
Decision: Accept (Abstract only)