Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper describes a novel dataset containing answers to a survey that asked participants to describe in short and long text their emotional states and rate their levels of  anger, anxiety, desire, disgust, fear, happiness, relaxation, and sadness. The dataset includes responses from 2500 UK residents collected during a critical period of the cover-19 pandemic.

The paper also investigated the predictive performance of lexicon-based and supervised classifiers. Preliminary analyses of the data showed significant correlations between the respondents emotional ratings and the scores inferred by LIWC for the same emotions. However, these correlations are much higher for longer texts, which is not surprising since LIWC was not created for microblog style content and longer text has more words that can match with the lexicon. Similar trends were observed with a supervised model trained to predict the emotional ratings given the text answers. Finally, topic analysis of the data showed that the main topics expressed in shorter text are different than in longer text. 

The paper is well written, well scoped and well executed. The dataset is very interesting and it will be useful for the NLP community not only from the standpoint of understanding how people respond to global crises but also to better understand how the characteristics of social media might influence what kinds of information people decide to share. The cautionary tale about using Twitter to study the public's reactions to these kinds of events is worth a closer look. I am not sure how much we can extrapolate from this, given that in real life people can post more than one tweet about a subject, and they do so spontaneously which is different from being asked to post about something specific on a survey.  Some "experimenter effect” might be at play here. 
review_title: Valuable resource

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The focus of this manuscript is to 1) describe a corpus of long and short lengthened emotional responses to the covid19 pandemic by the public, 2) identify meaningful associations between linguistic measures and emotional responses, and 3) develop prediction models for automatically classifying unseen texts according to their emotional response categories.

Quality: The usage of LIWC lexicon to identify topical/linguistic information is a good start. I'm interested in how differently tools like empath (https://github.com/Ejhfast/empath-client) would perform in identifying pre-configured and new topics on this corpus and what additional insights could be drawn from it. 

The correlation values appear low, but I'm wondering if that's due to out of vocabulary terms. Perhaps, a quick manual review of LIWC term coverage and a lexicon enrichment might help identify a stronger signal. 

Some emotions seem a little hard to tease out or have close relationships e.g., worry, anxiety, fear, etc. It would be interesting to understand commonalities and distinguishing characteristics in terms of linguistic measures of related/close categories. 

Clarity: A few points should be clarified or explained: 1) did you collect any additional meta-data about the participants e.g., sex, gender, age, race, professions (essential vs. non-essential workers), etc. that could be useful to contextualize or identify particular worries among groups? 2) did these "texts" also include emoticons that could be used to convey emotional response and topical information? 

It would also be interesting to sample more than 2 days. I wonder how the topics will shift as the pandemic unfolds. 

Also, I would recommend revisiting the corpus title as its very UK-centric and covers a broad range of emotions. What about "UK COVID19 Public Emotional Response dataset"?

Originality: The coupling of the open survey with traditional linguistic and topic modeling approaches for examining the global threat has some originality. The predictive model serves as an initial baseline. It would be interesting to evaluate other traditional machine learning classifiers to establish reasonable baseline performance.  

Significance: The topic is certainly significant and a nice first attempt at obtaining self-reported emotional concerns of the public. This is also a great tool that if deployed more broadly could capture insights across regions, countries, and continents.

Thank you for making this invaluable resource publicly available to researchers!
review_title: Good first steps toward understanding the UK public's emotional response toward covid19

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The paper introduces a self-reported dataset of 2500 short and long (each) texts about emotions of people during the COVID-19 pandemic. The paper then analyses the dataset in terms of LIWC properties, output of a topic model (most probable topics) and a linear regression model to predict emotion based on TFIDF and POS-based features. The dataset will certainly be useful. The fact that it is self-reported and has both short and long texts are noteworthy.

Suggestions:
1) Short and long texts by the same participant do not necessarily have to be 'parallel'/'analogous', it seems. If this is indeed the case, I would suggest mentioning so.

2) It would be good to know the reason behind picking worry as the key emotion. (In other words, the choice of calling this a 'worry dataset' and not an 'emotion dataset' is not clear). The question asked to the participants (when they draft their text) does not mention 'worry' explicitly. They are asked to rate how worried they feel. However, in addition, the participants are also asked to record other emotions.

3a) Section 2.2: The description accompanying worry clubs it with 'anxiety'. However, table 1 shows that only 55% participants reported anxiety.

3b) Please elaborate "The participants’ self-reported ability to express their feelings in the long text". Was it a part of the form?

4) It is not clear if the number of tokens is the same as vocabulary. It would be useful to know the vocabulary sizes of the datasets.

5) The github repo also includes LIWC statistics of the texts. This could also potentially be useful.

6) There is a low correlation between the 'concerns' (work, money, death?) and worry. In contrast, the top topics from the model include job and family. Is it surprising?

7) It would be useful to add statistics on how frequently the participants reported using Twitter. This would be helpful to understand the quality of the short text.

8) Observation: The classifier is not state-of-the-art. It would be useful to add citations to papers which use linear regression for emotion analysis.

9) Was the linear regression model also trained using spaCy?

10) Is the low MAE for worry related to the fact that worry was central to the annotation? Could there have been a bias in the annotations? The stdev for worry is also the lowest (as shown in Table 1).

The dataset would certainly be useful for future work. The analysis of the dataset (LIWC, topic models) is very interesting. The paper is easy to follow as well.

review_title: Useful dataset

Meta-review: Thank you for your submission!

We are very pleased to accept your paper for inclusion in the proceedings. 

Both first submission and first acceptance to the Workshop!

Looking forward to having it presented. (Details to follow.)
Decision: Accept