Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Core review:**

This paper considers the hypothesis that the impact of COVID-19 in public mental health (specifically in individuals with mental health issues) can be measured by analyzing the users' behaviour in online forums related to mental health. Specifically, the authors focus on 3 mental-health-related forums in the Reddit social network, measuring the increase or decrease in both the user activities as well as in the relative prevalence of certain keywords of interest. Authors apply time series analysis to evidence a significant difference in user's activity post-COVID with respect to the behaviour predicted by autoregressive models. Furthermore, they analyze the relative change of word usage in certain categories, using LIWC clusters and an LDA topic model. 

The main findings indicate that these forums indeed exhibit abnormal behaviour, i.e., it does show an increased activity in these forums that coincides in time and textual content with the COVID-19 pandemic. However, two of the three forums analyzed actually exhibit a decrease in user activity, an issue which is noted by the authors but there is no attempt to explain it. I also consider that the techniques and analysis applied are an initial approximation, but they lack the sophistication expected in a top-level NLP venue. Specifically, the paper relies on descriptive statistics (word counting of LIWC clusters) and topic modelling techniques (LDA) which fail to account for complex linguistic phenomena that may be acting as confounding factors. Is my understanding that the fact that certain keywords show an increased prevalence does not necessarily imply that users are actually feeling an increased mental-health burden. Reaching this conclusion would require a deeper semantic analysis of the text.

As such, I believe the paper establishes a clear and important hypothesis, provides initial evidence for it, and it is clearly written, but the contributions are not sufficient to demonstrate an actual increase in mental health issues in users of these platforms as opposed to an increase in the general discussion of topics related to the COVID-19 pandemic. I commend the authors for their efforts in this clearly important endeavour and I suggest they continue strengthening this analysis.

**Reasons to accept:** The paper is clearly written, the methodology is thoroughly explained and carried out and the problem under discussion is relevant to the workshop.

**Reasons to reject:** I believe the paper does not advance our understanding of the scientific problem under study, that is, the impact of COVID-19 in public mental health, as it does not provide any new insight or result in this matter.
review_title: An interesting hypothesis in an important problem, but with a superficial analysis that does not provide a novel understanding of such problem.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes the analysis of messages published on Reddit under a set of specific subforums related with mental health topics. The authors have performed several analysis to understand the levels of preoccupation of the population on these forums with respect to the pandemia and how the people that use these forums rise. On the other hand a second analysis about the language and topics of discussion has been performed. The topic addressed is interesting and the technical background is solid. My main concern is refered to several points where some conclusions were arisen by the authors with respect the implications of the results obtained. Since the authors didn't present any analysis of the contextualization of the topics or, for example, some sentiment analysis performed over the texts some of these conclusions can be false. In this context, I recommend to check the paper and consider rewriting some of these parts.
review_title: Interesting paper; some conclusions need to be reviewed

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: #### Recommendation
Paper has room for improvement, but makes clear additions to the field.

#### Summary
**Strengths**
1. Comparison between anxiety, depression, suicide is very relevant
2. Methods are well described
3. Use of LIWC to supplement topic-modeling was a good decision
**Weaknesses**
1. Traditional **Discussion** section is intermixed with the section 5: **Findings**
2. An embedded topic model (Dieng, Ruiz, and Blei 2019) offers benefits over LDA 
3. Discussion of LIWC and depression/anxiety prediction for NLP is not particularly thorough

#### Review
The authors examine how the content of three subreddits related to mental-health help seeking changes as a result of the COVID-19 pandemic. To do so, they combine established methods for topic analysis (LDA) and psychological assessment of text (LIWC) with a time-series prediction approach (Prophet). Their analysis contributes to the field and their approach is satisfactorily documented and executed; however, the authors could improve their work with better organization and use of more current methods.

The authors comparison of anxiety and depression is an important contribution. USC's Understanding America Survey COVID-19 is doing this as well. Useful to show that NLP can be used to the same end. The authors longitudinal analysis of established NLP techniques, LIWC and LDA, usefully allows them to identify topic-changes across the various subreddits.

The paper could be made more accessible through a more typical distinction between analysis and discussion. As it is written, the authors analysis and discuss implications simultaneously--leading to temporary uncertainty over what is an evidence-backed finding and what is author speculation.

The authors could also update their methodological choices, e.g., using [Embedding Topic Modeling](https://arxiv.org/pdf/1907.04907.pdf)(Dieng, Ruiz, and Blei 2019) in place of LDA. fastText embeddings would be a natural choice for this, as the authors have already cited Wolohan, 2020.

Lastly, the authors could also discuss more thoroughly the history of LIWC and word-count approaches in NLP for depression / mental health detection. Rude et al. (2004) is cited but only as a possible explanation for a finding. That paper is a significant forebearer to this work.

#### Reproducability
This paper is highly reproducible, though associated code or Reddit post-IDs would help.

review_title: Clear contribution with opportunities for improvement