Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper explores gender differences in linguistic productions between two groups of Redditors who self-identify as either "male" or "female". It examines a corpus of covid-19 pandemic threads in relation to two areas: emotion analysis (employing a VAD lexicon and word embedding representations); and topic analysis (employing the tool MALLET).

The paper's novelty is in the application of an established method to a new corpus that the authors have developed pertaining to covid-19 threads. As expected, the language usage for covid-19 posts had a lower Valence when compared to language used in a baseline corpus. There is also a general trend for the language used in the female sub-corpus scoring slightly higher in the Valence scale than male sub-corpus. The trends are reversed when Arousal and Dominance were examined: overall higher for men, and when comparing baseline to the covid-19 posts, the baselines score slightly lower for both male and female data.

To compare and contrast the different topics covered between the male and female authored posts, topic modelling was applied to each sub-corpus and the topics with the highest coherence scores were presented. However, applying topic modelling to the corpus as a whole and analysing the topic allocation of the male and female posts would give a better indication of similarities and differences of the topics covered in the sub-corpora. However, two different topic models for each sub-corpus were developed and the most cohesive topics were presented.

In general the VAD study is interesting, although unsurprsing. The goal of discovering if different or similar topics were covered in the two sub-corpora may be best approached by discovering the topics covered by the corpus as a whole and analysing the topic allocation of the sub-corpora.

review_title: nice application to new data set to be made available

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to understand the difference between male and female discourse in social media looking at a manually annotated set of Reddit threads related to Covid-19 compared to a baseline set. They confirm existing results about male and female discourse on the VAD scale.

The paper is clear and well-written and seems to be an interesting analysis, but fails to provide the significance of the work. Further the only novelty of the work is the application to Covid-19, otherwise all methods are utilizing previous work. This is not to say the authors should re-invent the wheel. 


Pros:
- An interesting exploration of gender differences that confirms previous results.
- A good use of previous work on a new corpus. 

Cons:
- Missing the overall significance for researchers, clinicians, epidemiologists, etc.
- It is unclear why Reddit specifically is used. They mention it is the 19th most visited site world. What about the other ones that are more visited? Is Reddit truly representative of the population at large? A description of the basic characteristics of Reddit users and posts would be helpful. 
- There is also a large imbalance between male and female posts (2:1 ratio). 
- This is very heteronormative.
- The dataset is pulled from 15 weeks starting from Feb 1 to June 1, which was a rapidly changing time. The paper would benefit from a discussion of the different topics discussed over that time in comparison to the topics pulled out by the models. Currently we are in a new "normal" and I think that would be reflected during the different weeks.
- The baseline is pulled from the same time period of Covid-19. An explanation of why the baseline should be the same time frame would be helpful to understand why the baseline is not from before Covid-19 when males and females were posting "normal" stuff. 
- The overall results in table 1 are confusing in what is being compared and what is statistically significant. The difference between males and females for the VAD criteria may be statistically significant but it is a minor increase (< 0.2). It is unclear how important this is and what implications it has.
- A more in depth discussion on the relevance of the most coherent topics for males and females would be helpful. 
review_title: Overall the paper is okay but fails to provide the significance of the work.

final_score: rating: 7: Good paper, accept
review_content: Quality:
Overall the paper is well written, contains re-usable data, and describes clear results.

Clarity:
Authors aims of analysis were clearly stated, as were the methods employed. Results are elucidated clearly. The paper is well written, concise, and easy to follow logically.

Originality:
Given the findings corroborate already established patterns of F / M speech, the exact findings that those patterns persist in covid-related speech is not particularly original. However, within the context of studying phenomena amidst a completely novel world event, covid, the findings regarding how people talk about said event are original. Combination of methodologies to perform analysis are somewhat original.

Significance:
Mohammed's VAD showed low inter-annotator agreement for A & D types. This may reduce the impact of any findings, distinctions, or variances (even if statistically significant) between the genders in these categories. Even if statistically significant, the cohen-d effect sizes between F & M are still very small (< .2 in all categories). What is the _human significance_ (not mathematical significance) of the analyzed differences? 

Editing suggestions: 
  Clarify Fig1 caption by including "re covid" or something to that effect
  Typo in sentence, missing "of" :  "COVID-related data trends (Figure 2) show comparatively low scores for valence and high scores
for arousal in the early weeks [OF] our analysis (February to mid-March)". 
  This sentence comes off as sexist: "women tend to use more positive language, while men score higher on arousal and dominance." Use similar terms to describe characteristics for both genders instead of saying what women do and what men score, e.g, "Women score higher in use of positive language, while men score ..."

pros
  Straightforward, solid results that established F / M speech patterns persist in novel corpus.
  Probably a decent baseline paper to use in further research on gender differences re covid speech or other domains.
  
cons
  Statistical significance does not explain human importance of findings. 
  
review_title: Overall the paper is well written, contains re-usable data, and describes clear results.

Meta-review: Thank you for your submission.

The reviewers have identified that this is a well-written paper presenting interesting results over a new data set. While there are a number of issues raised, we are satisfied by the response from the authors and believe that the concerns can be addressed in a revision for the final proceedings (which we will provide instructions about after the workshop).

We therefore welcome your presentation at the workshop on Thursday (5:30-9:30pm PDT). Please plan on a 10-minute presentation, pre-recorded or live.

Sincere apologies for the lateness of the decision!
Decision: Accept