Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents an approach to detecting COVID-19-related symptoms in social media data (Twitter). The authors use word embeddings generated by BERT and build a graph of tokens based on their similarity to the learnt context embedding which allows them to iteratively identify more and more symptoms. The paper would benefit from some technical clarifications and, in my mind, can be extended to the long format. 

Strengths:
The study presents an original methodology. Since the method is unsupervised the authors validate its performance by manually calculating precision for a few identified symptoms. They also test the method on a previously developed and annotated dataset for adverse drug reaction detection and show that the method generalises quite well. 

Areas for improvement: 
- To improve the reproducibility of the study, the details about BERT implementation should be included.
- The process of graph generation is not straightforward and should be accompanied by examples and/or illustrations. 
- It is unclear what the similarity scores reported in table 1 represent for manual and graph-based approaches. Are these the cosine similarities between a given token and the seed word “cough”?
- I am not sure what this notation means “##raine”?
- Legends should be added to Figures 2 and 3 to show how node colours correspond to similarity values. 
- For the annotated ADR dataset, is it possible to calculate other metrics (e.g. recall, f1, AUC)?
review_title: A well written concise study presenting an original methodology

final_score: rating: 5: Marginally below acceptance threshold
review_content: The author represented an unsupervised graph-based approach for the detection of symptoms of COVID-19. The paper and idea is good. 

Pros 
1-The propose method will have several useful applications if it can be generalized and better than traditional methods.

2- The author test their method on 2 different dataset 

Cons 
1-The methodology needs further explanation 
2- There is no comparison with traditional method that are able to capture such symptoms for instance LDA , W2V approaches. Sometime a simple method works better than complex one , particularly when the language context is simple 

review_title: Good paper and needs further improvement and comparisons 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work reports an approach to symptom detection on Twitter data. Graph-based relations between a seed word and the more similar words are obtained by means of a similarity measure; the context embedding value is used to represent each word and compute the cosine similarity value with regard to the seed word. The authors applied their method to a collection of COVID-19-related tweets (using the word "cough"), and also conducted their experiment on an ADR corpus (Sarker & González 2015), using the term "pain". Their method obtained similar symtom words in both cases and yielded promising results that could be pursued in other pharmacovigilance tasks.

Strengths:
- I found this is an original method that will interest the audience of the workshop.
- The article is clear, although sone methodological aspects need further clarification and some excerpts minor need revision. 

Weaknesses:
- As authors already point out, the method currently detects only single-word terms, not multi-words, which may be essential to detect fine-grained description of symptoms (e.g. "dry cough" vs "wet cough"). 
- The experiment is only conducted using one word ("cough"); it could provide more solid results if more terms related to the COVID-19 symptomatology had been explored.
- I also missed a pragmatic perspective and brief discussion as to how these types of approaches would be valuable in the real medical-use or pharmacovigilance context. Healthcare professionals could validate these lists of medical terms. This could yield a quality reference dataset, and vice versa, healthcare professionals could get valuable, unknown data about symptoms to take into consideration (and probably not reported to date). The authors could develop this aspect. Namely, developing an interface to explore and show these associated terms would be a great contribution of this work.

The first column in Table 1---"Cough (Manual)"---needs more explanation. Do authors mean that they selected manually a set of words related to "cough", then computed the cosine similarity value? Please, provide more details in the running text.

In Section 2.1, authors have to provide more details about the BERT model they used: e.g. BERT cased or uncase? Did authors use the models released in the paper by Devlin et al. 2019? Did they resort to domain embeddings such as SciBERT or BioBERT? Or did they train their own model? What were the hyperparameters used (learning rate, dimension of hidden state, use of dropout, batch size...)?

- Others (grammar, style...):

    P. 1, Introduction: "SARS-nCOV" -> "SARS-CoV-2" or "SARS-nCoV-2"

    P. 1, Introduction: "pharmacovigilence" -> "pharmacovigilance"

    P. 2, Sect. 2.2: "didn't" -> "did not". Likewise in p. 3. Sect. 2.3: "doesn't" -> "does not"

    P. 2, Fig. 1: colors are not distinguished when printed in black and white; authors should think of an alternative to improve this.

    P. 3, Figure 3, and p. 4, Figure 4: These figures are, in my opinion, small to be read adequately. I suggest authors to make them larger and move one or two to an Appendix. 


review_title: Original method that deserves to be pursued