Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper is about creating variations of COVID-19 drugs in Twitter using three different methods to capture posts with misspelled drug names.  

Comments:

The Title and the Abstract gives the impression that paper is about finding potential COVID treatments in social media. However, the research is about the use of automatic techniques for dealing with misspelled drug names in Twitter. 

The Number of English tweets used in this study and the duration of data collection needs to be mentioned. It is mentioned as 424 million, where only 67% is English. Its not stated how many of the tweets went through labelling process for COVID drugs.  If we only are targeting English tweets, then what is the point of labelling all 424 million tweets?

In the phrase “ For each term, each alphabet…” its more clear to use character/letter instead of alphabet.

Explanation of the third methodology seems to assume prior knowledge of the technology used. 

Text tagging and annotation is used interchangeably, and I suggest using just tagging as annotation could imply manual effort in examining and labelling the tweets.

Zinc is stated as being irrelevant whereas it is talked about as enhancing the clinical efficacy of chloroquine.
 
Table 1 is unclear as what the “total annotated terms” represents? Is it the number of tweets?  And what is delta in the last row showing? 

In Table 2 the total frequency is around 472,000 so where did the rest of 723,129 tweets with mentions of drug go? It also would be helpful to show the total number of misspelled drug names frequency.

In table 5, the calculation of percentage increase column needs clarification. The total of 222,428 is calculated as 15% of what number?

review_title: The paper is interesting, well presented, and concise with need for some corrections/clarifications.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper points out that spelling correction is an important step in preprocessing covid-19-related tweets and enables identification of additional data. The authors explore several approaches to correcting spelling mistakes in drug names using a corpus of covid-19 related tweets. The results suggest that several spelling correction techniques should be applied together to comprehensively address the problem.
At the same time, my feeling is that the title of the paper does not correspond to its content. The paper does not provide a characterisation of potential treatments but rather lists the most commonly mentioned drugs. There is no further characterisation of the context in which those drugs are mentioned. Moreover, both the title and the abstract put emphasis on using machine learning, however, only 1 method out 4 is machine learning-based.
Overall, the format of this study seems more appropriate for a short paper. For a long paper, I would expect the authors to further analyse why the implemented algorithms give different results, what potential errors might be introduced as well as show some validation of the approach.

Areas for improvement:

- When describing the second method used to generate misspellings the authors state “For each term, each alphabet is replaced with the closest alphabet on the QWERTY keyboard.” It is not clear what the authors mean by an “alphabet”, I am assuming it means a “letter”? How the closest letter on the keyboard is defined? Do variations include digits? Where is the recursion in this approach? I suggest the authors either cite a related research paper or provide a more detailed explanation.
- Table 2 shows the number of mentions of the top 10 drugs. It is unclear whether these numbers were calculated after applying spelling correction techniques or not.
- I suggest moving the clarification for the data reported in Table 3 from the bottom of p.3 to when the table is first mentioned.
- It would be great to see more analysis of why the overlap between the three methods is only 15%, ideally, with specific examples.
- Table 5 provides the numbers of additional terms identified using the described methods. How many additional tweets does it allow to recover? How can the authors ensure the additional data is identified correctly?
Minor comments:
- The abbreviation for the Social Media Mining Toolkit should be introduced in the main text. 
- It would be helpful if the authors could clarify whether the total number of annotated terms reported in Table 1 includes only drug-related terms or not? 

review_title: Interesting work but requires further analysis and validation

final_score: rating: 7: Good paper, accept
review_content: In this paper the authors mine the COVID-19 chatter tweets previously released to identify the usage of COVID-19 drugs related vocabulary. While the paper is interesting, well-written and concise, the authors over-sell some of its aspects. 

The paper is straightforward, yet interesting and I would say it would be easily understood by a wide audience. The number of experiments are adequate, the results are clearly presented and the discussion of each of the steps of the methodology makes it easy to follow. 

On the downside, starting by the title, I considering it misleading as the paper solely addresses the use of drug-related terms in tweets. The sole  use of COVID-19 drugs-related vocabulary is not an indicator of a possible treatment being followed by the author of a given tweet. The same idea is further reinforced in the conclusion, which refers to potential drug treatments available for COVID-19 patients. However, the experiments led by the authors only take into consideration drug vocabulary frequencies, so I fail to see how are these indicative for potential drug treatments. The work presented is of interest and clearly exposed and should be presented as it is: a COVID-19 drug terms frequency on Twitter chatter. The abstract and introduction also lead the reader into thinking a 424 million tweets corpus was mined and annotated, however the corpus size is considerably smaller (the clean corpus has approx. 100 million tweets, out of which only 67% are in English and were considered for this study). Other remarks: in Table 1, what is delta in the last row? Could the authors elaborate on the differences of the drugs found using with the 3 different spelling methods? I would expect the list of drugs to be more similar, but with different frequencies. 
review_title: Clear short paper on Twitter discourse mining for drug mentions discovery