Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a dataset of 1700 questions related to covid which
are hand labeled and divided into 15 general categories and 207
clusters of semantic equivalence. The value is potentially useful for
general question classification, for semantic similarity, with
particular application to reducing load on question answerers by
removing redundancy, The data set is on the small side and some
important details about how the data was collection are omitted. The
authors make a number of factual errors. All of these could be easily
corrected and the data set is a useful resource.


"we scraped questions about covid". how is a 'question about covid'
determined? Are keywords used? If so what keywords? Additionally how
were questions that had location/time-specific versions vs. questions
with only one version determined? There are number of ways this could
be done, some noisier than others, some more scalable than others.

What are all the 'synonymous ways of saying COVID?'

same answer -> same question cluster. "In which country is SARS-CoV-2
believed to have originated" and "Which country manufactures the most
face masks" have the same answer but are not the same question nor are
even related. Plenty of useful questions do not have an answer (yet);
how are these to be clustered?

"there are fewer ways to ask how long COVID will last than ways to
write a positive movie review" -- I would argue both are countably
infinite. 

The cluster classification task was oddly formed. This is ultimately a
sentence similarity task or a coreference/nil clustering task. One
could use the data to pose the binary question "are these two
sentences asking the same question?" One could also posit the far more
useful "is this a question that has already been asked [and if so
which one] or is this a novel question?" task of coref/nil
clustering. Static assignment to clusters seems wrong for that kind of
data. By specifically excluding clusters with a small number of
questions you specifically skirt the issue you would have to deal with
in real application of this data.




review_title: timely contribution, could be better positioned with regard to previous work

final_score: rating: 7: Good paper, accept
review_content: The paper presents a dataset of questions related to COVID-19; included are annotations mapping questions to specific categories and classes.

It's always good to see papers that contribute datasets to the research community, particularly if they require significant efforts in annotation as this one does.

I think that some changes are needed in terms of positioning of this paper to make clear the contributions and benefits. The other reviewer, I believe, also struggled a little with this. I suggest an explicit "who would use this data" and "how would this dataset be used" section. I think limiting it to just people who want to train an QA IR system is a mistake. First, not all question have answers (Table 1); second, it's not obvious how the classes and categories focus, which are a large part of the paper, is core to training a QA IR system. Instead, I think the classes and categories can be used a lot more in the query intent / query understanding area of research. Pitching the paper more broadly - and being explicit on areas - would help to clarify and strengthen the contributions.

On the methodology of annotation:
- The paper rests on the technical soundness of the manual annotation process so the authors should really focus on ensuring the methodology / rigour is clear - especially as all the annotation work was done by the authors themselves. 
- Categories were made up by the author. How subjective and context specific was this? What are the limitations?
- Seems like one question is mapped to one class and one category - what about overlapping questions? Surely there are questions that fall in multiple - and if there are not in this dataset then there will certainly be out in the wild. The paper, at leasts, needs to address this.
- Categories may remain static (i.e., new questions about covid-19 would likely map to existing categories). But classes will be ever evolving. For example, if a vaccine is developed then a whole host of new question classes will arise. Having these static list of classes, at this snapshot in time, risks them becoming out-of-date very quickly in a rapidly changing environment.
- Figure 2 - questions/class highly skewed. So is it actually worth having the classes? How valuable are they?
- Having an explicit limitation section at the end of the paper will really help with the above points.


Some changes in presentation could really elevate the paper:
- The definition and difference between categories and classes was not clear at the beginning of the paper. In the middle section 2, the definition finally becomes clear. A definition of both is really needed earlier in the intro. 
- I found “classes” not really the best term. What about question “topics” or even question “clusters” since they are essential groups of duplicate questions. When classes are introduced, there needs to be a better justification for their purpose. It comes clearer much later but I think is needed earlier.
- Matched vs. unmatched is confusing. It’s not that they were not matched, it’s just that they were classes with only one question, right? Can a better term found? Also Table 1 presents matched/unmatched without any reference / definition - only much later in the paper do you find out.
- At some point, the paper suddenly starts talking about “labels” - it’s not clear whether this is classes or categories or both - why not just use the actual terms.

On the classifier / experiments
- The contribution of the paper is the dataset and not the ML methods. As stated, in the response to the review, the performance is merely an indication of how well current methods do on this task. The fact that the performance is good or bad does not really change the papers worth. However, insights from the experiments are part of the contribution; and these could be improved: e.g., were certain categories much harder than others? where two categories often confused for each other? 

Overall, this paper provided a good dataset for the community. But it needs to be much more explicit and clear about how people might use the dataset rather than a generic reference to training QA IR system. Addressing the presentation issues and directly addressing a number of limitations will greatly improve the paper.

review_title: A good dataset paper - some clarification and statement of limitation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a corpus of 1690 documents about COVID-19 manually annotated with questions classes, along with basic question classification algorithms.

Pros
- in scope for the conference
- dataset is publicly available
- technical details are kept to a minimum
- accessible to a large audience

Cons
- no clear statement of objectives
- no clear statement of specific contribution
- no explicit annotation guidelines (making it difficult to reproduce this effort)
- ad hoc categories (not modeled after general types of questions)
- no justification for the selection of documents
- mediocre performance of the basic question classification algorithms
- no discussion (the so-called discussion section is merely a summary)
- the claim that this dataset can help train QA systems is unwarranted at this stage

Overall
The significance of this preliminary work is extremely limited

Other comments
- the presentation of figure 2 is unnecessarily confusing
review_title: Preliminary work, limited significance

Meta-review: Formally recording the decision on the revision previously communicated.
Decision: Accept