Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes the construction of a website (system) that provides users with COVID-19-related information aggregated and translated from multiple reliable sources. Extensive use of crowdsourcing is done to check the reliability of the resources as well as to annotate them. The resources in languages other than English and Japanese are translated using a machine translation system called TexTra.

Reasons to accept: A large amount of human-annotated web documents is a valuable resource on its own. It is both costly and labor-consuming to build such a dataset.

Reason to reject: I am not sure if the evaluation of the topic classification is fair. A keyword-based baseline sounds too weak in comparison with a BERT-based neural network. A more sensible baseline would be an embedding-based similarity measure.
Also, even with the BERT-based topic classifier, the precision of some topics is very low. It would have been interesting to see if the inter-annotator agreement is also low for such topics. I think this is especially important because, in order to justify the validity of the dataset, the authors should have posted the reliability of the annotations among the annotators.
In addition, as the system makes heavy use of machine translation, a simple result table about the quality of the translation would be have been informative.

All in all, I acknowledge that the authors went to great lengths to construct a potentially useful dataset. However, its potential value of the dataset is not convincingly presented in the paper in its current state.

review_title: Large amount of COVID-19-related web-crawled data with human annotation. The evaluation and the usefulness of the topic classifier is less convincing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses an important topic. I visited the website and it certainly seems like a complicated and useful tool.

I would really like to accept this paper based on the magnitude of work that the authors have put into the pipeline and dataset, but the paper in its current form seems a bit lacking in analysis and evaluation. A couple of notes:

How is this paper beneficial to the NLP community? It certainly is a useful website for the general public, but it is unclear to me what I have learned and what ideas can be useful for NLP researchers or others working in the area.

What are the details for the topic classification? E.g., what is the distribution of topics and how were they annotated, and with what annotator agreement? Did some articles belong to other topics? What was the ground truth? It could be helpful to show a confusion matrix for the BERT classifier.

Is there any evaluation of the translated text?

The main area of improvement that I see in this paper is that there is not much analysis of such a large dataset. Some potential ideas:
Are the topics that are annotated with the articles the same or similar to articles that could be discovered by unsupervised clustering methods? Did they vary for the countries based on the number of cases at the time?
Any possible analysis on the distribution of n-grams?

If there is a more thorough analysis of the system and/or an argument for what can be learned from this paper or how this paper can be useful for the NLP community, I would be happy to change my rating. 

review_title: Important system but could benefit from more science and more rigorous evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper describes a system that aggregates news and official sources on Covid-19 and presents them with a web interface to the public, as a way of providing reliable, comprehensive and up-to-date information. The system is international, making extensive use of machine translation. It makes use of crowdsourcing in two places. The first use is to solicit recommendations for good information sources. The second is to gather training and test data on individual articles, asking about relevance to Covid-19, helpfulness, the quality of translation and membership of 9 topics. The system also makes use of NLP components. A classifier, based on BERT, classifies articles as to whether they are relevant to Covid-19, and assigns them to the 9 topics. Furthermore, they deploy a pre-existing machine-translation system.

Reasons to accept: The system being described potentially meets a need for citizens to have access to reliable, relevant and up-to-date information; the value of this paper largely depends on whether the system lives up to this promise. Also, the system is of some interest to the NLP community. The article classifier appears to get good results for relevance-to-Covid-19 - an F score on 0.84, substantially above a keyword-based baseline. The list of reliable information sources that they have collected is 
Furthermore it is often interesting to hear about the contexts into which NLP systems are to be deployed.

Reasons to reject:  In general this paper suffers from a lack of evaluation, or even thought about evaluation, except for the NLP components (the evaluation of the classifiers is OK and the machine translation is done using a published system). The introduction sets out a rationale for building such a system, but does not present any evidence that such a system would be taken up by the public, or result in them being better informed that without it - the benefits are simply asserted. The conclusion is very brief and makes no mention of how and even whether the impact of the system will be studied. I also have concerns about the reliability of the crowdsourced annotations. The system collects information from 10 people per article, yet no mention is made of how often the annotators agree. Relatedly, the F scores for classification into 9 topics are a lot lower than for the relevance-to-Covid-19 classifier. It is not clear whether this is caused by the classifier not being so good at the task, or whether the low scores reflect a task that is hard for humans to do consistenly. Some discussion of inter-annotator agreement would be helpful here.

Conclusion: This paper describes a system that has a lot of promise, but provides very little information about how well it lives up to its promise. As such I cannot recommend that the paper is published in its current state.
review_title: Potentially beneficial system, but with little evaluation.