Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a multilingual dataset for detecting COVID-19 specific intents and a Spanish test set for code-switching. The strength of this paper is that they introduced a dataset to the community, provided several baseline models in the experiments, and showed that using cross-lingual representation is useful in this task. However, there are some major problems about the dataset which the authors need to elaborate more on. 

Pros:   
* The paper presents to the community a public multilingual dataset for detecting covid-19 specific intents in user utterances.  
* The paper shows that cross-lingual representation could improve the baseline models in this task.  
* The paper is well written and provides relatively comprehensive comparison between baseline models. It also provides analysis about the performance of these models. 
   
Cons:  
* The dataset is synthetically created by annotators based on an ontology, but there is no description about the ontology. Furthermore, there is no discussion about how close this synthetic dataset is to real data. I wonder if the synthetic data could reflect the real world problem or it is just a much more simplified problem with utterances expressed in less diverse forms.  
* The authors did not mention how and why the 16 intents are chosen, and also lack the definition of these intents. Is the formulation of these intents based on the real data or the synthetic data? Do most of the real data fall into these intent categories?  
*  The authors need to elaborate more on the details of the annotation process, including the number of annotators to annotate each utterance and the inter-annotator agreement among annotators. How did they deal with a sample that received different annotation labels ( majority vote? adjudication among annotators? just throw them away? )? These details are important for other researchers to assess the quality of the dataset.  

    In general the authors need to provide more details about the dataset. Otherwise it is hard to assess the quality of the dataset and makes it less useful to the community.

Some suggestions:  
* I would suggest the authors give some significant test statistics when they claim the performance scores are significantly better.   
* It would be interesting to see the breakdown of scores(Precision, Recall, F1) for each intent category and some analysis of the difficulty/easiness to identify each category. Showing cases with correct/incorrect predictions could also be helpful. 
    
review_title: useful task but needs more clarification on the dataset

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work presents a multilingual dataset for intent detection of COVID-related utterances. Besides, the authors built various baselines and experimented with code-switched data (Spanglish). 

**Pros**
- The dataset is open-sourced and could be helpful for further COVID-related development.
- Reasonable analysis and discussions on baseline results.
- Hyperparameters of models were provided.

**Cons**
- The dataset `is synthetically created by annotators based on an ontology describing all intents with few representative examples`. However, no details on how synthetically created, the information about annotators, the ontology and how it was created, and the seed examples were provided. There is also no discussion on how close the dataset is to real utterances. These issues make the dataset, i.e., the main contribution, less useful. 


**Misc**
- For Table 3, it might be better to have a plot instead of a table to show the trend when increasing % training.

- The authors should discuss why zero-shot performance with XLM-R Large is better than using 10% Spanish data, and why it performs worse than the base model under the setting of 80% French data.

review_title: Interesting dataset and reasonable baselines

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a multilingual dataset of 6871 utterances across 4 language + a mixture of English and Spanish. The dataset itself is an interesting contribution for the research community, which the authors complement with the study of different cross-lingual transfer learning techniques. I consider the paper to be appealing for the audience of the workshop, however it is not clear whether it is a short or a long one. I am considering it a short one, although the authors use the "annex escape" to complement important aspects of the paper. 

In terms of methodology, could the authors elaborate more on the choice of the 16 intents? What was the basis for chosing them over others?Also, I would have liked to have more details on the ontology they mention in the beginning of Section 2. The C Section of the Appendix is a bonus point in terms of reproducibility of the results. However, I do not understand Table 6 in Section B of the same Appendix. Why was German considered less relevant than the results in Spanish and French (Table 3). Table 5 is quite explanatory and an important part of the paper. However, I would ask for the authors to elaborate more on the distinction between some of the intent categories, that seem quite similar. For example, "I traveled to new york recently" could also be understood as "travel".
review_title: Interesting contribution, needs minor clarifications