Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the correlation between Covid19 cases (as reported by JHU) and relevant tweets, on a per-country based granularity.
The results are remarkable, showing a very strong correlation (Table 2) for a model trained on Italian data. Moreover, using modern multi-lingual embeddings, the trained regression carries over to other languages; although the correlation there is very variable (and sometimes negative).
The results are intriguing and show once more how expressions on social media reflect physical reality. 

I have however two major concerns:

1/ Why is this useful?
Not in the sense of what could public health officials do with such models (although I have no clue what they could do). It seems to me (and please correct me if I am wrong) that you are using the tweets of day D to predict number of cases of day D. If that is the case, then the causality is the other way round (cases => tweets). It would be much more interesting to try to predict number of cases at day D+1, or even better at day D+n.
 
2/ Reproducibility 
There are lots of things that are not clear. Those are not fundamental issues, but I cannot recommend acceptance in the current state, as I could not understand exactly what the authors did, much less would I be able to reproduce it
Filtering: 
  - why do you remove tweets with hyperlinks?
 - "we remove [...] tweets discussing countries other than Italy". What does this mean?
 - "We further filter Italy’s tweets for a balanced representation of tweet embeddings." What does this mean?
Model:  what exactly is a model? Is it the embedding + the total frequency of the selected words? Could you enumerate all the "models" (feature set) you used and - maybe in the appendix - the values for all of them?
Evaluation: Why do you not report mean error? The correlation itself is not very actionable, the predicted value would be

Also, there is a confusion between language and country, which is conflated. The given argument is that the chosen languages have a majority country, but for rigourousity I would recommend replacing country everywhere by language as it is misleading.

Fig 1 seems to be done with the number of tweets before the filtering of Sect 2.5. Why?

Finally, I am not sure if I understand how the transfer is done. Is it correct to say that you use the same regression and provide as feature the Japanese/Indonesian/etc tweets after putting them through the multi-lingual embedding?


review_title: Interesting correlation, but not actionable and not clear

final_score: rating: 5: Marginally below acceptance threshold
review_content: The following paper hypothesizes that tweets can be used to model COVID-19 outbreaks in across countries. The paper does this by using cross-lingual sentence embeddings from mBERT and LASER to predict case-counts.

Although this is a good idea and Figure 1 is particularly compelling, I find that the paper is at best a work in progress and at worst intentionally misleading. I'd recommend substantially more work before it is ready for publication. 

My biggest concern is that the way the tweets are filtered and the time-chunks are specified, I have no reason to believe that this model is using tweets to actually predict an outbreak, as the authors imply (NB the authors are careful to say they are "aligning", not "predicting", although I think this intentionality is rather muddy). It seems to me that they are simply capturing the response to an outbreak already occurring -- I'm unclear what the value of this is to social scientists and policymakers.

The authors explicitly filter for words such as "lockdown", "quarantine", "social distancing", "epidemic", and "outbreak", to perform their alignment, which to me seem to describe tweet-responses to policy rather than tweet-responses to sickness. I think the authors' hypothesis would be better served by choosing words, topics or other indicators that are more personal and health-related -- words like "fever", "cough" or "lack of smell" -- to avoid such confounding.

Further, the authors utilized fixed time-periods to explore their regression, which is a little confusing to me. I think the authors should use date-ranges in each country relative to the kth case in that country. Since their date-ranges are so wide, I worry that the date-ranges are simply capturing the full policy response of a government that has already predicted an outbreak.

Why is spearman's correlation the only metric used? If the authors are conducting a regression experiment, there are other more compelling and interpretable metrics. Why aren't significance-values included for the correlation?

Why do the authors say that they observe "right-skewed Gaussians"? This is clearly more of a point process (i.e. a Hawkes process.)

Additionally, I would urge a more serious consideration of other confounders as well. I'm not sure what the best ones to use would be, but I think some policy analysis is necessary -- when did each country actually institute a lockdown? Do lockdowns effect overall non-COVID tweet-volume as well? I'm sure there are confounders in the literature.

If this is not possible, then I urge a reframing of the paper. The authors need to be clearer about what they are actually purporting to do, and not hide behind words like "align".
review_title: Good idea but some methodological concerns

final_score: rating: 7: Good paper, accept
review_content: # [REVIEW] Cross-lingual Transfer Learning for COVID-19 Outbreak Alignment

10th June 2020

## SUMMARY
This short paper describes work on cross-lingual transfer learning to track COVID-19 cases across Italy, Thailand, Japan, Turkey, and Indonesia (i.e. languages that are largely spoken in a single country).  Using case statistics derived from Johns Hopkins COVID dashboard, the researchers aim to determine (a) if the number of COVID-related tweets is associated with case numbers (good correlations were achieved for this), and (b) can this training be applied to other countries that may be at a slightly different point in their COVID trajectory i.e. cross-lingual transfer learning (a range of correlations with JHU data were achieved ranging from -.316 to 0.859, indicating that the approach has some utility — but see caveats below).  

It is difficulty for me to assess the quality of this paper relative to other submissions, but I suspect that this use of cross-lingual transfer learning is sufficiently interesting to justify a short paper.

## MAJOR COMMENTS
1.  There is a major methodological issue not related to the NLP and not something you can be reasonably expected to do anything about — but I think it’s important to point it out as a limitation:  The international ground truth data for COVID is  not very reliable regarding underlying prevalence/incidence of COVID-19.  Even in the US, there have been stark differences in approaches to testing, and the criteria for testing. International comparisons are even more difficult. This undermines the evaluation somewhat.

## MINOR COMMENTS [NITPICKING]
1.  [p1c1] “With globalization, it is intuitive that countries have followed earlier affected regions in patterns of outbreaks and measures to contain to them (Cuffe and Jeavans, 2020).” Consider rephrasing this to emphasise that it is the increased travel associated with globalization that is important, rather than globalisation per se.
2.  [p3c1] “Bert as service” probably requires a few words of  explanation (at least for this reviewer - I had to google it)
3.  [p3c2] “When measuring against new daily cases, the correlations are not as significant in time II”  suggest avoiding using the word significance here, unless you are referring to statistical significance.

review_title: Review of: Cross-lingual Transfer Learning for COVID-19 Outbreak Alignment

Meta-review: While the reviewers raise some methodological concerns, the methodology and results presented are interesting and worthy of dissemination, as well as being adequate for a short paper. It is my view that the concerns can be addressed in a revision of the paper (to be prepared for the final proceedings volume, after the workshop).

We look forward to a presentation of this work at the workshop on Thursday.
Decision: Accept (Abstract only)