Stay-at-home policy is a case of exception fallacy: an internet-based ecological study
Authors: R Kunst
G Pumi
J Dalzochio
R F Savaris
Journal: Scientific Reports (2021)
Discipline: Multidisciplinary
Keywords: mathematical model; COVID-19 transmission; second wave; Europe; staying at home

Abstract:  A recent mathematical model has suggested that staying at home did not play a dominant role in reducing COVID-19 transmission. The second wave of cases in Europe, in regions that were considered as COVID-19 controlled, may raise some concerns. Our objective was to assess the association between staying at home (%) and the reduction/increase in the number of deaths due to COVID-19 in several regions in the world. In this ecological study, data from www.google.com/covid19/mobility/ , ourworldindata.org and covid.saude.gov.br were combined. Countries with &gt; 100 deaths and with a Healthcare Access and Quality Index of ≥ 67 were included. Data were preprocessed and analyzed using the difference between number of deaths/million between 2 regions and the difference between the percentage of staying at home. The analysis was performed using linear regression with special attention to residual analysis. After preprocessing the data, 87 regions around the world were included, yielding 3741 pairwise comparisons for linear regression analysis. Only 63 (1.6%) comparisons were significant. With our results, we were not able to explain if COVID-19 mortality is reduced by staying at home in ~ 98% of the comparisons after epidemiological weeks 9 to 34. 

Retracted: Yes

PubPeer comments:
 https: twitter.com GidMK status 1369078568844595200 , 
 Dear Health Nerd Thank you very much for your comments. We agree with some of your comments. Residential areas provided by Google mobility report are in those that having Google apps in the mobile phone, thus, Android and IOS are included. Most of the studies use Google mobility reports. I cannot forsee another way to verify the compliance to lockdown. I suggest you may redo the analysis using working place instead of  residence . From the scientific point of view, we concluded that we were not able to explain if COVID-19 mortality is reduced by stay-at-home in ~98% of the comparisons after epidemiological weeks 9 to 34. The way lay people interpret the results is another issue. Finally, if proving the hypothesis of lockdown reduces COVID19 mortality is difficult, we agree that the efficacy of lockdown is not based on scientific data. Always a pleasure to discuss the data. , 
 Have you looked at the percentage change of cases and deaths VS google mobility trends? The correlation is more evident that way, and you don t have to compare different areas or countries where other variables might have an effect These are exponential (or logistic) growths processes, and the variable affected is the rate of growth percentage change, not the totals (even per million). For example, equal amount of lockdown applied 3 weeks early can result with a several-fold reduction in total cases and deaths, since the exponent was modified before the base had a chance to grow sufficiently. Here is a small analysis looking at the rate of growth VS % increase in stay at home time (google mobility trends): https: colab.research.google.com drive 1j6-oU5Ro1Gv_sfJGJLRauBM7r2y91WwK#scrollTo=68PgZVFh1P79 , 
 Dear Laticaudis Here is my [answer] . for some reason, I am not able to reply as an author. Ricardo F Savaris , 
 Dear Laticaudis Congratulations on your effort in doing your graphs. Your comments are interesting. The use of % of a change in population is like relative risk, it gives a precise finding on an imprecise base. If bases are different, I cannot compare different regions. For instance, my State in Brazil (RS) had an increase of 25% of death between epi-week 9 and 10 -2021 (from 945 to 1184 deaths; a difference of 239), while Uruguay had and an increase from 42 to 52 in the same period, an increase of 80%. Things would be more complicated if the base million were introduced. The non-stationary pattern of the disease does not allow us to make a statistical analysis of the data, using daily changes. This is a huge problem. I wonder what would happen if you remove the 6 and 22 days delay for the comparison. Besides, daily changes of  stay-at-home bring some concerns. If the % of people that stay at home is 40% for weeks, no changes will be displayed. The sharp drop in the graph will give the impression that the amount of restriction is reduced. Another issue is the use of new cases as a measure of the effects of the stringent policy. I work at the ER, patients are postponing visiting the ER because of the fear of getting the virus, and previous recommendations suggested to go to the ER only when advanced symptoms were present. Using a short time period to demonstrate the benefits of  stay-at-home policy may explain the positive results. The tricky part is the comparison. The slogan  Stay home, save lives has been the main message published in the lay media, scientific articles, and by many medical societies. Eventually, places, where COVID-19 cases were controlled, became the natural comparator. By April, 2020, Brazil was compared to Argentina ([Colombo, S, 2020] related to the efforts against COVID-19. The latter was praised for their stringent measures against COVID-19, while the former received criticisms. After a year, we can compare their deaths million curves. ![file] Finally, my co-authors and I believe that the most important part of our study is the way we presented the data, everything is publicly available since the pre-print publication, the methodology is new, and we have different levels of comparisons that may be used. Readers are able to compare major capitals or similar regions with different outcomes for COVID-19 (Tokyo- controlled- vs. Su00e3o Paulo city-not controlled-, P=0.7; Ireland-controlled- vs. Croatia- not controlled-, P=0.7; ). Finally, I would like to stress a phrase in our discussion,  It is unwise to try to explain a complex and multifactorial condition, with the inherent constant changes, using a single variable . Stay-at-home policy is different from social distancing. Hope our answer will satisfy you. Thank you very much for your comments, always a pleasure to do the scientific debate. R.F. Savaris, MD, MSc, PhD , 
 Dear authors, We have summarised some of our concerns about the manuscript in [the following preprint] to foster easier exchanges with you. We are looking forward to a response from you. On behalf of all the authors of the preprint, Lonni , 
 I d like to point out the following comments articles: - S. Schroen [highlights further problems with the analysis] - In 2020, Basellini et al. [found a statistically significant effect of mobility on mortality] (by using the different data and methods) - Carlos Gu00f3es: [Savaris et al (2021) erroneously interpreted their regressions] Raphael Wimmer (who apparently cannot register an account at PubPeer :( ) , 
 https: twitter.com GidMK status 1468349804992286720 , 
 Two  matters arising have been published by Nature Scientific Report on the topic now. The first one is available here: https: www.nature.com articles s41598-021-02096-3 The second one is available here: https: www.nature.com articles s41598-021-02461-2 They sum up some of the points that have been mentioned in this whole thread, but also add new one. The original article links to them and has a mention of the concerns around the findings. , 
 https: twitter.com GidMK status 1470552995091783683 , 
 Retraction Note: Stay-at-home policy is a case of exception fallacy: an internet-based ecological study https: www.nature.com articles s41598-021-03250-7 Retraction of: Scientific Reports https: doi.org 10.1038 s41598-021-84092-1, published online 05 March 2021 The Editors have retracted this Article. Following publication of this Article concerns have been raised about the methodological approach developed by the Authors to evaluate the impact of stay-at-home policies on the reduction of COVID-19-related fatalities. In particular, Meyerowitz-Katz et al.1 show that the approach fails to detect any signal when tested on a synthetic dataset where the ground truth is known, and under specific cases of data subsetting. In addition, Meyerowitz-Katz et al.1 failed to replicate the original results using a synthetic dataset. These suggest that the false negative rate of the approach is prohibitively high to allow for meaningful conclusions to be drawn regarding the impact of stay-at-home policies on COVID-19 fatality rates. The results of Meyerowitz-Katz et al.1 are further confirmed by Gu00f3es2 who, using a pure correlation analysis, shows that the coefficients for the impact of stay-at-home policies using the methodological approach developed by the Authors can be zero even with diametrically opposing indices of staying-at-home. Given these concerns, the Editors no longer have confidence that the conclusions presented are adequately supported. R.S. Savaris, G. Pumi, J. Dalzochio and R. Kunst do not agree with this retraction. ,