Assessing mandatory stay‐at‐home and business closure effects on the spread of COVID‐19
Authors: Jay Bhattacharya
John P A Ioannidis
Eran Bendavid
Christopher Oh
Journal: European Journal of Clinical Investigation (2021)
Discipline: CLINICAL MEDICINE
Keywords: NPIs; COVID-19; case growth; restrictive interventions; epidemic dynamics
Affiliations: Department of Medicine Stanford University Stanford CA USA, Center for Health Policy and the Center for Primary Care and Outcomes Research Stanford University Stanford CA USA, Department of Epidemiology and Population Health Stanford University Stanford CA USA, Department of Biomedical Data Science Stanford University Stanford CA USA, Department of Statistics Stanford University Stanford CA USA, Meta‐Research Innovation Center at Stanford (METRICS) Stanford University Stanford CA USA

Abstract:  Background and aims: The most restrictive nonpharmaceutical interventions (NPIs) for controlling the spread of COVID-19 are mandatory stay-at-home and business closures. Given the consequences of these policies, it is important to assess their effects. We evaluate the effects on epidemic case growth of more restrictive NPIs (mrNPIs), above and beyond those of less-restrictive NPIs (lrNPIs). Methods: We first estimate COVID-19 case growth in relation to any NPI implementation in subnational regions of 10 countries: England, France, Germany, Iran, Italy, Netherlands, Spain, South Korea, Sweden and the United States. Using first-difference models with fixed effects, we isolate the effects of mrNPIs by subtracting the combined effects of lrNPIs and epidemic dynamics from all NPIs. We use case growth in Sweden and South Korea, 2 countries that did not implement mandatory stay-at-home and business closures, as comparison countries for the other 8 countries (16 total comparisons). Results: Implementing any NPIs was associated with significant reductions in case growth in 9 out of 10 study countries, including South Korea and Sweden that implemented only lrNPIs (Spain had a nonsignificant effect). After subtracting the epidemic and lrNPI effects, we find no clear, significant beneficial effect of mrNPIs on case growth in any country. In France, for example, the effect of mrNPIs was +7% (95% CI: -5%-19%) when compared with Sweden and + 13% (-12%-38%) when compared with South Korea (positive means pro-contagion). The 95% confidence intervals excluded 30% declines in all 16 comparisons and 15% declines in 11/16 comparisons. Conclusions: While small benefits cannot be excluded, we do not find significant benefits on case growth of more restrictive NPIs. Similar reductions in case growth may be achievable with less-restrictive interventions. 

Retracted: No

PubPeer comments:
 https: twitter.com andrewbostom status 1348271129887191041 , 
 https: twitter.com GidMK status 1349164532627693570 , 
 https: twitter.com AndreasShrugged status 1349464781145731073 , 
 https: gidmk.medium.com do-lockdowns-work-879e93803d67 , 
 We read with interest the publication from the authors. It is an interesting read and definitely a relevant paper. We however would like to raise some concerns on the methodology and discussions limitations of the study provided by the authors. We will detail them below. 1 The sample size n=10 is rather small compared to previously published studies (e.g., Flaxman et al. that the authors mention as [15] or Brauner et al. [A] that the authors do not seem to mention). In [A] , 41 countries are used. These studies had somewhat different findings that the authors should acknowledge and discuss, in particular in light of their own contradicting findings. The authors also give inadequate reasoning for having chosen such a small pool of data. There are numerous countries for which case policy data by day is available at the administrative level (i.e. Switzerland by canton, Brazil by municipality and state, Canada by province, etc). These are all available publicly online. 2 The u2018control groupu2019 for countries not implementing lockdowns is composed of Sweden and South Korea. However, both countries, although they did not officially close businesses or ordered stay-at-home have implemented several other restrictions that we details below In Sweden, a ban on public gathering has been put in place and readjusted several time [A] some bars have been closed (see a couple of examples mentioned in [B]) and people have been recommended to work from home very early on (mid-March) [C]. While most schools stayed open in Sweden, high schools and universities were all using distance teaching [D]. Furthermore, the use of South Korea as a u2018controlu2019 is confusing. While the country did not explicitly impose what the authors term u201cmore restrictiveu201d interventions, schools in the country had one of the longest at-home learning periods in the world [E], officially lasting from March until late September (and resuming after a short in-person run). This is not correctly classified in the paper, and is an error that should be corrected. South Korea also, as the references cited by the authors show, has extraordinarily restrictive digital legislation implemented after the MERS outbreak that allows government tracking of individual location, credit card, and other information if they are an identified case. Whether or not this is a u201cmore restrictiveu201d intervention is entirely subjective, but we posit that such detailed government oversight for the reasons of contact tracing would not be welcome in countries such as the United States. Consequently, using these countries as a strict u201ccontrolu201d is misleading and inaccurate. This leads into a major error of the paper, which is dichotomizing responses into either u201clessu201d or u201cmoreu201d restrictive. This is an arbitrary designation decided on by the authors that is not substantiated in the paper. For example, school closures have been described by many as the most restrictive and or problematic government intervention (the phrase u201clast to close and first to openu201d is commonly used), but both of the u201clessu201d restrictive u201ccontrolsu201d closed (at least partially) schools for in-person teaching, one for far longer periods of time than many of the u201cmoreu201d restrictive u201ccasesu201d. 3 The analysis does not seem to take into account the timing of implementations unlike the study done by Brauner et al. [A]. Their findings are not surprising when considering timing. Brauner et al. [A] stated that:  In accordance with other studies that took this approach (2, 6), we found that issuing a stay-at-home order had a small effect when a country had already closed educational institutions, closed nonessential businesses, and banned gatherings. They also further add  Flaxman et al. (1) and Hsiang et al. (3) included the effect of several NPIs in the effectiveness of their stay-at-home order (or u201clockdownu201d) NPIs and accordingly found a large effect for this NPI. 4 Another major issue in this analysis is the complete lack of any attempt to correct for the relationship between interventions and case counts. While this is noted by the authors in the discussion of the paper, it is a fundamental problem that undermines the entire purpose of the study. It has previously been demonstrated that case counts and intervention adoption are closely interrelated, which is unsurprising - governments implement restrictions while cases are rising, not when they have plateaued or already started to fall. The current analysis is therefore geared towards finding a null effect, as in the vast majority of situations the case counts will continue to rise for some time after interventions are announced. There is also the question of implementation - in some cases, it is possible that behaviour changes occurred after interventions were announced, in others it may be the case that enforcement was needed after the initial date of implementation and thus the effect was instead delayed. These are merely two potential examples of situations that could have occurred, demonstrating the numerous sources of time-related bias that the authors have not attempted to control for in their analysis. 5 Given the small sample size, and lack of attempts to correct for confounders, the study is also at a very high risk of issues relating to unmeasured confounding. This has led to the problematic situation where, for example, Sweden and Iran are being directly compared in terms of the outcomes of their interventions without considering the plethora of cultural, social, and political differences between the countries that might impact their case counts during a pandemic. This is worrisome, because there are examples of other, much more culturally similar, nations which could have been included in either group, such as Denmark for which case counts by administrative region are publicly available online. While this would not have eliminated the potential for unmeasured confounding, it would have provided more support for the analysis - as it is, much of the study may simply be the result of meaningless comparisons. 6 Case data from these countries is clearly inadequate to generate any causal conclusions. The authors openly acknowledge this fact in the discussion, however the results and paper generally are riddled with statements implying a causal impact (or lack thereof) from these interventions. For example, the phrase u201ceffects ofu201d is used numerous times in the paper to refer to the case numbers after interventions are implemented, but this is incorrect - it is not the u201ceffectu201d of the intervention being measured, but the association between case counts after the intervention came into effect. It is well known that testing capacity was limited during the first wave in all of these countries, and as the authors acknowledge in their discussion this inherent flaw means that no causal conclusions can be drawn from this paper whatsoever. It is entirely possible that the lack of benefits highlighted in the discussion when comparing countries to Sweden, for example, is due to the fact that Sweden had incredibly restricted testing during the times that are assessed. We look forward to a response on these concerns by the authors. We would be happy to correct this review if there is something we missed or misunderstood from the paper. REFS: [A] https: science.sciencemag.org content early 2020 12 15 science.abd9338 [B] https: www.thelocal.se 20200427 stockholm-health-chiefs-close-five-bars-and-restaurants-over-social-distancing-breaches [C] https: www.krisinformation.se en news 2020 march the-public-health-agency-is-calling-for-working-remotely [D] Public Health Agency (2020, November 25). COVID-19 in children and Adolescents: A knowledge summary u2013 Version 2. Retrieved January 11, 2021, from https: www.folkhalsomyndigheten.se contentassets 1e5e09395b9a4f498ff635cdd2b1a888 covid-19-children-adolescents.pdf (p.6). [E] http: m.koreaherald.com amp view.php?ud=20200921000340 Lonni Besanu00e7on and Gideon Meyerowitz-Katz , 
 #6 We read the anonymous response (under the name of u201cContopus Cooperiu201d) that tackled two of the six points we mentioned in our post-publication comment. Concerning our point 2 about how the control group was made in the manuscript, Contopus Cooperi mentions that u201cAll these restrictions have been included in the model.u201d Our initial comment pertained to the fact that the authors of the paper constituted two groups: one group that implemented stay-at-home measures and one group that did not. In the paper we can directly read: u201cUsing first-difference models with fixed effects, we isolate the effects of mrNPIs by subtracting the combined effects of lrNPIs and epidemic dynamics from all NPIs. We use case growth in Sweden and South Korea, two countries that did not implement mandatory stay-at-home and business closures,u201d explaining that the authors have used 8 countries that they consider as having implemented more-restrictive measures and two countries (Sweden and South Korea) as having implemented less-restrictive measures. As we note in our initial comment: u201cThis leads into a major error of the paper, which is dichotomizing responses into either u201clessu201d or u201cmoreu201d restrictive. This is an arbitrary designation decided on by the authors that is not substantiated in the paper.u201d We therefore point to the fact that the categorization made by the authors is arbitrary and not substantiated and we remind our readers that South Korea, for instance, has had one of the longest (if not the longest) school closure in the world, which is considered by many as one of the most restrictive measures. As we mentioned in point 5, using South Korea for instance is also potentially dangerous with all the socio-demographics differences between western and asian countries that are therefore not accounted for. To conclude on this point from Contopus Cooperi, our initial comment therefore did not pertain to the inclusion or not of specific restrictive measures implemented in South Korea and Sweden in the model presented by the authors but rather on the arbitrary grouping of countries into two categories. Concerning our point 3 on the timing of NPI and their implementation, Contopus Cooperi responds that u201cTime was a parameter of the model.u201d We are slightly puzzled. Our comment clearly mentions u201ctimingu201d, which, although the two words bear some common linguistic roots, has a completely different meaning from u201ctimeu201d. We never suggested that time was not included in the model presented by the authors. We instead provided evidence that other previously-published models tried to account for the moment, duration, periodicity, and the order in which measures have been implemented. These parameters are essential to understand and analyse the impact of NPIs against COVID-19 [F,G]. While we are not claiming that the authors should successfully model these, we are however suggesting that not including them constitutes a serious limitation of the study that should be acknowledged. To reiterate on our other points (not mentioned by Contopus Cooperi), the presented study has obvious limitations which are not discussed and does not currently present contradicting findings from previously published studies to mitigate the authorsu2019 findings. In light of the current use of scientific literature and the way that the media and laypeople share studies without the potentially needed nuance to present their results, we argue that the manuscript would benefit from clearer discussions of the limitations and clarifications on the use of Sweden and South Korea as a control group. [F] https: www.thelancet.com journals lanpub article PIIS2468-2667(20)30073-6 fulltext [G] http: www.thelancet.com retrieve pii S246826672030133X Lonni Besanu00e7on and Gideon Meyerowitz-Katz , 
 I read with interest the publication from the authors. It is an interesting read and definitely a relevant paper. However, I believe there is a potential data error in this paper regarding the dataset from the Netherlands. Upon inspecting the data used for the analysis, I believe I stumbled upon some data errors that seem to be vital for the analysis. It concerns the data for the Dutch covid cases. Upon inspection of the data and scripts provided by the authors, I found that the dataset u2018rivm_NL_covid19_province.csvu2019 (in covid_npi_analysis data case_counts) used cumulative case counts as raw data per province and date. In the processing scripts, the authors however use these cumulative case counts as normal case counts and then apply a cumsum formula to the data. This results in data per day and province that is, essentially, the result of two cumsum formulas (given that the raw data is already cumulative data). As far as I could check, the data for the other countries seems valid. I hope the authors can address this data issue and I d ask for some clarification or potentially rerun the analyses with accurate cumulative case data from the Netherlands. Marino van Zelst , 
 Lonni Besanu00e7on and Gideon Meyerowitz-Katz overlooked important information, undermining their comment. Just to give two examples. 2 The u2018control groupu2019 for countries not implementing lockdowns is composed of Sweden and South Korea. However, both countries, although they did not officially close businesses or ordered stay-at-home have implemented several other restrictions that we details below In Sweden, a ban on public gathering has been put in place and readjusted several time [A] some bars have been closed (see a couple of examples mentioned in [B]) and people have been recommended to work from home very early on (mid-March) [C]. While most schools stayed open in Sweden, high schools and universities were all using distance teaching [D]. All these restrictions have been included in the model. ![file] 3 The analysis does not seem to take into account the timing of implementations unlike the study done by Brauner et al. [A]. Time was a parameter of the model. ![file] , 
 #7 While we are not claiming that the authors should successfully model these, we are however suggesting that not including them constitutes a serious limitation of the study that should be acknowledged. The notation: $Policy_{pcit}$ Takes 1 if the policy $p$ was implemented in the country $c$, sub-unit $i$, at the time $t$, and takes 0 otherwise. **Therefore, such a notation embeds all necessary informations: when a policy has been implemented, for how long, and how it temporally relates to other policies (implemented before, after, overlapping or not).** For the sake of simplicity, let s keep the country $c$ and sub-unit $i$ constant. A policy $p_1$ put in place between time $t_{10}$ and $t_{11}$ becomes in the equation $Policy_{p_1cit}$ = 1 if $t_{10} le t le t_{11}$, 0 otherwise. A policy $p_2$ implemented between time $t_{20}$ and $t_{21}$ is encoded such as $Policy_{p_2cit}$ = 1 if $t_{20} le t le t_{21}$, 0 otherwise. It is obvious that, depending on $t_{10}$, $t_{11}$, $t_{20}$ and $t_{21}$, at a time $t$ we can very well have $Policy_{p_1cit}=1$ and $Policy_{p_2cit}=0$, or $Policy_{p_1cit}=0$ and $Policy_{p_2cit}=1$, or have them both equal to 0, or 1. , 
 #9 Praxis, Contopous, I think there is simply a misunderstanding of what is intended by  timing . Yes, of course time was part of the model and each intervention has its own timing faithfully recorded, as you explain (thank you). But it seems to me that the model only considers the instantaneous (daily) effect of each policy on the case growth rate, where one could expect temporal delays. For instance, confining families might well increase intra-family infections initially. Moreover, the model includes no interactions between interventions, yet they almost certainly exist. One consequence is that the order in which interventions are imposed may also matter, but that isn t captured by the model, even if the sequence information is encoded in the model. To give a silly example, mandating masks two weeks into a strict lockdown would suggest that masks were totally ineffective, but they might have quite a large effect if they were the only intervention. , 
 #9 As rightly pointed by Longidorus Apulus in response 10 and our response 7, considering timing as the time (i.e., date) is limiting. The order of intervention will affect their interactions. Timing also considers, in our reply, how the intervention s effect might be delayed for a lot of different reasons as rightfully explained in response 10. If our initial comment might not have been clear enough with the use of timing in point 3, I would argue that we make an excellent case for this in point 4 as well: There is also the question of implementation - in some cases, it is possible that behaviour changes occurred after interventions were announced, in others it may be the case that enforcement was needed after the initial date of implementation and thus the effect was instead delayed. These are merely two potential examples of situations that could have occurred, demonstrating the numerous sources of time-related bias that the authors have not attempted to control for in their analysis. , 
 #10 Ok I see,  timing actually does not refer to  moment, duration, periodicity, and the order in which measures have been implemented as explained, but lags and interactions. Makes much more sense. Nevertheless, I still think it s unfair to oppose this study to Brauner (as in #5). Brauner did not include interaction terms either. Policies were assumed to have independent, constant and multiplicative effects on R, so their model would not have been any better at isolating the effect of mandating masks two weeks into a strict lockdown. Regarding the infection to reporting delay in Brauner, they used very early epidemiological evidences obtained from a few symptomatic patients. One can argue that this delay is almost certainly largely overestimated, given that it became common practice to also (mostly in some countries) test asymptomatic subjects. In the end, I find it particularly interesting that two studies, using drastically different models, reached very similar conclusions. , 
 #12 Maybe  relative timing would be a slightly better term, but that still doesn t really bring out the fact that it s critical as the context of the interactions. Speaking generally (I haven t looked at Brauner), it s one thing to construct a viable model, but quite another for there to be sufficient and sufficiently variable data to identify things like time-dependence and interactions. I assume that nearly all countries went through a similar sequence of interventions of increasing severity and cost. Low-dimensional data could be one reason why different models might generate similar results. Just going by the quote in #5,  In accordance with other studies that took this approach (2, 6), we found that issuing a stay-at-home order had a small effect when a country had already closed educational institutions, closed nonessential businesses, and banned gatherings. it seems that models differ in how they classified interventions. If you consider school closures, business closures, etc as separate from a lockdown, it will naturally appear relatively ineffective. I think we saw that in good weather and or a spacious country, relatively severe measures short of complete lockdown sufficed to reduce propagation of the original variant. The point in #11 about behavioural changes anticipating regulatory changes is very well taken because this was definitely a feature of the UK response. During the first wave, it felt like the Prime Minister was the last person in the country to realise that a lockdown was necessary. , 
 The point in #11 about behavioural changes anticipating regulatory changes is very well taken because this was definitely a feature of the UK response. During the first wave, it felt like the Prime Minister was the last person in the country to realise that a lockdown was necessary. This is correct. But I don t see how previous studies did anything better to account for putative behavioral changes. Perhaps Lonni Besanu00e7on and Gideon Meyerowitz-Katz should explain precisely how previous studies better integrated timing considerations, beyond  moment, duration, periodicity, and the order in which measures have been implemented which is just wrong, in what way exactly this study suffered more than the previous ones from the possible  fact that Sweden had incredibly restricted testing during the times that are assessed , how precisely this study did not do as well as the previous ones to integrate the worrisome aspect of cultural similarity. It seems to me that most of their points, if not all, are not less applicable to other studies attempting to model the effects of NPIs. , 
 #14 It will be extremely difficult for us to defend a position we did not have in the first place. To begin with, we never said that other studies had integrated behavioural changes. We only stated that this was not accounted for in the present study and should therefore be discussed as a limitation of the model. This is clearly stated in our initial comment. Then, as explained in the initial comment #5, the very small sample size makes the study particularly sensitive to confounding factors such as cultural differences. This is, once again clearly said in comment #5. A larger sample size or more appropriate choice of countries would logically limit the impact of such confounding factors. 5 Given the small sample size, and lack of attempts to correct for confounders, the study is also at a very high risk of issues relating to unmeasured confounding. This has led to the problematic situation where, for example, Sweden and Iran are being directly compared in terms of the outcomes of their interventions without considering the plethora of cultural, social, and political differences between the countries that might impact their case counts during a pandemic. Readers can also feel free to check that some of the first studies did not include Sweden for instance, a country in which testing only became available very late (see Flaxman et al. for instance not including Sweden). It seems to me that most of their points, if not all, are not less applicable to other studies attempting to model the effects of NPIs. Well some of our points are indeed also applicable to other studies, but one can clearly see that not all of them would be applicable. The small sample size argument would not apply to Brauner et al. and, while Flaxman et al. only looked at 11 countries, all of them were located in Europe (thus likely to limit the effect of the confounding factor of cultural differences). Finally, previous studies did not use a control group of only 2 countries that were arbitrarily classified as less-restrictive although both of them implemented at least partially school closure (one for the longest time in the world). Again, this is clearly stated in our initial comment #5. Consequently, using these countries as a strict u201ccontrolu201d is misleading and inaccurate. This leads into a major error of the paper, which is dichotomizing responses into either u201clessu201d or u201cmoreu201d restrictive. This is an arbitrary designation decided on by the authors that is not substantiated in the paper. For example, school closures have been described by many as the most restrictive and or problematic government intervention (the phrase u201clast to close and first to openu201d is commonly used), but both of the u201clessu201d restrictive u201ccontrolsu201d closed (at least partially) schools for in-person teaching, one for far longer periods of time than many of the u201cmoreu201d restrictive u201ccasesu201d. To answer comments #12 and #13 indeed perhaps  relative timing would be a better term. We used timing to imply all of the things mentioned (periodicity, order, etc...) and behavioural changes as well. Perhaps this was misguided, but has been now cleared. Finally, about our opposition of this study and the one from Brauner et al. we mostly indicated that the present manuscript does not present the findings from one of the most recent study on the topic, nor does it propose to explain why the authors findings differ from it (or from Flaxman et al.). Considering that the sample size of Brauner et al. was 4 times the sample size used in the present study, I would argue that discussing these results and contrasting the methodologies and composition of the sample are vital to position the authors findings within related work. The fact that it is not done combined with the use of strong causal language gives the reader a feeling of strong and  undisputed findings which is clearly one of the limitations that we mentioned in our initial comment #5. , 
 Comment #5 clearly explained limitations, which make the article irrelevant, except for people who don t know the difference between  time and  timing . Here are my main concerns : - Comparing strategies country by country is open to many confounding factors and does not allow to conclude on lockdown effects. - Correlation and causation are totally being confused in this paper. In fact, this study could also have been entitled  Assessing the spread of COVID- 19 effects on mandatory stay- at- home and business closure - 2 countries only in lrNPI is irrelevant , 
 https: onlinelibrary.wiley.com doi 10.1111 eci.13518 , 
 For future readers, all our concerns have been summed up in a letter to the editor that is now available here https: doi.org 10.1111 eci.13518 , 
 The authors response has recently been published. https: onlinelibrary.wiley.com doi 10.1111 eci.13553 , 
 #19 So as #19 explained the authors published a response to the concerns raised in 3 different letters to the editors. The first letter is accessible here: https: doi.org 10.1111 eci.13518 The second letter is accessible here: https: doi.org 10.1111 eci.13556 The third letter is accessible here: https: doi.org 10.1111 eci.13529 The authors response is available here: https: onlinelibrary.wiley.com doi 10.1111 eci.13553 In order to group their responses, the authors of the first three letters with concerns on the publication replied together here: https: doi.org 10.1111 eci.13599 ,