SAE./No.200/January 2022
/October 2021

A LITERATURE REVIEW AND META-ANALYSISOF THE EFFECTS OF LOCKDOWNS ON
COVID-19 MORTALITY
Ambika Kandasamy

Jonas Herby, Lars RTALITY
Jonung, and Steve H. Hanke

A Literature Review and Meta-Analysis of theEffects of Lockdowns on COVID-19 Mortality
By Jonas Herby, Lars Jonung, and Steve H. Hanke
About the Series
The Studies in Applied Economics series is under the general direction of Prof. Steve H. Hanke,
Founder and Co-Director of The Johns Hopkins Institute for Applied Economics, Global Health,and the Study of Business Enterprise (hanke@jhu.edu). The views expressed in each working
paper are those of the authors and not necessarily those of the institutions that the authors areaffiliated with.
About the Authors
Jonas Herby (herby@cepos.dk) is special advisor at Center for Political Studies in Copenhagen,Denmark. His research focuses on law and economics. He holds a master’s degree in economics
from University of Copenhagen.
Lars Jonung (lars.jonung@nek.lu.se) is professor emeritus in economics at Lund University,
Sweden. He served as chairperson of the Swedish Fiscal Policy Council 2012-13, as researchadvisor at the European Commission 2000-2010, and as chief economic adviser to Prime Minister
Carl Bildt in 1992-94. He holds a PhD in Economics from the University of California, Los Angeles.
Steve H. Hanke is a Professor of Applied Economics and Founder & Co-Director of The Johns
Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise. He
is a Senior Fellow and Director of the Troubled Currencies Project at the Cato Institute, acontributor at National Review, a well-known currency reformer, and a currency and commodity
trader. Prof. Hanke served on President Reagan’s Council of Economic Advisers, has been anadviser to five foreign heads of state and five foreign cabinet ministers, and held a cabinet-levelrank in both Lithuania and Montenegro. He has been awarded seven honorary doctorate degrees
and is an Honorary Professor at four foreign institutions. He was President of Toronto TrustArgentina in Buenos Aires in 1995, when it was the world’s best-performing mutual fund.
Currently, he serves as Chairman of the Supervisory Board of Advanced Metallurgical Group N.V.
in Amsterdam. In 1998, he was named one of the twenty-five most influential people in the world
by World Trade Magazine. In 2020, Prof. Hanke was named a Knight of the Order of the Flag.

1

Abstract
This systematic review and meta-analysis are designed to determine whether there is empiricalevidence to support the belief that “lockdowns” reduce COVID-19 mortality. Lockdowns aredefined as the imposition of at least one compulsory, non-pharmaceutical intervention (NPI).
NPIs are any government mandate that directly restrict peoples’ possibilities, such as policies that
limit internal movement, close schools and businesses, and ban international travel. This study
employed a systematic search and screening procedure in which 18,590 studies are identified
that could potentially address the belief posed. After three levels of screening, 34 studies
ultimately qualified. Of those 34 eligible studies, 24 qualified for inclusion in the meta-analysis.
They were separated into three groups: lockdown stringency index studies, shelter-in-placeorder (SIPO) studies, and specific NPI studies. An analysis of each of these three groups support
the conclusion that lockdowns have had little to no effect on COVID-19 mortality. Morespecifically, stringency index studies find that lockdowns in Europe and the United States only
reduced COVID-19 mortality by 0.2% on average. SIPOs were also ineffective, only reducing
COVID-19 mortality by 2.9% on average. Specific NPI studies also find no broad-based evidenceof noticeable effects on COVID-19 mortality.
While this meta-analysis concludes that lockdowns have had little to no public health effects,
they have imposed enormous economic and social costs where they have been adopted. In
consequence, lockdown policies are ill-founded and should be rejected as a pandemic policy
instrument.
Acknowledgements
The authors thank Line Andersen, Troels Sabroe Ebbesen, Nicholas Hanlon, and Anders Lund
Mortensen for their research assistance.
The authors also with to thank Douglas Allen, Fredrik N. G. Andersson, Jonas Björk, ChristianBjørnskov, Joakim Book, Gunnar Brådvik, Kristoffer Torbjørn Bæk, Ulf Gerdtham, Daniel B. Klein,
Fredrik Charpentier Ljungqvist, Christian Heebøl-Nielsen, Martin Paldam, Jonas Ranstam,
Spencer Ryan, John Strezewski, Roger Svensson, Ulf Persson, Anders Waldenström, and Joakim
Westerlund for their comments.
Key Words: COVID-19, lockdown, non-pharmaceutical interventions, mortality, systematic
review, meta-analysis
JEL Classification: I18; I38; D192

1

Introduction

The global policy reaction to the COVID-19 pandemic is evident. Compulsory nonpharmaceutical interventions (NPIs), commonly known as “lockdowns” – policies that restrict
internal movement, close schools and businesses, and ban international travel – have been
mandated in one form or another in almost every country.
The first NPIs were implemented in China. From there, the pandemic and NPIs spread first to
Italy and later to virtually all other countries, see Figure 1. Of the 186 countries covered by the
Oxford COVID-19 Government Response Tracker (OxCGRT), only Comoros, an island country
in the Indian Ocean, did not impose at least one NPI before the end of March 2020.

Figure 1: Share of countries with OxCGRT stringency index above thresholds, January June 2020

Comment: The figure shows the share of countries, where the OxCGRT stringency index on a given date surpassed index 65, 70
and 75 respectively. Only countries with more than one million citizens are included (153 countries in total). The OxCGRT
stringency index records the strictness of NPI policies that restrict people’s behavior. It is calculated using all ordinalcontainment and closure policy indicators (i.e., the degree of school and business closures, etc.), plus an indicator recording
public information campaigns.
Source: Our World in Data.

Early epidemiological studies predicted large effects of NPIs. An often cited model simulation
study by researchers at the Imperial College London (Ferguson et al. (2020)) predicted that a3

suppression strategy based on a lockdown would reduce COVID-19 mortality by up to 98%.1
These predictions were questioned by many scholars. Our early interest in the subject was
spurred by two studies. First, Atkeson et al. (2020) showed that “across all countries and U.S.
states that we study, the growth rates of daily deaths from COVID-19 fell from a wide range ofinitially high levels to levels close to zero within 20-30 days after each region experienced 25
cumulative deaths.” Second, Sebhatu et al. (2020) showed that “government policies are strongly
driven by the policies initiated in other countries,” and less by the specific COVID-19-situationof the country.
A third factor that motivated our research was the fact that there was no clear negative
correlation between the degree of lockdown and fatalities in the spring of 2020 (see Figure 2).
Given the large effects predicted by simulation studies such as Ferguson et al. (2020), we would
have expected to at least observe a simple negative correlation between COVID-19 mortality andthe degree to which lockdowns were imposed. 2
Figure 2: Correlation between stringency index and COVID-19 mortality in European
countries and U.S. states during the first wave in 2020

Source: Our World in Data

1

With R0 = 2.4 and trigger on 60, the number of COVID-19-deaths in Great Britain could be reduced to 8,700
deaths from 510,000 deaths (-98%) with a policy consisting of case isolation + home quarantine + social
distancing + school/university closure, cf. Table 4 in Ferguson et al. (2020). R0 (the basic reproduction rate) is the
expected number of cases directly generated by one case in a population where all individuals are susceptible toinfection.
2
In addition, the interest in this issue was sparked by the work Jonung did on the expected economic effects of the
SARS pandemic in Europe in 2006 (Jonung and Röger, 2006). In this model-based study calibrated from Spanish
flu data, Jonung and Röger concluded that the economic effects of a severe pandemic would be rather limited—asharp contrast to the huge economic effects associated with lockdowns during the COVID-19 pandemic.

4

Today, it remains an open question as to whether lockdowns have had a large, significant effecton COVID-19 mortality. We address this question by evaluating the current academic literatureon the relationship between lockdowns and COVID-19 mortality rates.3 We use “NPI” todescribe any government mandate which directly restrict peoples’ possibilities. Our definition
does not include governmental recommendations, governmental information campaigns, accessto mass testing, voluntary social distancing, etc., but do include mandated interventions such as
closing schools or businesses, mandated face masks etc. We define lockdown as any policy
consisting of at least one NPI as described above. 4Compared to other reviews such as Herby (2021) and Allen (2021), the main difference in thismeta-analysis is that we carry out a systematic and comprehensive search strategy to identify all
papers potentially relevant to answer the question we pose. We identify 34 eligible empiricalstudies that estimate the effect of mandatory lockdowns on COVID-19 mortality using acounterfactual difference-in-difference approach. We present our results in such a way that they
can be systematically assessed, replicated, and used to derive overall meta-conclusions.5

2

Identification process: Search strategy and eligibility criteria

Figure 3 shows an overview of our identification process using a flow diagram designed
according to PRISMA guidelines (Moher et al. (2009). Of 18,590 studies identified during ourdatabase searches, 1,048 remained after a title-based screening. Then, 931 studies were excluded,
because they either did not measure the effect of lockdowns on mortality or did not use anempirical approach. This left 117 studies that were read and inspected. After a more thoroughassessment, 83 of the 117 were excluded, leaving 34 studies eligible for our meta-analysis. A
table with all 83 studies excluded in the final step can be found in Appendix B, Table 8.

We use “mortality” and “mortality rates” interchangeably to mean COVID-19 deaths per population.
For example, we will say that Country A introduced the non-pharmaceutical interventions school closures andshelter-in-place-orders as part of the country’s lockdown.
5
An interesting question is, “What damage lockdowns do to the economy, personal freedom and rights, and publichealth in general?” Although this question is important, it requires a full cost-benefit study, which is beyond thescope of this study.
3
45

Figure 3: PRISMA flow diagram for the selection of studies.

Below we present our search strategy and eligibility criteria, which follow the PRISMA
guidelines and are specified in detail in our protocol Herby et al. (2021).
2.1

Search strategy

The studies we reviewed were identified by scanning Google Scholar and SCOPUS for Englishlanguage studies. We used a wide range of search terms which are combinations of three searchstrings: a disease search string (“covid,” “corona,” “coronavirus,” “sars-cov-2”), a government

6

response search string6, and a methodology search string7. We identified papers based on 1,360
search terms. We also required mentions of “deaths,” “death,” and/or “mortality.” The searchterms were continuously updated (by adding relevant terms) to fit this criterion.8
We also included all papers published in Covid Economics. Our search was performed between
July 1 and July 5, 2021 and resulted in 18,590 unique studies.9 All studies identified using
SCOPUS and Covid Economics were also found using Google Scholar. This made us
comfortable that including other sources such as VOXeu and SSRN would not change the result.
Indeed, many papers found using Google Scholar were from these sources.
All 18,590 studies were first screened based on the title. Studies clearly not related to our
research question were deemed irrelevant.10
After screening based on the title, 1,048 papers remained. These papers were manually screenedby answering two questions:
1. Does the study measure the effect of lockdowns on mortality?
2. Does the study use an empirical ex post difference-in-difference approach (see eligibility
criteria below)?
Studies to which we could not answer “yes” to both questions were excluded. When in doubt, wemade the assessment based on reading the full paper, and in some cases, we consulted withcolleagues.11
After the manual screening, 117 studies were retrieved for a full, detailed review. These studies
were carefully examined, and metadata and empirical results were stored in an Excel
The government response search string used was: “non-pharmaceutical,” “nonpharmaceutical,” ”NPI,” ”NPIs,””lockdown,” “social distancing orders,” “statewide interventions,” “distancing interventions,” “circuit breaker,”“containment measures,” “contact restrictions,” “social distancing measures,” “public health policies,” “mobility
restrictions,” “covid-19 policies,” “corona policies,” “policy measures.”
7
The methodology search string used was: (“fixed effects,” “panel data,” “difference-in-difference,” “diff-in-diff,”“synthetic control,” “counterfactual” , “counter factual,” “cross country,” “cross state,” “cross county,” “crossregion,” “cross regional,” “cross municipality,” “country level,” “state level,” “county level,” “region level,”“regional level,” “municipality level,” “event study.”
8
If a potentially relevant paper from one of the 13 reviews (see eligibility criteria) did not show up in our search, we
added relevant words to our search strings and ran the search again. The 13 reviews were: Allen (2021); Brodeur
et al. (2021); Gupta et al. (2020); Herby (2021); Johanna et al. (2020); Nussbaumer-Streit et al. (2020); Patel et al.
(2020); Perra (2020); Poeschl and Larsen (2021); Pozo-Martin et al. (2020); Rezapour et al. (2021); Robinson(2021); Zhang et al. (2021).
9SCOPUS was continuously monitored between July 5th and publication using a search agent. Although the search
agent returned several hits during this period, only one of them, An et al. (2021), was eligible according to oureligibility criteria. The study is not included in our review, but the conclusions are in line with our conclusions, as
An et al. (2021) conclude that “The analysis shows that the mask mandate is consistently associated with lower
infection rates in the short term, and its early adoption boosts the long-term efficacy. By contrast, the other five
policy instruments— domestic lockdowns, international travel bans, mass gathering bans, and restaurant andschool closures—show weaker efficacy.”
10This included studies with titles such as “COVID-19 outbreak and air pollution in Iran: A panel VAR analysis”
and “Dynamic Structural Impact of the COVID-19 Outbreak on the Stock Market and the Exchange Rate: A
Cross-country Analysis Among BRICS Nations.”
11
Professor Christian Bjørnskov of University of Aarhus was particularly helpful in this process.
6

7

spreadsheet. All studies were assessed by at least two researchers. During this process, another64 papers were excluded because they did not meet our eligibility criteria. Furthermore, ninestudies with too little jurisdictional variance (< 10 observations) were excluded,12 and 10
synthetic control studies were excluded.13 A table with all 83 studies excluded in the final step
can be found in Appendix B, Table 8. Below we explain why these studies are excluded.
2.2

Eligibility criteria

Focus on mortality and lockdowns
We only include studies that attempt to establish a relationship (or lack thereof) between
lockdown policies and COVID-19 mortality or excess mortality. We exclude studies that use
cases, hospitalizations, or other measures.14
Counterfactual difference-in-difference approach
We distinguish between two methods used to establish a relationship (or lack thereof) between
mortality rates and lockdown policies. The first uses registered cross-sectional mortality data.
These are ex post studies. The second method uses simulated data on mortality and infection
rates.15 These are ex ante studies.
We include all studies using a counterfactual difference-in-difference approach from the formergroup but disregard all ex ante studies, as the results from these studies are determined by model
assumptions and calibrations.
Our limitation to studies using a “counterfactual difference-in-difference approach” means that
we exclude all studies where the counterfactual is based on forecasting (such as a SIR-model)
rather than derived from a difference-in-difference approach. This excludes studies like
Duchemin et al. (2020) and Matzinger and Skinner (2020). We also exclude all studies based oninterrupted time series designs that simply compare the situation before and after lockdown, as
12

The excluded studies with too few observations were: Alemán et al. (2020), Berardi et al. (2020), Conyon et al.
(2020a), Coccia (2021), Gordon et al. (2020), Juranek and Zoutman (2021), Kapoor and Ravi (2020), Umer andKhan (2020), and Wu and Wu (2020).
13
The excluded synthetic control studies were: Conyon and Thomsen (2021), Dave et al. (2020), Ghosh et al.
(2020), Born et al. (2021), Reinbold (2021), Cho (2020), Friedson et al. (2021), Neidhöfer and Neidhöfer (2020),
Cerqueti et al. (2021), and Mader and Rüttenauer (2021).
14
Analyses based on cases may pose major problems, as testing strategies for COVID-19 infections varyenormously across countries (and even over time within a given country). In consequence, cross-country
comparisons of cases are, at best, problematic. Although these problems exist with death tolls as well, they are far
more limited. Also, while cases and death tolls are correlated, there may be adverse effects of lockdowns that are
not captured by the number of cases. For example, an infected person who is isolated at home with family under a
SIPO may infect family members with a higher viral load causing more severe illness. So even if a SIPO reduces
the number of cases, it may theoretically increase the number of COVID-19-deaths. Adverse effects like this may
explain why studies like Chernozhukov et al. (2021) finds that SIPO reduces the number of cases but have nosignificant effect on the number of COVID-19-deaths. Finally, mortality is hierarchically the most important
outcome, cf. GRADEpro (2013)
15
These simulations are often made in variants of the SIR-model, which can simulate the progress of a pandemic in
a population consisting of people in different states (Susceptible, Infectious, or Recovered) with equations
describing the process between these states.

8the effect of lockdowns in these studies might contain time-dependent shifts, such as seasonality.
This excludes studies like Bakolis et al. (2021) and Siedner et al. (2020).
Given our criteria, we exclude the much-cited paper by Flaxman et al. (2020), which claimed
that lockdowns saved three million lives in Europe. Flaxman et al. assume that the pandemic
would follow an epidemiological curve unless countries locked down. However, this assumptionmeans that the only interpretation possible for the empirical results is that lockdowns are the only
thing that matters, even if other factors like season, behavior etc. caused the observed change inthe reproduction rate, Rt. Flaxman et al. are aware of this and state that “our parametric form of
Rt assumes that changes in Rt are an immediate response to interventions rather than gradualchanges in behavior.” Flaxman et al. illustrate how problematic it is to force data to fit a certain
model if you want to infer the effect of lockdowns on COVID-19 mortality.16
The counterfactual difference-in-difference studies in this review generally exploit variation
across countries, U.S. states, or other geographical jurisdictions to infer the effect of lockdowns
on COVID-19 fatalities. Preferably, the effect of lockdowns should be tested using randomized
control trials, natural experiments, or the like. However, there are very few studies of this type.17
Synthetic control studies
The synthetic control method is a statistical method used to evaluate the effect of an intervention
in comparative case studies. It involves the construction of a synthetic control which functions as
the counter factual and is constructed as an (optimal) weighted combination of a pool of donors.
For example, Born et al. (2021) create a synthetic control for Sweden which consists of 30.0%
Denmark, 25.3% Finland, 25.8% Netherlands, 15.0% Norway, and 3.9% Sweden. The effect of
the intervention is derived by comparing the actual developments to those contained in thesynthetic control.
We exclude synthetic control studies because of their inherent empirical problems as discussed
by Bjørnskov (2021b). He finds that the synthetic control version of Sweden in Born et al. (2021)
deviates substantially from “actual Sweden,” when looking at the period before mid-March 2020,
when Sweden decided not to lock down. Bjørnskov estimates that actual Sweden experienced

16

Several scholars have criticized Flaxman et al. (2020), e.g. see Homburg and Kuhbandner (2020), Lewis (2020),
and Lemoine (2020).
17
Kepp and Bjørnskov (2021) is one such study. They use evidence from a quasi-natural experiment in the Danish
region of Northern Jutland. After the discovery of mutations of Sars-CoV-2 in mink – a major Danish export –seven of the 11 municipalities of the region went into extreme lockdown in early November, while the four othermunicipalities retained the moderate restrictions of the remaining country. Their analysis shows that while
infection levels decreased, they did so before lockdown was in effect, and infection numbers also decreased inneighbor municipalities without mandates. They conclude that efficient infection surveillance and voluntarycompliance make full lockdowns unnecessary, at least in some circumstances. Kepp and Bjørnskov (2021) is notincluded in our review, because they focus on cases and not COVID-19 mortality. Dave et al. (2020) is anothersuch study. They see the Wisconsin Supreme Court abolishment of Wisconsin’s “Safer at Home” order (a SIPO)
as a natural experiment and find that “the repeal of the state SIPO impacted social distancing, COVID-19 cases, or
COVID-19-related mortality during the fortnight following enactment.” Dave et al. (2020) is not included in ourreview, because they use a synthetic control method.

9

approximately 500 fewer deaths the first 11 weeks of 2020 and 4,500 fewer deaths in 2019
compared to synthetic Sweden.
This problem is inherent in all synthetic control studies of COVID-19, Bjørnskov argues,
because the synthetic control should be fitted based on a long period of time before theintervention or the event one is studying the consequences of – i.e., the lockdown Abadie (2021).
However, this is not possible for the coronavirus pandemic, as there clearly is no long periodwith coronavirus before the lockdown. Hence, the synthetic control study approach is by design
not appropriate for studying the effect of lockdowns.
Jurisdictional variance - few observations
We exclude all interrupted time series studies which simply compare mortality rates before andafter lockdowns. Simply comparing data from before and after the imposition of lockdowns
could be the result of time-dependent variations, such as seasonal effects. For the same reason,
we also exclude studies with little jurisdictional variance.18 For example, we exclude Conyon etal. (2020b) who “exploit policy variation between Denmark and Norway on the one hand andSweden on the other” and, thus, only have one jurisdictional area in the control group. Although
this is a difference-in-difference approach, there is a non-negligible risk that differences arecaused by much more than just differences in lockdowns. Another example is Wu and Wu(2020), who use all U.S. states, but pool groups of states so they end with basically threeobservations. None of the excluded studies cover more than 10 jurisdictional areas.19 One study
is a special case of the jurisdictional variance criteria (Auger et al. (2020). Those researchers
analyze the effect of school closures in U.S. states and find that those closures reduce mortality
by 35%. However, all 50 states closed schools between March 13, 2020, and March 23, 2020,
which means that all difference-in-difference is based on maximum 10 days. Given the long lag
between infection and death, there is a risk that Auger et al.’s approach is an interrupted timeseries analysis where they compare United States before and after school closures, rather than atrue difference-in-difference approach. However, we choose to include this study, as it is eligible
under our protocol Herby et al. (2021).
Publication status and date
We include all ex post studies regardless of publication status and date. That is, we cover bothworking papers and papers published in journals. We include the early papers because theknowledge of the COVID-19-pandemic grew rapidly in the beginning, making later papers able
to stand on the shoulders of previous work. Also, in the early days of COVID-19, speed was

A jurisdictional area can be countries, U.S. states, or counties. With "jurisdictional variance” we refer to variationin mandates across jurisdictional areas.
19
All studies excluded on this criterion are listed in footnote 12.
18

10crucial which may have affected the quality of the papers. Including them makes it possible tocompare the results of early studies to studies carried out at a later stage.20
The role of optimal timing
We exclude papers which analyze the effect of early lockdowns in contrast to later lockdowns.
There’s no doubt that being prepared for a pandemic and knowing when it arrives at yourdoorstep is vital. However, at least two problems arise with respect to evaluating the effect ofwell-timed lockdowns.
First, when COVID-19 hit Europe and the United States, it was virtually impossible to determine
the right timing. The World Health Organization declared the outbreak a pandemic on March 11,
2020, but at that date, Italy had already registered 13.7 COVID-19 deaths per million. On March
29, 2020, 18 days after the WHO declared the outbreak a pandemic and the earliest a lockdown
response to the WHO’s announcement could potentially have an effect, the mortality rate in Italy
was a staggering 178 COVID-19 deaths per million with an additional 13 per million dying eachday.21
Secondly, it is extremely difficult to differentiate between the effect of public awareness and theeffect of lockdowns when looking at timing because people and politicians are likely to react tothe same information. As Figure 4 illustrates, all European countries and U.S. states that were hit
hard and early by COVID-19 experienced high mortality rates, whereas all countries hit
relatively late experienced low mortality rates. Björk et al. (2021) illustrate the difficulties inanalyzing the effect of timing. They find that a 10-stringency-points-stricter lockdown would
reduce COVID-19 mortality by a total of 200 deaths per million22 if done in week 11, 2020, butwould only have approximately 1/3 of the effect if implemented one week earlier or later and noeffect if implemented three weeks earlier or later. One interpretation of this result is that
lockdowns do not work if people either find them unnecessary and fail to obey the mandates or if
people voluntarily lock themselves down. This is the argument Allen (2021) uses for the
ineffectiveness of the lockdowns he identifies. If this interpretation is true, what Björk et al.
(2021) find is that information and signaling is far more important than the strictness of the
lockdown. There may be other interpretations, but the point is that studies focusing on timing
cannot differentiate between these interpretations. However, if lockdowns have a notable effect,
we should see this effect regardless of the timing, and we should identify this effect morecorrectly by excluding studies that exclusively analyze timing.

20

We also intended to exclude studies which were primarily based on data from 2021 (as these studies would beheavily affected by vaccines) and studies that did not cover at least one EU-country, the United States, one U.S.
U.S. state or Latin America, and where at least one country/state was not an island. However, we did not find any
such studies.
21
There’s approximately a two-to-four-week gap between infection and deaths. See footnote 29.
22
They estimate that 10-point higher stringency will reduce excess mortality by 20 “per week and million” in the 10
weeks from week 14 to week 23.

11

Figure 4: Taken by surprise. The importance of having time to prepare
United States

900

1,800

800

1,600

1st wave deaths pr. million

1st wave deaths pr. million

Europe

700
600
500
400
300
200

1,400
1,200
1,000
800
600

400
200

100
0
10-Mar-20

14-Apr-20

19-May-20

010-Mar-20

23-Jun-20

14-Apr-20

19-May-20

23-Jun-20

Date to reach 20 COVID-19-deaths per million

Date to reach 20 COVID-19-deaths per million

Comment: The figure shows the relationship between early pandemic strength and total 1st wave of COVID-19 death toll. On the
X-axis is “Days to reach 20 COVID-19-deaths per million (measured from February 15, 2020).” The Y-axis shows mortality(deaths per million) by June 30, 2020.
Source: Reported COVID-19 deaths and OxCGRT stringency for European countries and U.S. states with more than one million
citizens. Data from Our World in Data.

We are aware of one meta-analysis by Stephens et al. (2020), which looks into the importance oftiming. The authors find 22 studies that look at policy and timing with respect to mortality rates,
however, only four were multi-country, multi-policy studies, which could possibly account forthe problems described above. Stephens et al. conclude that “the timing of policy interventions
across countries relative to the first Wuhan case, first national disease case, or first national
death, is not found to be correlated with mortality.” (See Appendix A for further discussion of
the role of timing.)

3

The empirical evidence

In this section we present the empirical evidence found through our identification process. We
describe the studies and their results, but also comment on the methodology and possible
identification problems or biases.
3.1

Preliminary considerations

Before we turn to the eligible studies, we present some considerations that we adopted wheninterpreting the empirical evidence.
Empirical interpretation
While the policy conclusions contained in some studies are based on statistically significant
results, many of these conclusions are ill-founded due to the tiny impact associated with said
statistically significant results. For example, Ashraf (2020) states that “social distancing
12

measures has proved effective in controlling the spread of [a] highly contagious virus.”
However, their estimates show that the average lockdown in Europe and the U.S only reduced
COVID-19 mortality by 2.4%.23 Another example is Chisadza et al. (2021). The authors argue
that “less stringent interventions increase the number of deaths, whereas more severe responses
to the pandemic can lower fatalities.” Their conclusion is based on a negative estimate for thesquared term of stringency which results in a total negative effect on mortality rates (i.e. fewer
deaths) for stringency values larger than 124. However, the stringency index is limited to values
between 0 and 100 by design, so the conclusion is clearly incorrect. To avoid any such biases, webase our interpretations solely on the empirical estimates and not on the authors’ owninterpretation of their results.
Handling multiple models, specifications, and uncertainties
Several studies adopt a number of models to understand the effect of lockdowns. For example,
Bjørnskov (2021a) estimates the effect after one, two, three, and four weeks of lockdowns. For
these studies, we select the longest time horizon analyzed to obtain the estimate closest to thelong-term effect of lockdowns.
Several studies also use multiple specifications including and excluding potentially relevant
variables. For these studies, we choose the model which the authors regard as their main
specification. Finally, some studies have multiple models which the authors regard as equallyimportant. One interesting example is Chernozhukov et al. (2021), who estimate two models
with and without national case numbers as a variable. They show that including this variable intheir model alters the results substantially. The explanation could be that people responded tonational conditions. For these studies, we present both estimates in Table 1, but – following
Doucouliagos and Paldam (2008) – we use an average of the estimates in our meta-analysis inorder to not give more weight to a study with multiple models relative to studies with just one
principal model.
For studies looking at different classes of countries (e.g. rich and poor), we report both estimatesin Table 1 but use the estimate for rich Western countries in our meta-analysis, where we derivecommon estimates for Europe and the United States.
Effects are measured “relative to Sweden in the spring of 2020”
Virtually all countries in the world implemented mandated NPIs in response to the COVID-19
pandemic. Hence, most estimates are relative to “doing the least,” which in many Westerncountries means relative to doing as Sweden has done, especially during the first wave, when
Sweden, do to constitutional constraints, implemented very few restrictions compared to otherwestern countries (Jonung and Hanke 2020). However, some studies do compare the effect of
doing something to the effect of doing absolutely nothing (e.g. Bonardi et al. (2020)).
The consequence is that some estimates are relative to “doing the least” while others are relative
to “doing nothing.” This may lead to biases if “doing the least” works as a signal (or warning)23

We describe how we arrive at the 2.4% in Section 4.

13

which alters the behavior of the public. For example, Gupta et al. (2020) find a large effect of
emergency declarations, which they argue “are best viewed as an information instrument thatsignals to the population that the public health situation is serious and they act accordingly,” onsocial distancing but not of other policies such as SIPOs (shelter-in-place orders). Thus, if wecompare a country issuing a SIPO to a country doing nothing, we may overestimate the effect ofa SIPO, because it is the sum of the signal and the SIPO. Instead, we should compare the country
issuing the SIPO to a country “doing the least” to estimate the marginal effect of the SIPO.
To take an example, Bonardi et al. (2020) find relatively large effects of doing something but noeffect of doing more. They find no extra effect of stricter lockdowns relative to less strict
lockdowns and state that “our results point to the fact that people might adjust their behaviorsquite significantly as partial measures are implemented, which might be enough to stop the
spread of the virus.” Hence, whether the baseline is Sweden, which implemented a ban on large
gatherings early in the pandemic, or the baseline is “doing nothing” can affect the magnitude ofthe estimated impacts. There is no obvious right way to resolve this issue, but since estimates inmost studies are relative to doing less, we report results as compared to “doing less” whenavailable. Hence, for Bonardi et al. we state that the effect of lockdowns is zero (compared toSweden’s “doing the least”).

3.2

Overview of the findings of eligible studies

Table 1 covers the 34 studies eligible for our review.24 Out of these 34 studies, 22 were peerreviewed and 12 were working papers. The studies analyze lockdowns during the first wave.
Most of the studies (29) use data collected before September 1st, 2020 and 10 use data collected
before May 1st, 2020. Only one study uses data from 2021. All studies are cross-sectional,
ranging across jurisdictions. Geographically, 14 studies cover countries worldwide, four coverEuropean countries, 13 cover the United States, two cover Europe and the United States, and one
covers regions in Italy. Seven studies analyze the effect of SIPOs, 10 analyze the effect of stricter
lockdowns (measured by the OxCGRT stringency index), 16 studies analyze specific NIP’s
independently, and one study analyzes other measures (length of lockdown).
Several studies find no statistically significant effect of lockdowns on mortality. For example,
this includes Bjørnskov (2021a) and Stockenhuber (2020) who find no significant effect ofstricter lockdowns (higher OxCGRT stringency index), Sears et al. (2020) and Dave et al.
(2021), who find no significant effect of SIPOs, and Chaudhry et al. (2020), Aparicio and
Grossbard (2021) and Guo et al. (2021) who find no significant effect of any of the analyzed
NIP’s, including business closures, school closures and border closures.
Other studies find a significant negative relationship between lockdowns and mortality. Fowler
et al. (2021 find that SIPOs reduce COVID-19 mortality by 35%, while Chernozhukov et al.

24

The following information can be found for each study in Table 2.

14

(2021) find that employee mask mandates reduces mortality by 34% and closing businesses andbars reduces mortality by 29%.
Some studies find a significant positive relationship between lockdowns and mortality. This
includes Chisadza et al. (2021), who find that stricter lockdowns (higher OxCGRT stringency
index) increases COVID-19 mortality by 0.01 deaths/million per stringency point and Berry etal. (2021), who find that SIPOs increase COVID-19 mortality by 1% after 14 days.
Most studies use the number of official COVID-19 deaths as the dependent variable. Only one
study, Bjørnskov (2021a), looks at total excess mortality which – although is not perfect – weperceive to be the best measure, as it overcomes the measurement problems related to properly
reporting COVID-19 deaths.
Several studies explicitly claim that they estimate the actual causal relationship between
lockdowns and COVID-19 mortality. Some studies use instrumental variables to justify thecausality associated with their analysis, while others make causality probable using anecdotal
evidence.25 But, Sebhatu et al. (2020) show that government policies are strongly driven by thepolicies initiated in neighboring countries rather than by the severity of the pandemic in theirown countries. In short, it is not the severity of the pandemic that drives the adoption of
lockdowns, but rather the propensity to copy policies initiated by neighboring countries. The
Sebhatu et al. conclusion throws into doubt the notion of a causal relationship between
lockdowns and COVID-19 mortality.

Table 1: Summary of eligible studies
1. Study (Author &title)

2.
3. Description
Measure4. Results

Alderman and Harjoto(2020); "COVID-19: U.S.
shelter-in-place orders
and demographic
characteristics linked tocases, mortality, andrecovery rates"
Aparicio and Grossbard(2021); "Are Covid
Fatalities in the U.S.
Higher than in the EU,
and If so, Why?"

COVID19
mortality

Use State-level data from the COVID-19
Tracking Project data all U.S. states, and amultivariate regression analysis toempirically investigate the impacts of theduration of shelter-in-place orders on
mortality.

Find that shelter-inplace orders are - forthe average duration associated with 1%(insignificant) fewer
deaths per capita.

COVID19
mortality

Their main focus is to explain the gap in
COVID-19-fatalities between Europe andthe United States based on COVID-deaths
and other data from 85 nations/states.
They include status for "social events"(ban on public gatherings, cancellation ofmajor events and conferences), school
closures, shop closures "partial
lockdowns" (e.g. night curfew) and"lockdowns" (all-day curfew) 100 days
after the pandemic onset in acountry/state. None of theseinterventions have a significant effect on
COVID-19 mortality. They also find no

Find no effect of "social
events" (ban on publicgatherings, cancellationof major events andconferences), school
closures, shop closures"partial lockdowns" (e.g.
night curfew) and"lockdowns" (all-day
curfew) 100 days after
the pandemic onset.

25

5. Comments

In the abstract the authors states that "various
types of social distance measures such as school
closings and lockdowns, and how soon they
were implemented, help explain the
U.S./EUROPE gap in cumulative deaths
measured 100 days after the pandemic’s onset
in a state or country" although their estimates
are insignificant.

E.g. Dave et al. (2021) states that “estimated case reductions accelerate over time, becoming largest after 20 days
following enactment of a SIPO. These findings are consistent with a causal interpretation.”15

1. Study (Author &title)

2.
3. Description
MeasureAshraf (2020);"Socioeconomic
conditions, government
interventions and healthoutcomes during COVID19"

COVID19
mortality

Auger et al. (2020);"Association between
statewide school closure
and COVID-19 incidence
and mortality in the U.S."

COVID19
mortality

Berry et al. (2021);"Evaluating the effects ofshelter-in-place policies
during the COVID-19
pandemic"

COVID19
mortality

Bjørnskov (2021a); "Did
Lockdown Work? An
Economist's CrossCountry Comparison"

Excessmortality

Blanco et al. (2020); "Do
Coronavirus Containment
Measures Work?
Worldwide Evidence"

COVID19
mortality

4. Results

5. Comments

significant effect of early cancelling ofsocial events, school closures, shop
closures, partial lockdowns and full
lockdowns.
Their main focus is on the effectiveness ofpolicies targeted to diminish the effect ofsocioeconomic inequalities (economic
support) on COVID-19-deaths. They use
data from 80 countries worldwide andinclude the OxCGRT stringency as acontrol variable in their models. The paper
finds a significant negative (fewer deaths)
effect of stricter lockdowns. The effect oflockdowns is insignificant, when they
include an interaction term between thesocioeconomic conditions index and theeconomic support index in their model.

For each 1-unit increasein OxCGRT stringency
index, the cumulative
mortality changes by 0.326 deaths per million(fewer deaths). The
estimate is -0.073
deaths per million but
insignificant, when
including an interaction
term between thesocioeconomic
conditions index andthe economic support
index.
U.S. population-based observational study State that they adjust
which uses interrupted time series
for several factors (e..ganalyses incorporating a lag period topercentage of state’s
allow for potential policy-associatedpopulation aged 15changes to occur. To isolate theyears and 65 years,
association of school closure with
CDC's social
outcomes, state-level nonpharmaceutical vulnerability index,
interventions and attributes werestay-at-home or
included in negative binomial regression
shelter-in-place order,models. Models were used to derive therestaurant and bar
estimated absolute differences between
closure, testing rate perschools that closed and schools that1000 residents etc.),
remained open. The main outcome of the but does not specify
study is COVID-19 daily incidence andhow and do not present
mortality per 100000 residents.
estimates.
The authors use U.S. county data on
SIPO increases the
COVID-19 deaths from Johns Hopkin and number of deaths by
SIPO data from the University of
0,654 per million after
Washington to estimate the effect of
14 days (see Fig. 2)
SIPO's. They find no detectable effects of
SIPO on deaths. The authors stress that
their findings should not be interpreted as
evidence that social distancing behaviors
are not effective. Many people hadalready changed their behaviors before
the introduction of shelter-in-placeorders, and shelter-in-place orders appear
to have been ineffective precisely because
they did not meaningfully alter social
distancing behavior.
Uses excess mortality and OxCGRT
A stricter lockdown
stringency from 24 European countries to (OxCGRT stringency)
estimate the effect of lockdown on thedoes not have anumber of deaths one, two, three andsignificant effect onfour weeks later. Finds no effect (negative excess mortality.
but insignificant) of (stricter) lockdowns.
The author’s specification using
instrument variables yields similar results.
Use data for deaths and NPIs from Hale et When using the naïve
al. (2020) covering 158 countries between dummy variable
January and August 2020 to evaluate the approach, alleffect of eight different NPIs (stay atparameters are
home, bans on gatherings, bans on public statistically16

All 50 states closed schools between March 13,
2020, and March 23, 2020. Hence, alldifference-in-difference is based on maximum
10 days, and given the long lag between
infection and death, there is a risk that theirapproach is more an interrupted time series
analysis, where they compare United States
before and after school closures, rather than atrue difference-in-difference approach.
However, we choose to include the study in ourreview as it - objectively speaking - lives up to
the eligibility criteria specified in our protocol.

The authors conclude that "We do not finddetectable effects of these policies [SIPO] on
disease spread or deaths.” However, thisstatement does not correspond to their results.
In figure 2 they show that the effect on deaths
is significant after 14 days. Looks at the effect
14 days after SIPO's are implemented which is a
short lag given that the time between infection
and deaths is at least 2-3 weeks.

Finds a positive (more deaths) effect after one
and two weeks, which could indicate that otherfactors (omitted variables) affect the results.

Run the same model four times for each of thedifferent NPIs (stay at home-orders, ban on
meetings, ban on public events and mobility
restrictions). These NPIs were often introduced
almost simultaneously so there is a high risk of1. Study (Author &title)

2.
3. Description
Measureevents, closing schools, lockdowns ofworkplaces, interruption of public
transportation services, and international
border closures. They address thepossible endogeneity of the NPIs by using
instrumental variables.

Bonardi et al. (2020);"Fast and local: How did
lockdown policies affectthe spread and severity of
the covid-19"

Growthrates

Use NPI data scraped from news
headlines from LexisNexis and death datafrom Johns Hopkins University up to April
1st 2020 in a panel structure with 184
countries. Controls for country fixed
effects, day fixed effects and withincountry evolution of the disease.

Bongaerts et al. (2021);"Closed for business: The
mortality impact ofbusiness closures during
the Covid-19 pandemic"

COVID19
mortality

Uses variation in exposure to closed
sectors (e.g. tourism) in municipalities
within Italy to estimate the effect ofbusiness closures. Assuming thatmunicipalities with different exposures toclosed sectors are not inherently
different, they find that municipalities
with higher exposure to closed sectors
experienced subsequently lower mortality
rates.
Uses information on COVID-19 related
national policies and health outcomesfrom the top 50 countries ranked bynumber of cases. Finds no significant
effect of any NPI on the number of
COVID-19-deaths.

Chaudhry et al. (2020); "A COVIDcountry level analysis
19
measuring the impact ofmortality
government actions,country preparedness andsocioeconomic factors on
COVID-19 mortality andrelated health outcomes"

Chernozhukov et al.
(2021); "Causal impact ofmasks, policies, behavior
on early covid-19
pandemic in the U.S."

Growthrates

Uses COVID-deaths from the New YorkTimes and Johns Hopkins and data for
U.S. States from Raifman et al. (2020) toestimate the effect of SIPO, closed
nonessential businesses, closed K-12
schools, closed restaurants except
takeout, closed movie theaters, and facemask mandates for employees in public
facing businesses.

17

4. Results

5. Comments

insignificant. On thecontrary, estimatesusing the instrumental
variable approachindicate that NPIs are
effective in reducing
the growth rate in thedaily number of deaths
14 days later.
Find that certain
interventions (SIPO,
regional lockdown andpartial lockdown) work(in developed
countries), but thatstricter interventions(SIPO) do not have alarger effect than lessstrict interventions (e.g.
restrictions ongatherings). Find noeffect of borderclosures.
Business shutdown
saved 9,439 Italian lives
by April 13th 2020. This
corresponds to a
reduction of deaths by
32%, as there were20,465 COVID-19deaths in Italy by mid
April 2020.

multicollinearity with each run capturing thesame underlying effect. Indeed, the size andstandard errors of the estimates are worryingly
similar. Looks at the effect 14 days after NPIs
are implemented which is a fairly short lag given
the time between infection and deaths is 2-3
weeks, cf. e.g. Flaxman et al. (2020), whichaccording to Bjørnskov (2020) appears to be theminimum typical time from infection to death).
Find a positive (more deaths) effect on day 1
after lockdown which may indicate that theirresults are driven by other factors (omitted
variables). We rely on their publicly available
version submitted to CEPR Covid Economics,
but estimates on the effect of deaths can befound in Supplementary material, which isavailable in an updated version hosted on theDanish Broadcasting Corporation's webpage:https://www.dr.dk/static/documents/2021/03/
04/managing_pandemics_e3911c11.pdf

Finds no significant
effect on mortality of
any of the analyzed
interventions (partial
border closure,complete borderclosure, partial
lockdown (physicaldistancing measuresonly), complete
lockdown (enhanced
containment measures
including suspension of
all non-essential
services), and curfews).
Finds that mandatory
masks for employees
and closing K-12
schools reduces deaths.
SIPO and closing
business (average of
closed businesses,
restaurants and movie
theaters) has no
statistically significant
effect. The effect ofschool closures is highly
sensitive to the

They (implicitly) assume that municipalities withdifferent exposures to closed sectors are notinherently different. This assumption could be
problematic, as more touristed municipalities
can be very different from e.g. moreindustrialized municipalities.

States that ”our regression specification for caseand death growths is explicitly guided by a SIR
model although our causal approach does nothinge on the validity of a SIR model.” We areuncertain if this means that data are managed tofit an SIR-model (and thus should fail our
eligibility criteria).

1. Study (Author &title)

2.
3. Description
Measure4. Results

5. Comments

inclusion of nationalcase and death data.
Chisadza et al. (2021);"Government
Effectiveness and the
COVID-19 Pandemic"

COVID19
mortality

Uses COVID-19-deaths and OxCGRT
stringency from 144 countries to estimate
the effect of lockdown on the number of
COVID-19-deaths. Find a significant
positive (more deaths) non-linear
association between government
response indices and the number ofdeaths.

Dave et al. (2021); "WhenDo Shelter-in-Place
Orders Fight Covid-19
Best? Policy
Heterogeneity AcrossStates and Adoption
Time"

COVID19
mortality

Dergiades et al. (2020);"Effectiveness ofgovernment policies in
response to the COVID19 outbreak"

COVID19
mortality

Uses smartphone location tracking andstate data on COVID-19 deaths and SIPO
data (supplemented by their own
searches) collected by the New YorkTimes to estimate the effect of SIPO's.
Finds that SIPO was associated with a
9%–10% increase in the rate at whichstate residents remained in their homes
full-time, but overall they do not find an
significant effect on mortality after 20+
days (see Figure 4). Indicate that thelacking significance may be due to longterm estimates being identified of a few
early adopting states.
Uses daily deaths from the European
Centre for Disease Prevention andControl and OxCGRT stringency from 32
countries worldwide (including U.S.) toestimates the effect of lockdown on thenumber of deaths.

Fakir and Bharati (2021);
COVID"Pandemic catch-22: The 19
role of mobility
mortality
restrictions andinstitutional inequalities inhalting the spread of
COVID-19"

Fowler et al. (2021);"Stay-at-home orders
associate withsubsequent decreases in
COVID-19 cases andfatalities in the UnitedStates"COVID19
mortality

An increase by 1 on"stringency index"
increases the number ofdeaths by 0.0130 per
million. The sign of thesquared term isnegative, but the
combined non-linear
estimate is positive(increases deaths) andlarger than the linear
estimate for all valuesof the OxCGRT
stringency index.
Finds no overallsignificant effect of
SIPO on deaths but
does find a negative
effect (fewer deaths) inearly adopting states.

The author states that "less stringent
interventions increase the number of deaths,
whereas more severe responses to thepandemic can lower fatalities.” However,according to their estimates this is not correct,
as the combined non-linear estimate cannot be
negative for relevant values of the OxCGRT
stringency index (0 to 100).

Finds that the greater
the strength ofgovernment
interventions at an earlystage, the more
effective these are in
slowing down orreversing the growthrate of deaths.
Uses data from 127 countries. combining Find large causal effects
high-frequency measures of mobility data of stricter restrictions
from Google’s daily mobility reports,on the weekly growthcountry-date-level information on therate of recorded deaths
stringency of restrictions in response to
attributed to COVIDthe pandemic from Oxford’s Coronavirus
19. Show that more
Government Response Tracker (OxCGRT), stringent interventions
and daily data on deaths attributed tohelp more in richer,
COVID-19 from Our World In Data andmore educated, more
the Johns Hopkins University. Instrument democratic, and lessstringency using day-to-day changes in
corrupt countries withthe stringency of the restrictions in theolder, healthierrest of the world.
populations and more
effective governments.
Uses U.S. county data on COVID-19
Stay-at-home orders
deaths and SIPO data collected by theare also associated withNew York Times to estimate the effect of a 59.8 percent (18.3 to
SIPO's using a two-way fixed-effects80.2) average reduction
difference-in-differences model. Find ain weekly fatalities afterlarge and early (after few days) effect of
three weeks. These
SIPO on COVID-19 related deaths.
results suggest thatstay-at-home orders

Focus is on the effect of early stage NPIs andthus does not absolutely live up to our eligibility
criteria. However, we include the study as it
differentiates between lockdown strength at an
early stage.

18

Find large effects of SIPO on deaths after 6-14
days in early adopting states (see Table 8),
which is before an SIPO-related effect would beseen. This could indicate that other factors
rather than SIPO's drive the results.

Finds a larger effect on deaths after 0 days thanafter 14 and 21 days (Table 3). This is surprising
given that it takes 2-3 weeks from infection todeath, and it may indicate that their results are
driven by other factors.

Finds the largest effect of SIPO on deaths after
10 days (see Figure 4), before a SIPO-related
effect could possibly be seen as it takes 2-3
weeks from infection to death. This could
indicate that other factors drive their results.

1. Study (Author &title)

Fuller et al. (2021);"Mitigation Policies and
COVID-19–AssociatedMortality — 37 European
Countries, January 23–
June 30, 2020"

2.
3. Description
MeasureCOVID19
mortality

Gibson (2020);
COVID"Government mandated
19
lockdowns do not reduce mortality
Covid-19 deaths:
implications for evaluating
the stringent New
Zealand response"

Goldstein et al. (2021);"Lockdown Fatigue: The
Diminishing Effects of
Quarantines on the
Spread of COVID-19 "COVID19
mortality

Guo et al. (2021);"Mitigation Interventionsin the United States: An
Exploratory Investigationof Determinants andImpacts"

COVID19
mortality

4. Results

Uses COVID-19-deaths and OxCGRT
stringency in 37 European countries toestimate the effect of lockdown on thenumber of COVID-19-deaths. Find asignificant negative (fewer deaths) effectof stricter lockdowns after mortality
threshold is reached (the threshold is adaily rate of 0.02 new COVID-19 deaths
per 100,000 population (based on a 7-day
moving average))
Uses data for every county in the UnitedStates from March through June 1, 2020,
to estimate the effect of SIPO (called"lockdown") on COVID-19 mortality.
Policy data are acquired from American
Red Cross reporting on emergency
regulations. His control variables include
county population and density, the elder
share, the share in nursing homes, nineother demographic and economic
characteristics and a set of regional fixed
effects. Handles causality problems using
instrument variables (IV).
Uses panel data from 152 countries withdata from the onset of the pandemic until
December 31, 2020. Finds that lockdowns
tend to reduce the number of COVID-19
related deaths, but also that this benign
impact declines over time: after fourmonths of strict lockdown, NPIs have asignificantly weaker contribution in termsof their effect in reducing COVID-19
related fatalities.

might have reduced
confirmed cases by
390,000 (170,000 to
680,000) and fatalities
by 41,000 (27,000 to
59,000) within the first
three weeks in localities
that implemented stayat-home orders.
For each 1-unit increasein OxCGRT stringency
index, the cumulative
mortality decreases by
0.55 deaths per
100,000.

Find no statisticallysignificant effect of
SIPO.

Stricter lockdowns
reduce deaths for the
first 60 days,
whereafter the
cumulative effectbegins to decrease. If
reintroduced after 120,the effect of lockdowns
is smaller in the shortrun, but after 90 days
the effect is almost thesame as during first
lockdown (only app.
10% lower).
Uses policy data from 1,470 executive
Two mitigation
orders from the state–government
strategies (all school
websites for all 50 states and Washington closure and mandatory
DC and COVID-19-deaths from Johns
self-quarantine of
Hopkins University in a random-effecttravelers) showedspatial error panel model to estimate thepositive (more deaths)
effect of nine NPIs (SIPO, strengthened
impact on COVID-19SIPO, public school closure, all school
deaths per 10,000. Six
closure, large-gathering ban of more than mitigation strategies
10 people, any gathering ban,
(SIPO, public school
restaurant/bar limit to dining out only,
closure, large gathering
nonessential business closure, andbans (>10), any
mandatory self-quarantine of travelers) on gathering ban,
COVID-19 deaths.
restaurant/bar limit todining out only, andnonessential business19

5. Comments

Gibson use the word "lockdown" as synonym
for SIPO (writes "technically, governmentordered community quarantine")

There is little documentation in the study (e.g.
no tables with estimates).

Only conclude on NPIs which reduce mortality.
However, the conclusion is based on one-tailed
tests, which means that all positive estimates(more deaths) are deemed insignificant. Thus, intheir mortality-specification (Table 3, Proportionof Cumulative Deaths Over the Population), theestimate of all school closures (.204) andmandatory self-quarantine of travelers (0.363) isdeemed insignificant based on schools CI [.029,.379] and quarantine CI [.193, .532]. We
believe, these results should be interpreted as asignificant increase in mortality, and that theseresults should have been part of theirconclusion.

1. Study (Author &title)

2.
3. Description
MeasureHale et al. (2020); "Global
assessment of therelationship between
government response
measures and COVID-19
deaths"COVID19
mortality

Hunter et al. (2021);"Impact of nonpharmaceuticalinterventions against
COVID-19 in Europe: A
quasi-experimental nonequivalent group andtime-series"

COVID19
mortality

Langeland et al. (2021);"The Effect of State Level
COVID-19 Stay-at-Home
Orders on Death Rates"

COVID19
mortality

Leffler et al. (2020);"Association of countrywide coronavirus
mortality withdemographics, testing,lockdowns, and public
wearing of masks"

COVID19
mortality

Mccafferty and Ashley
Other(2021); "Covid-19 Social
Distancing Interventions
by Statutory Mandate andTheir Observational

4. Results

closure) did not show
any impact (Table 3,"Proportion of
Cumulative Deaths
Over the Population).
Uses the OxCGRT stringency and COVID- Finds that higher
19-deaths from the European Centre forstringency in the past
Disease Prevention and Control for 170
leads to a lower growthcountries. Estimates both cross-sectional
rate in the present, withmodels in which countries are the unit ofeach additional point ofanalysis, as well as longitudinal models on stringency
time-series panel data with country-day
corresponding to aas the unit of analysis (including models
0.039%-point reductionthat use both time and country fixedin daily deaths growtheffects).
rates six weeks later.
Uses death data from the European
Finds that mass
Centre for Disease Prevention andgathering restrictions
Control (ECDC) and NPI-data from theand initial businessInstitute of Health Metrics and Evaluation. closures (businesses
Argues that they use a quasi-experimental such as entertainment
approach to identify the effect of NPIs
venues, bars andbecause no analyzed intervention was
restaurants) reduces theimposed by all European countries andnumber of deaths,
interventions were put in place atwhereas closing
different points in the development of the educational facilities
epidemics.
and issuing SIPO
increases the number ofdeaths. Finds no effectof closing non-essential
services andmandating/recommending masks (Table 3)
Estimates the effect of state-level
Finds no significant
lockdowns on COVID-19 deaths using
effect of SIPO on themultiple quasi-Poisson regressions withnumber of deaths after
lockdown time length as the explanatory
2-4, 4-6 and 6+ weeks.
variable. Does not specify how lockdown
is defined and what their data sources are.
Use COVID-19 deaths from Worldometerand info about NPIs (mask/mask
recommendations, international travel
restrictions and lockdowns (defined as any
closure of schools or workplaces, limits on
public gatherings or internal movement, or
stay-at-home orders) from Hale et al.
(2020) for 200 countries to estimate theeffect of the duration of NPIs on thenumber of deaths.

Use data from 27 U.S. states and 12
European countries to analyze the effectof NPIs on peak morality rate using
general linear mixed effects modelling.

20

Finds that masking(mask
recommendations)
reduces mortality. For
each week that masks
were recommended theincrease in per-capita
mortality was 8.1%
(compared to 55.7%
increase when masks
were notrecommended). Finds
no significant effect of
the number of weeks
with internal lockdowns
and international travel
restrictions (Table 2).
Finds that no mandate(school closures,
prohibition on mass
gatherings, businessclosures, stay at home

5. Comments

Finds an effect of closing educational facilities
and non-essential services after 1-7 days before
lockdown could possibly have an effect on thenumber of deaths. This may indicate that otherfactors are driving their results.

They write that "6+ weeks of lockdown is the
only setting where the odds of dying are
statistically higher than in the no lockdown
case.” However, all estimates are insignificant inTable C. Looks as if lockdown duration may
cause a causality problem, because politicians
may be less likely to ease restrictions when
there are many cases/deaths.
Their "mask recommendation" category includes
some countries, where masks were mandated(see Supplemental Table A1) and may (partially)
capture the effect of mask mandates. Looks at
duration which may cause a causality problem,
because politicians may be less likely to ease
restrictions when there are many cases/deaths.

1. Study (Author &title)

2.
3. Description
Measure4. Results

Correlation to Mortality in
the United States andEurope"Pan et al. (2020); "Covid19: Effectiveness of nonpharmaceuticalinterventions in theunited states before
phased removal of social
distancing protections
varies by region"

COVID19
mortality

Pincombe et al. (2021);"The effectiveness ofnational-levelcontainment and closure
policies across income
levels during the COVID19 pandemic: an analysisof 113 countries"

COVID19
mortality

Sears et al. (2020); "Arewe #stayinghome to
Flatten the Curve?"

COVID19
mortality

Uses county-level data for all U.S. states.
Mortality is obtained from Johns Hopkins,
while policy data are obtained fromofficial governmental websites.
Categorizes 12 policies into 4 levels of
disease control; Level 1 (low) - State ofEmergency; Level 2 (moderate) - school
closures, restricting access (visits) tonursing homes, or closing restaurants andbars; Level 3 (high) - non-essential
business closures, suspending non-violent
arrests, suspending elective medical
procedures, suspending evictions, or
restricting mass gatherings of at least 10
people; and Level 4 (aggressive) sheltering in place / stay-at-home, publicmask requirements, or travel restrictions.
Use stepped-wedge cluster randomized
trial (SW-CRT) for clustering and negative
binomial mixed model regression.
Uses daily data for 113 countries on
cumulative COVID-19 death counts over
130 days between February 15, 2020,and June 23, 2020, to examine changes inmortality growth rates across the WorldBank’s income group classifications
following shelter-in-place
recommendations or orders (they use one
variable covering both recommendations
and orders).
Uses cellular location data from all 50
states and the District of Columbia toinvestigate mobility patterns during the
pandemic across states and time. Adding
COVID-19 death tolls and the timing of
SIPO for each state they estimate theeffect of stay-at-home policies on
COVID-19 mortality.

21

orders, severe travel
restrictions, and closureof non-essential
businesses) was
effective in reducing
the peak COVID-19
mortality rate.
Concludes that only
(duration of, seecomment in next
column) level 4
restrictions areassociated with reduced
risk of death, with anaverage 15% decline in
the COVID-19 death
rate per day.
Implementation of level
3 and level 2
restrictions increased
death rates in 6 of 6
regions, while longer
duration increased
death rates in 5 of 6
regions.
Finds that shelter-inplace
recommendations/orde
rs reduces mortality
growth rates in highincome countries(although insignificant)but increases growth
rates in countries inother income groups.
Find that SIPOs lower
deaths by 0.13- 0.17
per 100,000 residents,
equivalent to death
rates 29-35% lower
than in the absence ofpolicies. However,these estimates are
insignificant at a 95%
confidence interval (seeTable 4). The study also
finds reductions in
activity levels prior to
mandates. Human
encounter rate fell by
63 percentage points
and nonessential visitsby 39 percentage
points relative to preCOVID-19 levels, prior
to any state
implementing astatewide mandate

5. Comments

They focus on the negative estimate of durationof Level 4. However, their implementation
estimate is large and positive, and the combined
effect of implementation and duration is
unclear.

In the abstract the authors state that death
rates would be 42-54% lower than in theabsence of policies. However, this includes
averted deaths due to pre-mandate social
distancing behavior (p. 6). The effect of SIPO isa reduction in deaths by 29%-35% compared toa situation without SIPO but with pre-mandatesocial distancing. These estimates are
insignificant at a 95% confidence interval.

1. Study (Author &title)

2.
3. Description
Measure4. Results

Shiva and Molana (2021);"The Luxury of
Lockdown"

COVID19
mortality

A stricter lockdown (1stringency point)
reduces deaths by 0,1%
after 4 weeks. After 8
weeks the effect isinsignificant.

Spiegel and Tookes(2021); "Businessrestrictions and Covid-19
fatalities"

COVID19
mortality

Stockenhuber (2020);"Did We Respond Quickly
Enough? How PolicyImplementation Speed in
Response to COVID-19
Affects the Number ofFatal Cases in Europe"
Stokes et al. (2020); "Therelative effects of nonpharmaceuticalinterventions on early
Covid-19 mortality:
natural experiment in 130
countries"

COVID19
mortality

COVID19
mortality

Uses COVID-19-deaths and OxCGRT
stringency from 169 countries to estimate
the effect of lockdown on the number ofdeaths 1-8 weeks later. Finds that stricter
lockdowns reduce COVID-19-deaths 4
weeks later (but insignificant 8 weeks
later) and have the greatest effect in highincome countries. Finds no effect ofworkplace closures in low-income
countries.
Use data for every county in the UnitedStates from March through December
2020 to estimate the effect of various
NPIs on the COVID-19-deaths growthrate. Derives causality by 1) assuming thatstate regulators primarily focus on thestate’s most populous counties, so state
regulation in smaller counties can beviewed as a quasi randomized experiment,
and 2) conducting county pair analysis,
where similar counties in different states(and subject to different state policies) arecompared.

Uses data for the number of COVID‐19
infections and deaths and policy
information for 24 countries fromOxCGRT to estimate the effect of stricter
lockdowns on the number of deaths using
principal component analysis and ageneralized linear mixed model.
Uses daily Covid-19 deaths for 130
countries from the European Centre for
Disease Prevention and Control (ECDC)
and daily policy data from the Oxford
COVID-19 Government Response Tracker(OxCGRT). Looks at all levels of
restrictions for each of the nine subcategories of the OxCGRT stringency
index (school, work, events, gatherings,
transport, SIPO, internal movement,
travel).

22

5. Comments

Finds that someinterventions (e.g. mask
mandates, restaurant
and bar closures, gym
closures, and high-riskbusiness closures)
reduces mortality
growth, while otherinterventions (closuresof low- to medium-riskbusinesses and personalcare/spa services) did
not have an effect andmay even haveincreased the numberof deaths.
Finds no significant
effect of stricter
lockdowns on thenumber of fatalities(Table 4).

In total they analyze the lockdown effect of 21
variables. 14 of 21 estimates are significant, andof these 6 are negative (reduces deaths) while 8
are positive (increases deaths). Some results arefar from intuitive. E.g. mask recommendations
increases deaths by 48% while mask mandates
reduces deaths by 12%, and closing restaurants
and bars reduces deaths by 50%, while closing
bars but not restaurants only reduces deaths by
5%.

Of the nine subcategories in the
OxCGRT stringency
index, only travel
restrictions are
consistently significant(with level 2"Quarantine arrivals
from high-risk regions"having the largest
effect, and the strictest
level 4 "Total borderclosure" having thesmallest effect).
Restrictions on verylarge gatherings(>1,000) has a large
significant negative(fewer deaths) effect,
while the effect ofstricter restrictions ongatherings are
insignificant. Authors
recommend that theclosing of schools (level1) has a very large (inabsolute terms it's twice
the effect of border
quarantines) positive

Their results are counter intuitive andsomewhat inconclusive. Why does limiting verylarge gatherings (>1,000) work, while stricter
limits do not? Why do recommending school
closures cause more deaths? Why is the effectof border closures before 1st death insignificant,while the effect of closing borders after 1st
death is significant (and large)? And why does
quarantining arrivals from high-risk regions work
better than total border closures? With 23
estimated parameters in total these counter
intuitive and inconclusive results could becaused by multiple test bias (we correct for thisin the meta-analysis), but may also be caused byother factors such as omitted variable bias.

Groups data on lockdown strictness into fourgroups and lose significant information andvariation.

1. Study (Author &title)

2.
3. Description
Measure4. Results

Toya and Skidmore
COVID(2020); "A Cross-Country 19
Analysis of themortality
Determinants of Covid-19
Fatalities"

Uses COVID-19-deaths and lockdown
info from various sources from 159
countries in a cross-country event study.
Controls for country specifics by including
socio-economic, political, geographic, andpolicy information. Finds little evidence
for the efficacy of NPIs.

Tsai et al. (2021);"Coronavirus Disease2019 (COVID-19)
Transmission in the
United States Before
Versus After Relaxationof Statewide Social
Distancing Measures"

Uses data for NPIs that were
implemented and/or relaxed in U.S. states
between 10 March and 15 July 2020.
Using segmented linear regression, they
estimate the extent to which relaxation ofsocial distancing affected epidemic
control, as indicated by the time-varying,state-specific effective reproduction
number (Rt). Rt is based on death tolls.

Reproduction rate,Rt

effect (more deaths)
while stricter
interventions on
schools have nosignificant effect.
Required cancelling of
public events also has asignificant positive(more deaths) effect.
We focus on their 1438 days results, as they
catch the longest timeframe (their 0-24 day
model returns mostly
insignificant results).
Complete travel
restrictions prior to
April 2020 reduced
deaths by -0.226 per
100.000 by April 1st2021, while mandatory
national lockdown prior
to April 2020 increased
deaths by 0.166 byApril 1st 2021.
Recommended local
lockdowns reduced
deaths but results are
based on one
observation. Partial
travel restrictions,mandatory local
lockdowns andrecommended national
lockdowns did not havea significant effect on
deaths.
Finds that in the 8
weeks prior to relaxing
NPIs, Rt was declining,
while after relaxation Rt
started to increase.

5. Comments

The study looks at the lockdown status prior to
April 2020 and the effect on deaths thefollowing year (until April 1st 2021). The authors
state this is to reduce concerns about
endogeneity but do not explain why the
lockdowns in the spring of 2020 are a good
instrument for lockdowns during later waves
are.

Their Figure 1 shows that Rt on averageincreases app. 10 days before relaxation, which
could indicate that other factors (omitted
variables) affect the results.

Note: All comments on the significance of estimates are based on a 5% significance level unless otherwise stated.

It is difficult to make a conclusion based on the overview in Table 1. Is -0.073 to -0.326
deaths/million per stringency point, as estimated by Ashraf (2020), a large or a small effectrelative to. the 98% reduction in mortality predicted by the study published by the Imperial
College London (Ferguson et al. (2020). This is the subject for our meta-analysis in the next
section. Here, it turns out that -0.073 to -0.326 deaths/million per stringency point is a relatively
modest effect and only corresponds to a 2.4% reduction in COVID-19 mortality on average inthe U.S. and Europe.
23

4

Meta-analysis: The impact of lockdowns on COVID-19 mortality

We now turn to the meta-analysis, where we focus on the impact of lockdowns on COVID-19
mortality.
In the meta-analysis, we include 24 studies in which we can derive the relative effect oflockdowns on COVID-19 mortality, where mortality is measured as COVID-19-related deaths
per million. In practice, this means that the studies we included estimate the effect of lockdowns
on mortality or the effect of lockdowns on mortality growth rates, while using a counterfactual
estimate.26
Our focus is on the effect of compulsory non-pharmaceutical interventions (NPI), policies that
restrict internal movement, close schools and businesses, and ban international travel, among
others. We do not look at the effect of voluntary behavioral changes (e.g. voluntary mask
wearing), the effect of recommendations (e.g. recommended mask wearing), or governmental
services (voluntary mass testing and public information campaigns), but only on mandated NPIs.
The studies we examine are placed in three categories. Seven studies analyze the effect of stricter
lockdowns based on the OxCGRT stringency indices, 13 studies analyze the effect of SIPOs (6studies only analyze SIPOs, while seven analyze SIPOs among other interventions), and 11
studies analyze the effect of specific NPIs independently (lockdown vs. no lockdown).27 Each ofthese categories is handled so that comparable estimates can be made across categories. Below,
we present the results for each category and show the overall results, as well as those based on
various quality dimensions.
Quality dimensions
We include quality dimensions because there are reasons to believe that can affect a study’s
conclusion. Below we describe the dimensions, as well as our reasons to believe that they arenecessary to fully understand the empirical evidence.
•

Peer-reviewed vs. working papers: We distinguish between peer-reviewed studies andworking papers as we consider peer-reviewed studies generally being of higher quality thanworking papers.28•

Long vs. short time period: We distinguish between studies based on long time periods (withdata series ending after May 31, 2020) and short time periods (data series ending at or before
May 31, 2020), because the first wave did not fully end before late June in the U.S. andEurope. Thus, studies relying on short data periods lack the last part of the first wave andmay yield biased results if lockdowns only “flatten the curve” and do not prevent deaths.

26

As a minimum requirement, one needs to know the effect on the top of the curve.
The total is larger than 21 because the 11 SIPO studies include seven studies which look at multiple measures.
28
Vetted papers from CEPR Covid Economics are considered as working papers in this regard.
27

24

•

No early effect on mortality: On average, it takes approximately three weeks from infection
to death.29 However, several studies find effects of lockdown on mortality almost
immediately. Fowler et al. (2021) find a significant effect of SIPOs on mortality after justfour days and the largest effect after 10 days. An early effect may indicate that other factors(omitted variables) drive the results, and, thus, we distinguish between studies which find an
effect on mortality sooner than 14 days after lockdown and those that do not.30 Note thatmany studies do not look at the short term and thus fall into the latter category by default.

•

Social sciences vs. other sciences: While it is true that epidemiologists and researchers innatural sciences should, in principle, know much more about COVID-19 and how it spreads
than social scientists, social scientists are, in principle, experts in evaluating the effect ofvarious policy interventions. Thus, we distinguish between studies published by scholars insocial sciences and by scholars from other fields of research. We perceive the former as
being better suited for examining the effects of lockdowns on mortality. For each study, we
have registered the research field for the corresponding author’s associated institute (e.g., fora scholar from “Institute of economics” research field is registered as “Economics”). Whereno corresponding author was available, the first author has been used. Afterwards, allresearch fields have been classified as either from the “Social Science” or “Other.””31

We also considered including a quality dimension to distinguish between studies based on excess
mortality and studies based on COVID-19 mortality, as we believe that excess mortality ispotentially a better measure for two reasons. First, data on total deaths in a country is far more
precise than data on COVID-19 related deaths, which may be both underreported (due to lack oftests) or overreported (because some people die with – but not because of – COVID-19).
Secondly, a major purpose of lockdowns is to save lives. To the extend lockdowns shift deaths
from COVID-19 to other causes (e.g. suicide), estimates based on COVID-19 mortality will
overestimate the effect of lockdowns. Likewise, if lockdowns save lives in other ways (e.g. fewer
traffic accidents) lockdowns’ effect on mortality will be underestimated. However, as only one

Leffler et al. (2020) writes, “On average, the time from infection with the coronavirus to onset of symptoms is 5.1
days, and the time from symptom onset to death is on average 17.8 days. Therefore, the time from infection todeath is expected to be 23 days.” Meanwhile, Stokes et al. (2020) writes that “evidence suggests a mean lag
between virus transmission and symptom onset of 6 days, and a further mean lag of 18 days between onset ofsymptoms and death.”
30
Some of the authors are aware of this problem. E.g. Bjørnskov (2021a) writes ”when the lag length extends tothree or fourth weeks, that is, the length that is reasonable from the perspective of the virology of Sars-CoV-2, theestimates become very small and insignificant” and ”these results confirm the overall pattern by being negativeand significant when lagged one or two weeks (the period when they cannot have worked) but turning positive andinsignificant when lagged four weeks.”
31
Research fields classified as social sciences were economics, public health, management, political science,
government, international development, and public policy, while research fields not classified as social scienceswere ophthalmology, environment, medicine, evolutionary biology and environment, human toxicology,
epidemiology, and anesthesiology.
29

25

of the 34 studies (Bjørnskov (2021a)) is based on excess mortality, we are unfortunately forced
to disregard this quality dimension.
Meta-data used for our quality dimensions as well as other relevant information are shown in
Table 2.

Table 2: Metadata for the studies included in the meta-analysis
1. Study (Author & title)

2. Included
in metaanalysis

3.
Publication
status4. End ofdata
period5.
Earliest
effect

6. Field of
research

7.
Lockdown
measure8.
Geographicalcoverage

Alderman and Harjoto (2020); "COVID-19:
U.S. shelter-in-place orders anddemographic characteristics linked tocases, mortality, and recovery rates"
Aparicio and Grossbard (2021); "Are Covid
Fatalities in the U.S. Higher than in the EU,
and If so, Why?"
Ashraf (2020); "Socioeconomic conditions,
government interventions and healthoutcomes during COVID-19"
Auger et al. (2020); "Association between
statewide school closure and COVID-19
incidence and mortality in the U.S."
Berry et al. (2021); "Evaluating the effectsof shelter-in-place policies during the
COVID-19 pandemic"
Bjørnskov (2021a); "Did Lockdown Work?
An Economist's Cross-Country
Comparison"Blanco et al. (2020); "Do Coronavirus
Containment Measures Work? Worldwide
Evidence"
Bonardi et al. (2020); "Fast and local: How
did lockdown policies affect the spread andseverity of the covid-19"
Bongaerts et al. (2021); "Closed forbusiness: The mortality impact of businessclosures during the Covid-19 pandemic"
Chaudhry et al. (2020); "A country level
analysis measuring the impact ofgovernment actions, country preparedness
and socioeconomic factors on COVID-19
mortality and related health outcomes"
Chernozhukov et al. (2021); "Causal impact
of masks, policies, behavior on early covid19 pandemic in the U.S."
Chisadza et al. (2021); "Government
Effectiveness and the COVID-19
Pandemic"
Dave et al. (2021); "When Do Shelter-inPlace Orders Fight Covid-19 Best? Policy
Heterogeneity Across States and Adoption
Time"
Dergiades et al. (2020); "Effectiveness ofgovernment policies in response to the
COVID-19 outbreak"
Fakir and Bharati (2021); "Pandemic catch22: The role of mobility restrictions andinstitutional inequalities in halting the
spread of COVID-19"

Yes

Peer-review

11-Jun-20

n/aEconomics (Social
science)

SIPO

United States

Yes

Peer-review

22-Jul-20

n/a

Economics (Social
science)Specific NPIs

Europe andUnited States

Yes

WP

20-May20n/aEconomics (Social
science)

Stringency

World

Yes

Peer-review

07-May20>21 days

Medicine (Other)Specific NPIs

United States

Yes

Peer-review

30-May20

8-14 days Public policy (Social
science)

SIPO

United States

Yes

Peer-review

30-Jun-20<8 days

Stringency

EuropeNo

WP

31-Aug-20 8-14 days Economics (Social
science)Specific NPIs

World

Yes

WP

13-Apr-20<8 daysSpecific NPIs

World

Yes

Peer-review

13-Apr-20

8-14 days Management(Social science)Specific NPIs

One country

Yes

Peer-review

01-Apr-20

n/a

Anesthesiology(Other)Specific NPIs

World

Yes

Peer-review

03-Aun-20 n/a

Economics (Social
science)Specific NPIs

United States

Yes

Peer-review

01-Sep-20

n/aEconomics (Social
science)

Stringency

World

Yes

Peer-review

20-Apr-20

Finds noeffectEconomics (Social
science)

SIPO

United States

No

WP

30-Apr-20

n/aManagement(Social science)

Stringency

World

No

Peer-review

30-Jul-20<8 daysEconomics (Social
science)

Stringency

World

26

Economics (Social
science)

Economics (Social
science)1. Study (Author & title)

2. Included
in metaanalysis

3.
Publication
status4. End ofdata
period5.
Earliest
effect

6. Field of
research

7.
Lockdown
measure8.
Geographicalcoverage

Fowler et al. (2021); "Stay-at-home orders
associate with subsequent decreases in
COVID-19 cases and fatalities in the
United States"
Fuller et al. (2021); "Mitigation Policies and
COVID-19–Associated Mortality — 37
European Countries, January 23–June 30,
2020"

Yes

Peer-review

07-May20<8 days

Public Health(Social science)

SIPO

United States

Yes

WP

30-Jun-20

n/aEpidemiology(Other)Stringency

EuropeGibson (2020); "Government mandated
lockdowns do not reduce Covid-19 deaths:
implications for evaluating the stringent
New Zealand response"
Goldstein et al. (2021); "Lockdown Fatigue:
The Diminishing Effects of Quarantines on
the Spread of COVID-19 "Guo et al. (2021); "Mitigation Interventionsin the United States: An Exploratory
Investigation of Determinants and Impacts"
Hale et al. (2020); "Global assessment of
the relationship between government
response measures and COVID-19 deaths"
Hunter et al. (2021); "Impact of nonpharmaceutical interventions against
COVID-19 in Europe: A quasi-experimental
non-equivalent group and time-series"
Langeland et al. (2021); "The Effect of State
Level COVID-19 Stay-at-Home Orders on
Death Rates"
Leffler et al. (2020); "Association ofcountry-wide coronavirus mortality withdemographics, testing, lockdowns, andpublic wearing of masks"
Mccafferty and Ashley (2021); "Covid-19
Social Distancing Interventions by
Statutory Mandate and Their Observational
Correlation to Mortality in the UnitedStates and Europe"
Pan et al. (2020); "Covid-19: Effectivenessof non-pharmaceutical interventions in theunited states before phased removal ofsocial distancing protections varies byregion"Pincombe et al. (2021); "The effectivenessof national-level containment and closure
policies across income levels during the
COVID-19 pandemic: an analysis of 113
countries"
Sears et al. (2020); "Are we #stayinghome
to Flatten the Curve?"

Yes

Peer-review

01-Jun-20

Finds noeffectEconomics (Social
science)

SIPO

United States

Yes

WP

31-Dec-20 <8 days

Stringency

World

Yes

Peer-review

07-Apr-20

n/aInternational
Development(Social science)
Social work (Social
science)Specific NPIs

United States

No

WP

27-May20

n/aGovernment (Social
science)

Stringency

World

No

Peer-review

24-Apr-20<8 days

Medicine (Other)Specific NPIs

EuropeNo

WP

Not
specifiedFinds noeffectPolitical Science(Social science)

Other

United States

Yes

Peer-review

09-May20n/aOphthalmology(Other)Specific NPIs

World

No

Peer-review

12-Apr-20

Finds noeffectOphthalmology(Other)Specific NPIs

Europe andUnited States

No

WP

29-May20n/aEnvironment(Other)Specific NPIs

United StatesNo

Peer-review

23-Jun-20

n/aHealth Science(Social science)

SIPO

World

Yes

WP

29-Apr-20

Finds noeffectEconomics (Social
science)

SIPO

United States

Shiva and Molana (2021); "The Luxury of
Lockdown"

Yes

Peer-review

08-Jun-20

15-21
days

Government (Social
science)

Stringency

World

Spiegel and Tookes (2021); "Businessrestrictions and Covid-19 fatalities"

Yes

Peer-review

31-Dec-20 <8 days

Management(Social science)Specific NPIs

United States

Stockenhuber (2020); "Did We Respond
Quickly Enough? How PolicyImplementation Speed in Response to
COVID-19 Affects the Number of Fatal
Cases in Europe"
Stokes et al. (2020); "The relative effects of
non-pharmaceutical interventions on early

Yes

Peer-review

12-Jul-20

n/aEvolutionary
Biology andEnvironment(Other)Stringency

EuropeYes

WP

01-Jun-20

n/aEconomics (Social
science)Specific NPIs

World

27

1. Study (Author & title)

2. Included
in metaanalysis

3.
Publication
status4. End ofdata
period5.
Earliest
effect

6. Field of
research

7.
Lockdown
measure8.
Geographicalcoverage

Yes

WP

01-Apr-21

n/aEconomics (Social
science)Specific NPIs

World

No

Peer-review

15-Jul-20<8 days

Psychiatry (Social
science)Specific NPIs

United States

Covid-19 mortality: natural experiment in
130 countries"
Toya and Skidmore (2020); "A CrossCountry Analysis of the Determinants of
Covid-19 Fatalities"
Tsai et al. (2021); "Coronavirus Disease2019 (COVID-19) Transmission in the
United States Before Versus After
Relaxation of Statewide Social Distancing
Measures"Note: Research fields classified as social sciences were economics, public health, health science, management, political science, government,international development, and public policy, while research fields not classified as social sciences were ophthalmology, environment,
medicine, evolutionary biology and environment, human toxicology, epidemiology and anesthesiology.

Interpreting and weighting estimates
The estimates used in the meta-analysis are not always readily available in the studies shown in
Table 2. In Appendix B Table 9, we describe for each paper how we interpret the estimates andhow they are converted to a common estimate (the relative effect of lockdowns on COVID-19
mortality) which is comparable across all studies.
Following Paldam (2015) and Stanley and Doucouliagos (2010), we also convert standard
errors32 and use the precision of each estimate (defined as 1/SE) to calculate the precisionweighted average of all estimates and present funnel plots. The precision-weighted average is ourprimary indicator of the efficacy of lockdowns, but we also report arithmetic averages andmedians in the meta-analysis.
In the following sections, we present the meta-analysis for each of the three groups of studies(stringency index-studies, SIPO-studies, and studies analyzing specific NPIs).
4.1

Stringency index studies

Seven eligible studies examine the link between lockdown stringency and COVID-19 mortality.
The results from these studies, converted to common estimates, are presented in Table 3 below.
All studies are based on the COVID-19 Government Response Tracker’s (OxCGRT) stringency
index of Oxford University’s Blavatnik School of Government (Hale et al. (2020)).
The OxCGRT stringency index neither measures the expected effectiveness of the lockdowns
nor the expected costs. Instead, it describes the stringency based on nine equally weighted
parameters.33 Many countries followed similar patterns and almost all countries closed schools,

32Standard errors are converted such that the t-value, calculated based on common estimates and standard errors, isunchanged. When confidence intervals are reported rather than standard errors, we calculate standard errors usingt-distribution with ∞ degrees of freedom (i.e. 1.96 for 95% confidence interval).
33
The nine parameters are "C1 School closing,” "C2 Workplace closing,” "C3 Cancel public events,” "C4
Restrictions on gatherings,” "C5 Close public transport,” "C6 Stay at home requirements,” "C7 Restrictions on
internal movement,” "C8 International travel controls" and "H1 Public information campaigns.” The latter, "H128

while only a few countries issued SIPOs without closing businesses. Hence, it is reasonable toperceive the stringency index as continuous, although not necessarily linear. The index includes
recommendations (e.g. “workplace closing” is 1 if the government recommends closing (or workfrom home), cf. Hale et al. (2021)), but the effect of including recommendations in the index isprimarily to shift the index parallelly upward and should not alter the results relative to our focus
on mandated NPIs. It is important to note that the index is not perfect. As pointed out by Book
(2020), it is certainly possibly to identify errors and omissions in the index. However, the index
is objective and unbiased and as such, useful for cross-sectional analysis with several
observations, even if not suitable for comparing the overall strictness of lockdowns in two
countries.
Since the studies examined use different units of estimates, we have created common estimatesfor Europe and United States to make them comparable. The common estimates show the effectof the average lockdown in Europe and United States (with average stringencies of 76 and 74,
respectively, between March 16th and April 15th, 2020, compared to a policy based solely on
recommendations (stringency 44)). For example, Ashraf (2020) estimates that the effect ofstricter lockdowns is -0.073 to -0.326 deaths/million per stringency point. We use the average ofthese two estimates (-0.200) in the meta-analysis (see Table 9 in Appendix B for a description
for all studies). The average lockdown in Europe between March 16th and April 15th, 2020, was
32 points stricter than a policy solely based on recommendations (76 vs. 44). In United States, it
was 30 points. Hence, the total effect of the lockdowns compared to the recommendation policy
was -6.37 deaths/million in Europe (32 x -0.200) and -5.91 deaths/million in United States. With
populations of 748 million and 333 million, respectively the total effect as estimated by Ashraf(2020) is 4,766 averted COVID-19 deaths in Europe and 1,969 averted COVID-19 deaths inUnited States. By the end of the study period in Ashraf (2020), which is May 20, 2020, 164,600
people in Europe and 97,081 people in the United States had died of COVID-19. Hence, the4,766 averted COVID-19 deaths in Europe and the 1,969 averted COVID-19 deaths in the
United States corresponds to 2.8% and 2.0% of all COVID-19 deaths, respectively, with anarithmetic average of 2.4%. Our common estimate is thus -2.4%, cf. Table 3. So, this means that
Ashraf (2020) estimates that without lockdowns, COVID-19 deaths in Europe would have been
169,366 and COVID-19 deaths in the U.S. would have been 99,050. Our approach is not
unproblematic. First of all, the level of stringency varies over time for all countries. We use thestringency between March 16th and April 15th, 2020 because this period covers the main part of
the first wave which most of the studies analyze. Secondly, OxCGRT has changed the index overtime and a 10-point difference today may not be exactly the same as a 10-point difference when
the studies were finalized. However, we believe these problems are unlikely to significantly alter
our results.

Public information campaigns,” is not an intervention following our definition, as it is not a mandatory
requirement. However, of 97 European countries and U.S. States in the OxCGRT database, only Andorra, Belarus,
Bosnia and Herzegovina, Faeroe Islands, and Moldova – less than 1.6% of the population – did not get the
maximum score by March 20, 2020, so the parameter simply shifts the index parallelly upward and should nothave notable impact on the analyzes.

29

Table 3 demonstrates that the studies find that lockdowns, on average, have reduced COVID-19
mortality rates by 0.2% (precision-weighted). The results yield a median of -2.4% and anarithmetic average of -7.3%. Only one of the seven studies, Fuller et al. (2021), finds asignificant and (relative to the effect predicted in studies like Ferguson et al. (2020)) substantial
effect of lockdowns (-35%). The other six studies find much smaller effects. Hence, based on thestringency index studies, we find little to no evidence that mandated lockdowns in Europe andthe United States had a noticeable effect on COVID-19 mortality rates. And, as will be discussed
in the next paragraph, the fifth column of Table 3 displays the number of quality dimensions (outof 4) met by each study.

Table 3: Overview of common estimates from studies based on stringency indexes
Effect on COVID-19 mortality
Bjørnskov (2021)

Estimate(Estimated Averted Deaths/
Total Deaths)-0.3%

Standard
error

Weight(1/SE)

Qualitydimension
s

0.8%

119

3

0.4%

248

4

Shiva and Molana (2021)-4.1%

Stockenhuber (2020)*

0.0%

n/a

n/a

3

Chisadza et al. (2021)

0.1%

0.0%

7,390

4

Goldstein et al. (2021)

-9.0%

3.8%

26

2

Fuller et al. (2021)-35.3%

9.1%

11

2

Ashraf (2020)

-2.4%

0.4%

256

2

Precision-weighted average (arithmetic average /median)-0.2% (-7.3%/-2.4%)

Note: The table shows the estimates for each study converted to a common estimate, i.e. the implied effect on COVID-19
mortality in Europe and United States. A negative number corresponds to fewer deaths, so -5% means 5% lover COVID-19
mortality. For studies which report estimates in deaths per million, the common estimate is calculated as: (COVID-19 mortality
with "common area's" policy) / (COVID-19 mortality with recommendation policy) -1, where (COVID-19 mortality withrecommendation policy) is calculated as ((COVID-19 mortality with "common area's" policy) - Estimate x Difference instringency x population). Stringencies in Europe and United States are equal to the average stringency from March 16th to April
15th 2020 (76 and 74 respectively) and the stringency for the policy based solely on recommendations is 44 following Hale et al.
(2020). For the conversion of other studies see Table 9 in appendix B.
* It is not possible to calculate a common estimate for Stockenhuber (2020). When calculating arithmetic average / median, thestudy is included as 0%, because estimates are insignificant and signs of estimates are mixed (higher strictness can cause both
lower and higher COVID-19 mortality).

We now turn to the quality dimensions. Table 4 presents the results differentiated by the four
quality dimensions. Two studies, Shiva and Molana (2021) and Chisadza et al. (2021), meet all
quality dimensions. The precision-weighted average for these studies is 0.0%, meaning that
lockdowns had no effect on COVID-19 mortality. Two studies live up to 3 of 4 quality
dimensions (Bjørnskov (2021a) and Stockenhuber (2020)). The precision-weighted average forthese studies is -0.3%, meaning that lockdowns reduced COVID-19 mortality by 0.3%. Threestudies lack at least two quality dimensions.34 These studies find that lockdowns reduce COVID19 mortality by 4.2%. To sum up, we find that the studies that meet at least 3 of 4 quality
measures find that lockdowns have little to no effect on COVID-19 mortality, while studies that34

In fact, the working papers by P. Goldstein et al. (2021), Fuller et al. (2021) and Ashraf (2020) all lack exactly
two quality parameters.

30

meet 2 of 4 quality measures find a small effect on COVID-19 mortality. These results are far
from those estimated with the use of epidemiological models, such as the Imperial College
London (Ferguson et al. (2020).

Table 4: Overview of common estimates split on quality dimensions for studies based on
stringency indexes
Precision-weighted
average*Arithmetic
average

MedianPeer-reviewed [4]

0.0%

-1.1%

-0.2%

Working paper [3]-4.2%-15.6%-9.0%

Data series ends after 31 May 2020 [6]

-0.1%

-8.1%

-0.2%

Data series ends before 31 May 2020 [1]

-2.4%

-2.4%

-9.0%

Does not find an effect within the first 14 days (including n/a) [5]

-0.2%

-8.3%

-2.4%

Finds effect within the first 14 days [2]

-1.9%

-4.7%

-4.7%

Social sciences [5]

-0.1%

-3.1%

-2.4%

Other sciences [2]-35.3%-17.7%-17.7%

4 of 4 quality dimensions [2]

0.0%

-2.0%

-2.0%

3 of 4 quality dimensions [2]

-0.3%

-0.2%

-0.2%

2 of 4 quality dimensions or fewer [3]-4.2%-15.6%-9.0%

Values show effect on COVID-19 mortality
Peer-reviewed vs. working papers

Long vs. short time period

No early effect on mortality

Social sciences vs. other sciences

Note: The table shows the common estimate as described in Table 3 for each quality dimension. The number of studies in eachcategory is in square brackets. * The precision-weighted average does not include studies where no common standard error isavailable, cf. Table 3.

Figure 5 shows a funnel plot for the studies in Table 3, except Stockenhuber (2020), wherecommon estimate standard errors cannot be derived. Chisadza et al. (2021) has a far higher
precision than the other studies (1/SE is 7,398 and the estimate is 0.1%)35, and there areindications that the estimate from Fuller et al. (2021) (the bottom left) is an imprecise outlier.36
Figure 5 The plot also shows that the studies with at least 3 of 4 quality dimensions are centeredaround zero and generally have higher precision than other studies.

35
36

Excluding Chisadza et al. (2021) from the precision-weighted average changes the average to -3.5%.
Excluding Fuller et al. (2021) from the precision-weighted average only marginally changes the average because
the precision is very low.

31

Figure 5: Funnel plot for estimates from studies based on stringency indexes
8,000
7,000

Weight (1/SE)

6,000
5,000
4,000
3,000
2,000
1,000
0
-40.0%-35.0%-30.0%-25.0%-20.0%-15.0%-10.0%

-5.0%

0.0%

5.0%

Estimate (Estimated Averted Deaths / Total Deaths)
2 of 4 quality dimensions or fewer

At least 3 of 4 quality dimensions

Note: The figure displays all estimates and the precision of the estimate defined as one over the standard error. Studies where
standard errors are not available are not included. Studies which live up to at least 3 of 4 quality dimensions are marked withwhite, while studies which lives up to 2 of 3 quality dimensions or less are marked with black. The vertical line illustrates theprecision-weighted average.

Overall conclusion on stringency index studies
Compared to a policy based solely on recommendations, we find little evidence that lockdowns
had a noticeable impact on COVID-19 mortality Only one study, Fuller et al. (2021), finds a
substantial effect, while the rest of the studies find little to no effect. Indeed, according tostringency index studies, lockdowns in Europe and the United States reduced only COVID-19
mortality by 0.2% on average.
In the following section we will look at the effect of SIPOs. The section follows the samestructure as this section.
4.2

Shelter-in-place order (SIPO) studies

We have identified 13 eligible studies which estimate the effect of Shelter-In-Place Orders(SIPOs) on COVID-19 mortality, cf. Table 5. Seven of these studies look at multiple NPIs ofwhich a SIPO is just one, while six studies estimate the effect of a SIPO vs. no SIPO in the
United States. According to the containment and closure policy indicators from OxCGRT, 41
states in the U.S. issued SIPOs in the spring of 2020. But usually, these were introduced after
implementing other NPIs such as school closures or workplace closures. On average, SIPOs
32

were issued 7½ days after both schools and workplaces closed, and 12 days after the first of thetwo closed. Only one state, Tennessee, issued a SIPO before schools and workplaces closed. The
10 states that did not issue SIPOs all closed schools. Moreover, of those 10 states, three closed
some non-essential businesses, while the remaining 7 closed all non-essential businesses.
Because of this, we perceive estimates for SIPOs based on U.S.-data as the marginal effect of
SIPOs on top of other restrictions, although we acknowledge that the estimates may capture theeffects of other NPI measures as well.
The results of eligible studies based on SIPOs are presented in Table 5. The table demonstrates
that the studies generally find that SIPOs have reduced COVID-19 mortality by 2.9% (on aprecision-weighted average). There is an apparent difference between studies in which a SIPO is
one of multiple NPIs, and studies in which a SIPO is the only examined intervention. The formergroup generally finds that SIPOs increase COVID-19 mortality marginally, whereas the latter
finds that SIPOs decrease COVID-19 mortality. As we will see below, this difference could be
explained by differences in the quality dimensions, and especially the time period covered byeach study.

Table 5: Overview of estimates from studies based on SIPOs
Estimate(Estimated Averted Deaths /Total Deaths)

Values show effect on COVID-19 mortality

Standard
error

Weight (1/SE)

Qualitydimensions

Studies where SIPO is one of several examined interventions and not (as) likely to capture the effect of other interventions
Chernozhukov et al. (2021)-17.7%
14.3%

7

4

Chaudhry et al. (2020) *

0.0%

n/a

n/a

2

Aparicio and Grossbard (2021)

2.6%

2.8%

35

4

Stokes et al. (2020)

0.8%

11.1%

9

3

Spiegel and Tookes (2021)

13.1%

6.6%

15

3

Bonardi et al. (2020)

0.0%

n/a

n/a

1

Guo et al. (2021)

4.6%

14.8%

4

3

Average (median) where SIPO is one of several variables

2.8% (0.5%/0.8%)

Studies where SIPO is the only examined intervention and may capture the effect of other interventions
Sears et al. (2020)-32.2%

17.6%

6

2

Alderman and Harjoto (2020)

-1.0%

0.6%

169

4

Berry et al. (2020)

1.1%

n/a

n/a

2

Fowler et al. (2021)-35.0%

7.0%

14

2

Gibson (2020)

-6.0%

24.3%

4

4

Dave et al. (2020)-40.8%

36.1%

3

3

Average (median) where SIPO is the only variable-5.1% (-19.0%/-19.1%)

Precision-weighted average (arithmetic average / median) for all
studies-2.9% (-8.5%/0.0%)

Note: * Chaudhry et al. (2020) does not provide an estimate but states that SIPO is insignificant. We use 0% when calculating thearithmetic average and median. Chaudhry et al. (2020) and Berry et al. (2021) do not affect the precision-weighted average, as
we do not know the standard errors.

Table 6 presents the results differentiated by quality dimensions. Four studies (Chernozhukov etal. (2021), Aparicio and Grossbard (2021), Alderman and Harjoto (2020) and Gibson (2020))
33

meet all quality dimensions but find vastly different effects of SIPOs on COVID-19 mortality.
The precision weighted average of the four studies is -1.0%. Four studies meet 3 of 4 quality
dimensions. They overall find that SIPOs increase COVID-19 mortality, as the precisionweighted average is positive (3.7%). The five studies that meet 2 of 4 quality dimensions or
fewer37 find a substantial reduction in COVID-19-mortality (-34.2%). This substantial reduction
seems to be driven by relatively short data series. The latest data point for the three studies whichfind large effects of lockdowns (Sears et al. (2020), Fowler et al. (2021), and Dave et al. (2021))
are April 29, May 7, and April 20, respectively. This may indicate that SIPOs can delay deaths
but not eliminate them completely. Disregarding these studies with short data series, theprecision-weighted average is -0.1%.

Table 6: Quality dimensions for studies based on SIPOs
Precisionweighted average*

Arithmetic average

MedianPeer-review [10]

-2.4%

-7.9%

-0.5%

Working paper [3]-12.0%-10.5%

0.0%

Data serie ends after 31 May 2020 [6]

-0.1%

-1.4%

-0.1%

Data serie ends before 31 May 2020 [7]-25.9%-14.6%

0.0%

Finds effect within the first 14 days [9]-2.0%-10.0%-1.0%

Does not find an effect within the first 14 days (including n/a) [4]-10.3%

-5.2%

0.0%

Social sciences [12]

-2.9%

-9.2%

-0.5%

Other sciences [1]

n/a

0.0%

0.0%

4 of 4 quality dimensions [4]

-1.0%

-5.5%

-3.5%

3 of 4 quality dimensions [4]

3.7%

-5.6%

2.7%

2 of 4 quality dimensions or fewer [5]-34.2%-13.2%

0.0%

Values show effect on COVID-19 mortality
Peer-reviewed vs. working papers

Long vs. short time period

No early effect on mortality

Social sciences vs. other sciences

Note: The table shows the common estimate as described in Table 5 for each quality dimension. The number of studies in eachcategory is in square brackets. * The precision-weighted average does not include studies where no common standard error isavailable, cf. Table 5.

Figure 6 shows a funnel plot for the studies in Table 5, except Chaudhry et al. (2020) and Berryet al. (2021), where common standard errors cannot be derived. Sears et al. (2020) stands out
with a precision far higher than those of the other studies. But generally, the precisions of thestudies are low and the estimates are placed on both sides of the zero-line with some ‘tail’ to the37

Bonardi et al. (2020) only meet one quality dimension (social science).

34

left.38 Figure 5 also shows that four of eight studies with at least 3 of 4 quality dimensions findthat SIPOs increase COVID-19 mortality by 0.8% to 13.1%.

Figure 6: Funnel plot for estimates from SIPO studies
180
160
140

WEight (1/SE)

120
100
80
60
40
20
0
-50.0%-40.0%-30.0%-20.0%-10.0%

0.0%

10.0%

20.0%

Estimate (Estimated Averted Deaths / Total Deaths)
2 of 4 quality dimensions or fewer

At least 3 of 4 quality dimensions

Note: The figure displays all estimates and the precision of the estimate defined as one over the standard error. Studies where
standard errors are not available are not included. Studies which live up to at least 3 of 4 quality dimensions are marked withwhite, while studies which lives up to 2 of 4 quality dimensions or less are marked with black. The vertical line illustrates theprecision-weighted average.

Overall conclusion on SIPO studies
We find no clear evidence that SIPOs had a noticeable impact on COVID-19 mortality. Somestudies find a large negative relationship between lockdowns and COVID-19 mortality, but thisseems to be caused by short data series which does not cover a full COVID-19 ‘wave’. Several
studies find a small positive relationship between lockdowns and COVID-19 mortality. Although
this appears to be counterintuitive, it could be the result of an (asymptomatic) infected person
being isolated at home under a SIPO can infect family members with a higher viral load causing
more severe illness.39 The overall effect measured by the precision-weighted average is -2.9%.
The result is in line with Nuzzo et al. (2019), who state that “In the context of a high-impact

38

This could indicate some publication bias, but the evidence is weak and with only 13 estimates, this cannot be
formally tested
39
E.g. see Guallar et al. (2020), who concludes, “Our data support that a greater viral inoculum at the time of SARSCoV-2 exposure might determine a higher risk of severe COVID-19.”35

respiratory pathogen, quarantine may be the least likely NPI to be effective in controlling the
spread due to high transmissibility” and World Health Organization Writing Group (2006), whoconclude that “forced isolation and quarantine are ineffective and impractical.”40
In the following section, we will look at the effect found in studies analyzing specific NPIs.

4.3

Studies of specific NPIs

A total of 11 eligible studies look at (multiple) specific NPIs independently or simply lockdownvs. no lockdown.41 The definition of the specific NPIs varies from study to study and aresomewhat difficult to compare. The variety in the definitions can be seen in the analysis of nonessential business closures and bar/restaurant closures. Chernozhukov et al. (2021) focus on a
combined parameter (the average of business closure and bar/restaurant closure in each state),
Aparicio and Grossbard (2021) look at business closure but not bar/restaurant closure, Spiegel
and Tookes (2021) examine bar/restaurant closure but not business closure, and Guo et al. (2021)
look at both business closures and bar/restaurant closures independently.
Some studies include several NPIs (e.g. Stokes et al. (2020) and Spiegel and Tookes (2021)),
while others cover very few. Bongaerts et al. (2021) only study business closures, and Leffler etal. (2020) look at internal lockdown and international travel restrictions). Few NPIs in a model
are potentially a problem because they can capture the effect of excluded NPIs. On the other
hand, several NPIs in a model increase the risk of multiple test bias.
The differences in the choice of NPIs and in the number of NPIs make it challenging to create an
overview of the results. In Table 7, we have merged the results in six overall categories but notethat the estimates may not be fully comparable across studies. In particular, the lockdownmeasure varies from study to study and in some cases is poorly defined by the authors. Also,
there are only a few estimates within some of the categories. For instance, the estimate of theeffect of facemasks is based on only two studies.
Table 7 illustrates that generally there is no evidence of a noticeable relationship between the
most-used NPIs and COVID-19. Overall, lockdowns and limiting gatherings seem to increase
COVID-19 mortality, although the effect is modest (0.6% and 1.6%, respectively) and borderclosures has little to no effect on COVID-19 mortality, with a precision-weighted average of 0.1% (removing the imprecise outlier from Guo et al. (2021) changes the precision-weighted
average to -0.2%). We find a small effect of school closure (-4.4%), but this estimate is mainly
driven by Auger et al. (2020), who – as noted earlier – use an “interrupted time series study”

40

Both Nuzzo et al. (2019) and World Health Organization Writing Group (2006) focus on quarantining infected
persons. However, if quarantining infected persons is not effective, it should be no surprise that quarantining
uninfected persons could be ineffective too.
41
Note that we – according to our search strategy – did not search on specific measures such as “school closures”
but on words describing the overall political approach to the COVID-19 pandemic such as “non-pharmaceutical,”“NPIs,” ”lockdown” etc.

36

approach and may capture other effects such as seasonal and behavioral effects. The absence of anotable effect of school closures is in line with Irfan et al. (2021), who – based on a systematic
review and meta-analysis of 90 published or preprint studies of transmission in children –concluded that “risks of infection among children in educational-settings was lower than incommunities. Evidence from school-based studies demonstrate it is largely safe for young
children (<10 years of age ) to be at schools; however, older children (between 10 and 19 yearsof age) might facilitate transmission.” UNICEF (2021) and ECDC (2020) reach similar
conclusions.42
Mandating facemasks – an intervention that was not widely used in the spring of 2020, and inmany countries was even discouraged – seems to have a large effect (-21.2%), but thisconclusion is based on only two studies.43 Again, our categorization may play a role, as thelarger mask-estimate from Chernozhukov et al. (2021) is in fact “employee facemasks,” not a
general mask mandate. Our findings are somewhat in contrast to the result found in a review by
Liu et al. (2021), who conclude that “fourteen of sixteen identified randomized controlled trials
comparing face masks to no mask controls failed to find statistically significant benefit in theintent-to-treat populations.” Similarly, a pre-COVID Cochrane review concludes, “There is low
certainty evidence from nine trials (3507 participants) that wearing a mask may make little or nodifference to the outcome of influenza-like illness (ILI) compared to not wearing a mask (riskratio (RR) 0.99, 95% confidence interval (CI) 0.82 to 1.18). There is moderate certainty evidence
that wearing a mask probably makes little or no difference to the outcome of laboratory‐
confirmed influenza compared to not wearing a mask (RR 0.91, 95% CI 0.66 to 1.26; 6 trials;
3005 participants)” (Jefferson et al. (2020)).44 However, it should be noted that even if no effectis found in controlled settings, this does not necessarily imply that mandated face masks does not
reduce mortality, as other factors may play a role (e.g. wearing a mask may function as a tax onsocializing if people are bothered by wearing a face masks when they are socializing).

UNICEF (2021) concludes, “The preliminary findings thus far suggest that in-person schooling – especially whencoupled with preventive and control measures – had lower secondary COVID-19 transmission rates compared toother settings and do not seem to have significantly contributed to the overall community transmission risks.”
Whereas, ECDC (2020) conclude, “School closures can contribute to a reduction in SARS-CoV-2 transmission,
but by themselves are insufficient to prevent community transmission of COVID-19 in the absence of othernonpharmaceutical interventions (NPIs) such as restrictions on mass gathering,” and states, “There is a generalconsensus that the decision to close schools to control the COVID-19 pandemic should be used as a last resort.
The negative physical, mental health and educational impact of proactive school closures on children, as well as
the economic impact on society more broadly, would likely outweigh the benefits.”
43
Note again, that we – according to our search strategy – did not search on the specific measures such as “masks,”“face masks,” “surgical masks” but on words describing the overall political approach to the COVID-19 pandemic
such as “non-pharmaceutical,” “NPIs,” ”lockdown” etc. Thus, we do not include most of the studies in mask
reviews such as Liu et al. (2021) and Jefferson et al. (2020).
44
Lipp and Edwards (2014) also find no evidence of an effect and – looking at disposable surgical face masks forpreventing surgical wound infection in clean surgery – conclude, “Three trials were included, involving a total of2113 participants. There was no statistically significant difference in infection rates between the masked andunmasked group in any of the trials.” Meanwhile, Li et al. (2021) – based on six case-control studies – conclude,“In general, wearing a mask was associated with a significantly reduced risk of COVID-19 infection (OR = 0.38,
95% CI: 0.21-0.69, I2 = 54.1%).
42

37

Only business closure consistently shows evidence of a negative relationship with COVID-19
mortality, but the variation in the estimated effect is large. Three studies find little to no effect,
and three find large effects. Two of the larger effects are related to closing bars and restaurants.
The “close business” category in Chernozhukov et al. (2021) is an average of closed businesses,
restaurants, and movie theaters, while that same category is “closing restaurants and bars” inSpiegel and Tookes (2021). The last study finding a large effect is Bongaerts et al. (2021), theonly eligible single-country study.45
As a final observation on Table 7, studies with fewer quality dimensions seem to find larger
effects, but the pattern is not systematic.46

Table 7: Overview of estimates from studies of specific NPIs
Lockdown(complete/partial)Chernozhukov et al. (2021)

Facemasks/
Employee facemasks

Business closure(/bars &restaurants)-34.0%-28.6%

4-31.6%

2

Bongaerts et al. (2021)
Chaudhry et al. (2020)*

Toya & Skidmore (2021)

Border closure(/quarantine)

Qualitydimensions

0.0%

2

0.5%

-0.1%

3

-1.3%

0.5%

Auger et al. (2020)

0.8%

1.7%

Spiegel & Tookes (2021)

2-15.6%
0.3%-13.5%-24.6%-0.1%-50.2%

0.0%

-6.3%

3

11.8%

3
1

0.0%

Guo et al. (2021)

-0.4%

36.3%

-0.2%

5.7%

-21.2%-10.6%

-0.1%

-4.4%

1.6%

0.6%

-23.8%-18.6%-0.7%-14.4%

3.0%

4 of 4 quality dimensions

0.3%
n/a [0]-23.8%-34.0% [1]-14.9%-2.9% [2]0.0%n/a [0]-0.1%
0.5% [1]3.2%
0.8% [1]3 of 4 quality dimensions

0.5% [1]-13.5% [1]-21.5% [3]0.0% [3]-0.1% [2]5.6% [3]2 of 4 quality dimensions or fewer

1.7% [2]n/a [1]-31.6% [2]-15.6% [2]-58.0% [1]n/a [1]

Precision-weighted average

0.6%

Arithmetic average
MedianNote: * It is not possible to derive common estimates and standard errors from Chaudhry et al. (2020) and Bonardi et al. (2020). Chaudhry
et al. (2020) states that the effect of the various NPIs is insignificant without listing the estimates and standard errors. Bonardi et al.
(2020) states that partial or regional lockdowns are as effective as stricter NPIs but does not provide information to calculate common
estimates. Instead, we assume the estimate is 0% when calculating arithmetic average and median, while the estimates are excluded fromthe calculation of precision-weighted averages because there are no standard errors.

45

Bongaerts et al. (2021) (implicitly) assume that municipalities with different exposures to closed sectors are notinherently different, which may be a relatively strong assumption and could potentially drive their results.
46
We saw with SIPOs that studies based on short data series tended to find larger effects than studies based on shortdata series. This is also somewhat true for studies examining multiple specific measures. If we focus on studies
with long data series (>May 31st, 2020), the precision-weighted estimates are as follows (average for all studies inparentheses for easy comparison): Lockdown (complete/partial): 0.5% (0.6%), Facemasks/Employee face masks: 21.2% (-21.2%), Business closures (/bars & restaurants): -8.1% (-10.6%), Border closures (/quarantine): -0.1% (0.1%), School closures: 0.5% (-4.4%), Limiting gatherings: 1.4% (1.6%).

38

4
2

-58.0%

Stokes et al. (2020)
Bonardi et al. (2020) *Limiting
gatherings

0.0%

Aparicio & Grossbard (2021)
Leffler et al. (2020)

Schoolclosures3

Figure 7 shows a funnel plot for all estimates in Table 7, except Chaudhry et al. (2020) andBonardi et al. (2020), where common standard errors cannot be derived. Two estimates from
Toya and Skidmore (2020) stands out with a precision far higher than those of other studies, andestimates are placed with some ‘tail’ to the left, which could indicate some publication bias, i.e.
reluctance to publish results that show large positive (more deaths) effects of lockdowns. The
most precise estimates are gathered around 0%, while less precise studies are spread out between-58% and 36%. The precision-weighted average of all estimates across all NPIs is -0.6%.

Figure 7: Funnel plot for estimates from studies of specific NPIs
4,000
3,500

Weight (1/SE)

3,000
2,500

Border closure (/quarantine)
Business closure (/bars & restaurants)

2,000

Facemasks/Employee face masks
Limiting gatherings (<100)

1,500

Lockdown (complete/partial)
1,000

School closures

500
0
-60%

-30%

0%

30%

60%

Estimate (Estimated Averted Deaths / Total Deaths)
Note: The figure displays all estimates except two (se text in figure) of specific NPIs and the precision of the estimate defined as
one over the standard error. Studies where standard errors are not available are not included.

Overall conclusion on specific NPIs
Because of the heterogeneity in NPIs across studies, it is difficult to draw strong conclusions
based on the studies of multiple specific measures. We find no evidence that lockdowns, school
closures, border closures, and limiting gatherings have had a noticeable effect on COVID-19
mortality. There is some evidence that business closures reduce COVID-19 mortality, but thevariation in estimates is large and the effect seems related to closing bars. There may be an effectof mask mandates, but just two studies look at this, one of which one only looks at the effect of
employee mask mandates.

39

5Concluding observations

Public health experts and politicians have – based on forecasts in epidemiological studies such asthat of Imperial College London (Ferguson et al. (2020) – embraced compulsory lockdowns as
an effective method for arresting the pandemic. But, have these lockdown policies been effectivein curbing COVID-19 mortality? This is the main question answered by our meta-analysis.
Adopting a systematic search and title-based screening, we identified 1,048 studies published by
July 1st, 2020, which potentially look at the effect of lockdowns on mortality rates. To answer
our question, we focused on studies that examine the actual impact of lockdowns on COVID-19
mortality rates based on registered cross-sectional mortality data and a counterfactual differencein-difference approach. Out of the 1,048 studies, 34 met our eligibility criteria.
Conclusions
Overall, our meta-analysis fails to confirm that lockdowns have had a large, significant effect onmortality rates. Studies examining the relationship between lockdown strictness (based on the
OxCGRT stringency index) find that the average lockdown in Europe and the United States only
reduced COVID-19 mortality by 0.2% compared to a COVID-19 policy based solely on
recommendations. Shelter-in-place orders (SIPOs) were also ineffective. They only reduced
COVID-19 mortality by 2.9%.
Studies looking at specific NPIs (lockdown vs. no lockdown, facemasks, closing non-essential
businesses, border closures, school closures, and limiting gatherings) also find no broad-based
evidence of noticeable effects on COVID-19 mortality. However, closing non-essential
businesses seems to have had some effect (reducing COVID-19 mortality by 10.6%), which islikely to be related to the closure of bars. Also, masks may reduce COVID-19 mortality, butthere is only one study that examines universal mask mandates. The effect of border closures,
school closures and limiting gatherings on COVID-19 mortality yields precision-weighted
estimates of -0.1%, -4.4%, and 1.6%, respectively. Lockdowns (compared to no lockdowns) also
do not reduce COVID-19 mortality.

Discussion
Overall, we conclude that lockdowns are not an effective way of reducing mortality rates during
a pandemic, at least not during the first wave of the COVID-19 pandemic. Our results are in linewith the World Health Organization Writing Group (2006), who state, “Reports from the 1918
influenza pandemic indicate that social-distancing measures did not stop or appear todramatically reduce transmission […] In Edmonton, Canada, isolation and quarantine were
instituted; public meetings were banned; schools, churches, colleges, theaters, and other publicgathering places were closed; and business hours were restricted without obvious impact on theepidemic.” Our findings are also in line with Allen's (2021) conclusion: “The most recent
research has shown that lockdowns have had, at best, a marginal effect on the number of Covid19 deaths.” Poeschl and Larsen (2021) conclude that “interventions are generally effective in40

mitigating COVID-19 spread”. But, 9 of the 43 (21%) results they review find “no or uncertain
association” between lockdowns and the spread of COVID-19, suggesting that evidence fromthat own study contradicts their conclusion.
The findings contained in Johanna et al. (2020) are in contrast to our own. They conclude that“for lockdown, ten studies consistently showed that it successfully reduced the incidence,
onward transmission, and mortality rate of COVID-19.” The driver of the difference is threefold. First, Johanna et al. include modelling studies (10 out of a total of 14 studies), which wehave explicitly excluded. Second, they included interrupted time series studies (3 of 14 studies),
which we also exclude. Third, the only study using a difference-in-difference approach (as wehave done) is based on data collected before May 1st, 2020. We should mention that our results
indicate that early studies find relatively larger effects compared to later studies.
Our main conclusion invites a discussion of some issues. Our review does not point out why
lockdowns did not have the effect promised by the epidemiological models of Imperial College
London (Ferguson et al. (2020). We propose four factors that might explain the difference
between our conclusion and the view embraced by some epidemiologists.
First, people respond to dangers outside their door. When a pandemic rages, people believe insocial distancing regardless of what the government mandates. So, we believe that Allen (2021)
is right, when he concludes, “The ineffectiveness [of lockdowns] stemmed from individualchanges in behavior: either non-compliance or behavior that mimicked lockdowns.” In economic
terms, you can say that the demand for costly disease prevention efforts like social distancing
and increased focus on hygiene is high when infection rates are high. Contrary, when infection
rates are low, the demand is low and it may even be morally and economically rational not tocomply with mandates like SIPOs, which are difficult to enforce. Herby (2021) reviews studies
which distinguish between mandatory and voluntary behavioral changes. He finds that – onaverage – voluntary behavioral changes are 10 times as important as mandatory behavioral
changes in combating COVID-19. If people voluntarily adjust their behavior to the risk of thepandemic, closing down non-essential businesses may simply reallocate consumer visits awayfrom “nonessential” to “essential” businesses, as shown by Goolsbee and Syverson (2021), withlimited impact on the total number of contacts.47 This may also explain why epidemiologicalmodel simulations such as Ferguson et al. (2020) – which do not model behavior endogenously –fail to forecast the effect of lockdowns.
Second, mandates only regulate a fraction of our potential contagious contacts and can hardly
regulate nor enforce handwashing, coughing etiquette, distancing in supermarkets, etc. Countries
like Denmark, Finland, and Norway that realized success in keeping COVID-19 mortality rates
relatively low allowed people to go to work, use public transport, and meet privately at homeduring the first lockdown. In these countries, there were ample opportunities to legally meet withothers.

47

In economic terms, lockdowns are substitutes for – not complements to – voluntary behavioral changes.

41Third, even if lockdowns are successful in initially reducing the spread of COVID-19, thebehavioral response may counteract the effect completely, as people respond to the lower risk bychanging behavior. As Atkeson (2021) points out, the economic intuition is straightforward. If
closing bars and restaurants causes the prevalence of the disease to fall toward zero, the demandfor costly disease prevention efforts like social distancing and increased focus on hygiene also
falls towards zero, and the disease will return.48
Fourth, unintended consequences may play a larger role than recognized. We already pointed tothe possible unintended consequence of SIPOs, which may isolate an infected person at homewith his/her family where he/she risks infecting family members with a higher viral load, causing
more severe illness. But often, lockdowns have limited peoples’ access to safe (outdoor) places
such as beaches, parks, and zoos, or included outdoor mask mandates or strict outdoor gathering
restrictions, pushing people to meet at less safe (indoor) places. Indeed, we do find someevidence that limiting gatherings was counterproductive and increased COVID-19 mortality.
One objection to our conclusions may be that we do not look at the role of timing. If timing isvery important, differences in timing may empirically overrule any differences in lockdowns. We
note that this objection is not necessarily in contrast to our results. If timing is very important
relative to strictness, this suggests that well-timed, but very mild, lockdowns should work as wellas, or better than, less well-timed but strict lockdowns. This is not in contrast to our conclusion,as the studies we reviewed analyze the effect of lockdowns compared but to doing very little (seeSection 3.1 for further discussion). However, there is little solid evidence supporting the timing
thesis, because it is inherently difficult to analyze (see Section 2.2 for further discussion). Also,
even if it can be empirically stated that a well-timed lockdown is effective in combating apandemic, it is doubtful that this information will ever be useful from a policy perspective.
But, what explains the differences between countries, if not differences in lockdown policies?
Differences in population age and health, quality of the health sector, and the like are obvious
factors. But several studies point at less obvious factors, such as culture, communication, andcoincidences. For example, Frey et al. (2020) show that for the same policy stringency, countries
with more obedient and collectivist cultural traits experienced larger declines in geographic
mobility relative to their more individualistic counterpart. Data from Germany Laliotis andMinos (2020) shows that the spread of COVID-19 and the resulting deaths in predominantly
Catholic regions with stronger social and family ties were much higher compared to nonCatholic ones at the local NUTS 3 level.49
Government communication may also have played a large role. Compared to its Scandinavian
neighbors, the communication from Swedish health authorities was far more subdued andembraced the idea of public health vs. economic trade-offs. This may explain why Helsingen et

48

This kind of behavior response may also explain why Subramanian and Kumar (2021) find that increases in
COVID-19 cases are unrelated to levels of vaccination across 68 countries and 2947 counties in the United States.
When people are vaccinated and protected against severe disease, they have less reason to be careful.
49
The NUTS classification (Nomenclature of territorial units for statistics) is a hierarchical system for dividing up
the economic territory of the EU and the UK. There are 1215 regions at the NUTS 3-level.

42

al. (2020), found, based on questionnaire data collected from mid-March to mid-April, 2020, thateven though the daily COVID-19 mortality rate was more than four times higher in Sweden thanin Norway, Swedes were less likely than Norwegians to not meet with friends (55% vs. 87%),
avoid public transportation (72% vs. 82%), and stay home during spare time (71% vs. 87%).
That is, despite a more severe pandemic, Swedes were less affected in their daily activities (legal
in both countries) than Norwegians.
Many other factors may be relevant, and we should not underestimate the importance of
coincidences. An interesting example illustrating this point is found in Arnarson (2021) and
Björk et al. (2021), who show that areas where the winter holiday was relatively late (in week 9
or 10 rather than week 6, 7 or 8) were hit especially hard by COVID-19 during the first wave
because the virus outbreak in the Alps could spread to those areas with ski tourists. Arnarson(2021) shows that the effect persists in later waves. Had the winter holiday in Sweden been inweek 7 or week 8 as in Denmark, the Swedish COVID-19 situation could have turned out very
differently.50
Policy implications
In the early stages of a pandemic, before the arrival of vaccines and new treatments, a society
can respond in two ways: mandated behavioral changes or voluntary behavioral changes. Our
study fails to demonstrate significant positive effects of mandated behavioral changes(lockdowns). This should draw our focus to the role of voluntary behavioral changes. Here, more
research is needed to determine how voluntary behavioral changes can be supported. But itshould be clear that one important role for government authorities is to provide information sothat citizens can voluntarily respond to the pandemic in a way that mitigates their exposure.
Finally, allow us to broaden our perspective after presenting our meta-analysis that focuses on
the following question: “What does the evidence tell us about the effects of lockdowns on
mortality?” We provide a firm answer to this question: The evidence fails to confirm that
lockdowns have a significant effect in reducing COVID-19 mortality. The effect is little to none.
The use of lockdowns is a unique feature of the COVID-19 pandemic. Lockdowns have not beenused to such a large extent during any of the pandemics of the past century. However, lockdowns
during the initial phase of the COVID-19 pandemic have had devastating effects. They havecontributed to reducing economic activity, raising unemployment, reducing schooling, causing
political unrest, contributing to domestic violence, and undermining liberal democracy. These
costs to society must be compared to the benefits of lockdowns, which our meta-analysis has
shown are marginal at best. Such a standard benefit-cost calculation leads to a strong conclusion:
lockdowns should be rejected out of hand as a pandemic policy instrument.

50

Another case of coincidence is illustrated by Shenoy et al. (2022), who find that areas that experienced rainfall
early in the pandemic realized fewer deaths because the rainfall induced social distancing.

43

6

Appendix A. The role of timing

Some of the included papers study the importance of the timing of lockdowns, while several
other papers only looking at timing of (but not on the inherent effect of) lockdowns have been
excluded from the literature list in this review. There’s no doubt that being prepared for apandemic and knowing when it arrives at your doorstep is vital. However, two problems arise
with respect to imposing early lockdowns.
First of all, it was virtually impossible to determine the right timing when COVID-19 hit Europe
and the United States. The World Health Organization declared the outbreak of a pandemic on
11 March 2020, but at that date Italy had already registered 13.7 COVID-19-deaths per million(all infected before approximately 22 February, because of the roughly 18 day gap between
infection and death, c.f. e.g.. Bjørnskov (2021a)). On 29 March 2020, 18 days after WHO
declared the outbreak a pandemic and the earliest a lockdown response to WHO’s announcement
could have an effect, the death toll in Italy was a staggering 178 COVID-19-deaths per million
with an additionally 13 per million dying each day.
There are reasons to believe that many countries and regions were hit particularly hard during thefirst wave of COVID, because they had no clue about how bad it really was. This point is
illustrated in Figure 8 (and Figure 9), which show that countries (and states), which were hit hardand early, experienced large death tolls compared to countries where the pandemic had a slower
start. Björk et al. (2021) and Arnarson (2021) show that areas with a winter holiday in week 10
and – especially – week 9 were hit hard, because they imported cases from the Alps before theyknew the pandemic was wide spread at the ski resorts. Hence, while acting early by warning
citizens and closing business may be an effective strategy; this was not a feasible strategy formost countries in the spring of 2020.
The second problem is that it is extremely difficult to differentiate between the effect of publicawareness and the effect of lockdowns. If people and politicians react to the same information,for example deaths in geographical neighboring countries (many EU-countries reacted to deaths
in Italy) or in another part of the same country, the effect of lockdowns cannot easily beseparated from the effect of voluntary social distancing or, use of hand sanitizers. Hence, we findit problematic to use national lockdowns and differences in the progress of the pandemic indifferent regions to say anything about the effect of early lockdowns on the pandemic, as theestimated effect might just as well come from voluntary behavior changes, when people inSouthern Italy react to the situation in Northern Italy.
We have seen no studies which we believe credibly separate the effect of early lockdown fromthe effect of early voluntary behavior changes. Instead, the estimates in these studies capture theeffects of lockdowns and voluntary behavior changes. As Herby (2021) illustrates, voluntary
behavior changes are essential to a society’s response to an pandemic and can account for up to
90% of societies’ total response to the pandemic.
Including these studies will greatly overestimate the effect of lockdowns, and, hence, we chose
not to include studies focusing on timing of lockdowns in our review.
44Figure 8: Taken by surprise. The importance of having time to prepare in Europe
900

BE

800
700

1st wave deaths pr. million

ES
600

GB

IT

SE
500
400

FR
NL

IE

300
CH
200
100
0
10-Mar-20

PT
DK DE
AT

MK

MD

RO
SI
NO EE

HU
FI
BA
RSCZ
HR PL

14-Apr-20

RU
BY

BG
LT

19-May-20

Date to reach 20 COVID-19-deaths per million

Description: European countries with more than one million citizens.
Source: Our World in Data

45

UA

XK AL
23-Jun-20

Figure 9: Taken by surprise. The importance of having time to prepare in U.S. states
1,800

NJ

NY
1,600
1,400

1st wave deaths pr. million

CT

MA

1,200

RI

1,000

FL

800

KY

LA

MI
IL

600
400
200
0
10-Mar-20

MD

SC

PA
OK

VA

NM

KS

IN MS

CO

IA
NHMN
GA
OH ALAZ
NV MO
NC NE

WA

WI
ME

ID

CA

AR
OR WV

TN
TX
UT

14-Apr-20

MT AK
19-May-20

Date to reach 20 COVID-19-deaths per million

Description: U.S. states with more than one million citizens.
Source: Our World in Data

46

23-Jun-20

HI7

Appendix B. Supplementary information

7.1

Excluded studies

Below is a list will the studies excluded during the eligibility phase of our identification process
and a short description of our basis for excluding the study.

Table 8: Studies excluded during the eligibility phase of our identification process
1. Study (Author & title)

2. Reason for
exclusion

Alemán et al. (2020); "Evaluating the effectiveness of policies against a pandemic"
Alshammari et al. (2021); "Are countries' precautionary actions against COVID-19 effective? An assessment study of 175 countries worldwide"
Amuedo-Dorantes et al. (2020); "Timing is Everything when Fighting a Pandemic: COVID-19 Mortality in Spain"
Amuedo-Dorantes et al. (2021); "Early adoption of non-pharmaceutical interventions and COVID-19 mortality"
Amuedo-Dorantes, Kaushal and Muchow (2020); "Is the Cure Worse than the Disease? County-Level Evidence from the COVID-19 Pandemic in the United States"
Amuedo-Dorantes, Kaushal and Muchow (2021); "Timing of social distancing policies and COVID-19 mortality: county-level evidence from the U.S."
Arruda et al. (2021); "ASSESSING THE IMPACT OF SOCIAL DISTANCING ON COVID-19 CASES AND DEATHS IN BRAZIL: AN INSTRUMENTED DIFFERENCE-INDIFFERENCES…" "Changes in daily mental health service use and mortality at the commencement and lifting of COVID-19 ‘lockdown’ policy in 10 UK sites: a regression
Bakolis et al. (2021);
discontinuity
in timeand
design"Bardey, Fernández
Gravel (2021); "Coronavirus and social distancing: do non-pharmaceutical-interventions work (at least) in the short run?"
Berardi et. Al. (2020); "The COVID-19 pandemic in Italy: policy and technology impact on health and non-health outcomes"
Bhalla (2020); "Lockdowns and Closures vs COVID–19: COVID Wins"
Björk et al. (2021); "Impact of winter holiday and government responses on mortality in Europe during the first wave of the COVID-19 pandemic"
Bongaerts, Mazzola and Wagner (2020); "Closed for business"
Born, Dietrich and Müller (2021); "The lockdown effect: A counterfactual for Sweden"
Born, Dietrich and Müller (2021); "The lockdown effect: A counterfactual for Sweden"
Bushman et al. (2020); "Effectiveness and compliance to social distancing during COVID-19"
Castaneda and Saygili (2020); "The effect of shelter-in-place orders on social distancing and the spread of the COVID-19 pandemic: a study of Texas"
Cerqueti et al. (2021); "The sooner the better: lives saved by the lockdown during the COVID-19 outbreak. The case of Italy"
Chernozhukov, Kasahara and Schrimpf (2021); "Mask mandates and other lockdown policies reduced the spread of COVID-19 in the U.S."
Chin et al. (2020); "Effects of non-pharmaceutical interventions on COVID-19: A Tale of Three Models"Cho (2020); "Quantifying the impact of nonpharmaceutical interventions during the COVID-19 outbreak: The case of Sweden"
Coccia (2020); "The effect of lockdown on public health and economic system: findings from first wave of the COVID-19 pandemic for designing effective strategies to copewithfuturewaves"
Coccia(2021);"Different effects of lockdown on public health and economy of countries: Results from first wave of the COVID-19 pandemic"
Conyon and Thomsen (2021); "COVID-19 in Scandinavia"
Conyon et al. (2020); "Lockdowns and COVID-19 deaths in Scandinavia"
Dave et al. (2020); "Did the Wisconsin Supreme Court restart a COVID-19 epidemic? Evidence from a natural experiment"
Delis, Iosifidi and Tasiou (2021); "Efficiency of government policy during the COVID-19 pandemic"
Dreher et al. (2021); "Policy interventions, social distancing, and SARS-CoV-2 transmission in the United States: a retrospective state-level analysis"
Duchemin, Veber and Boussau (2020); "Bayesian investigation of SARS-CoV-2-related mortality in France"
Fair et. Al. (2021); "Estimating COVID-19 cases and deaths prevented by non-pharmaceutical interventions in 2020-2021, and the impact of individual actions: a retrospective
model…"
Filias (2020);"The impact of government policies effectiveness on the officially reported deaths attributed to covid-19."
Fowler et al. (2021); "Stay-at-home orders associate with subsequent decreases in COVID-19 cases and fatalities in the United States"Friedson et al. (2020); "Did California's shelter-in-place order work? Early coronavirus-related public health effects"
Friedson et al. (2020); "Shelter-in-place orders and public health: evidence from California during the COVID-19 pandemic"
Fuss, Weizman and Tan (2020); "COVID19 pandemic: how effective are interventive control measures and is a complete lockdown justified? A comparison of countries andstates" Ghosh and Narymanchi (2020); "A Study on The Effectiveness of Lock-down Measures to Control The Spread of COVID-19"
Ghosh,
Glogowsky et al. (2021); "How Effective Are Social Distancing Policies? Evidence on the Fight Against COVID-19"
Glogowsky, Hansen and Schächtele (2020); "How effective are social distancing policies? Evidence on the fight against COVID-19 from Germany"
Glogowsky, Hansen and Schächtele (2020); "How Effective Are Social Distancing Policies? Evidence on the Fight Against COVID-19 from Germany"
Gordon, Grafton and Steinshamn (2021); "Cross-country effects and policy responses to COVID-19 in 2020: The Nordic countries"
Gordon, Grafton and Steinshamn (2021); "Statistical Analyses of the Public Health and Economic Performance of Nordic Countries in Response to the COVID-19 Pandemic"
Guo et al. (2020); "Social distancing interventions in the United States: An exploratory investigation of determinants and impacts"
Huber and Langen (2020); "The impact of response measures on COVID-19-related hospitalization and death rates in Germany and Switzerland"
Huber and Langen (2020); "Timing matters: the impact of response measures on COVID-19-related hospitalization and death rates in Germany and Switzerland"
Jain et al. (2020); "A comparative analysis of COVID-19 mortality rate across the globe: An extensive analysis of the associated factors"
Juranek and Zoutman (2021); "The effect of non-pharmaceutical interventions on the demand for health care and mortality: evidence on COVID-19 in Scandinavia"
Kakpo and Nuhu (2020); "Effects of Social Distancing on COVID-19 Infections and Mortality in the U.S."
Kapoor and Ravi (2020); "Impact of national lockdown on COVID-19 deaths in select European countries and the U.S. using a Changes-in-Changes model"
Khatiwada and Chalise (2020); "Evaluating the efficiency of the Swedish government policies to control the spread of Covid-19."
Korevaar et al. (2020); "Quantifying the impact of U.S. state non-pharmaceutical interventions on COVID-19 transmission"
Kumar et. Al. (2020); "Prevention-Versus Promotion-Focus Regulatory Efforts on the Disease Incidence and Mortality of COVID-19: A Multinational Diffusion Study Using
Functional
Data "Impact…"
Leet al. (2020);
of government-imposed social distancing measures on COVID-19 morbidity and mortality around the world"
Liang et al. (2020); "Covid-19 mortality is negatively associated with test number and government effectiveness"Mader and Rütternauer (2021); "The effects of non-pharmaceutical interventions on COVID-19-related mortality: A generalized synthetic control approach across 169 countries"
Matzinger and Skinner (2020); "Strong impact of closing schools, closing bars and wearing masks during the Covid-19 pandemic: results from a simple and revealing analysis"
Mccafferty and Ashley (2020); "Covid-19 Social Distancing Interventions by State Mandate and their Correlation to Mortality in the United States"
Medline et al. (2020); "Evaluating the impact of stay-at-home orders on the time to reach the peak burden of Covid-19 cases and deaths: does timing matter?"

Too few observations
Is purely descriptive
Duplicate
Only looks at timing
Duplicate
Only looks at timing
Social distancing (notlockdowns)
Uses a time series approach
Only looks at timing
Too few observations
Uses modelling
Only looks at timing
Duplicate
Synthetic control study
DuplicateSocial distancing (notlockdowns)
Uses a time series approach
Synthetic control study
Duplicate
Uses modelling
Synthetic control study
Only looks at timing
Too few observations
Synthetic control study
Too few observations
Synthetic control study
Do not look at mortality
Do not look at mortality
Uses modelling
Uses modelling
Student paper
Duplicate
Duplicate
Synthetic control study
Do not look at mortality
Synthetic control study
Only looks at timing
Duplicate
Duplicate
Do not look at mortality
Too few observations
Duplicate
Duplicate
Only looks at timing
Do not look at mortality
Too few observations
Social distancing (notlockdowns)
Too few observations
Student paper
Do not look at mortality
Do not look at mortality
Uses a time series approach
Not effect of lockdowns
Synthetic control study
Uses modelling
Duplicate
Only looks at timing47

1. Study (Author & title)

2. Reason for
exclusion

Mu et al. (2020); "Effect of social distancing interventions on the spread of COVID-19 in the state of Vermont"
Nakamura (2020); "The Impact of Rapid State Policy Response on Cumulative Deaths Caused by COVID-19"
Neidhöfer and Neidhöfer (2020); "The effectiveness of school closures and other pre-lockdown COVID-19 mitigation strategies in Argentina, Italy, and South Korea"
Oliveira (2020); "Does' Staying at Home'Save Lives? An Estimation of the Impacts of Social Isolation in the Registered Cases and Deaths by COVID-19 in Brazil"
Palladina et al. (2020); "Effect of Implementation of the Lockdown on the Number of COVID-19 Deaths in Four European Countries"
Palladina et al. (2020); "Effect of timing of implementation of the lockdown on the number of deaths for COVID-19 in four European countries"
Palladino et al. (2020); "Excess deaths and hospital admissions for COVID-19 due to a late implementation of the lockdown in Italy"
Peixoto et al. (2020); "Rapid assessment of the impact of lockdown on the COVID-19 epidemic in Portugal"
Piovani et. Al. (2021); "Effect of early application of social distancing interventions on COVID-19 mortality over the first pandemic wave: An analysis of longitudinal data from 37
countries"(2021); "Effect of fall 2020 K-12 instruction types on CoViD-19 cases, hospital admissions, and deaths in Illinois counties"
Reinbold
Renne, Roussellet and Schwenkler (2020); "Preventing COVID-19 Fatalities: State versus Federal Policies"
Siedner et al. (2020); "Social distancing to slow the U.S. COVID-19 epidemic: Longitudinal pretest–posttest comparison group study"
Siedner et al. (2020); "Social distancing to slow the U.S. COVID-19 epidemic: Longitudinal pretest–posttest comparison group study"
Silva, Filho and Fernandes (2020); "The effect of lockdown on the COVID-19 epidemic in Brazil: evidence from an interrupted time series design"
Stamam et al. (2020); "IMPACT OF LOCKDOWN MEASURE ON COVID-19 INCIDENCE AND MORTALITY IN THE TOP 31 COUNTRIES OF THE WORLD."
Steinegger et al. (2021); "Retrospective study of the first wave of COVID-19 in Spain: analysis of counterfactual scenarios"
Stephens et al. (2020); "Does the timing of government COVID-19 policy interventions matter? Policy analysis of an original database."
Supino et al. (2020); "The effects of containment measures in the Italian outbreak of COVID-19"
Timelli and Girardi (2021); "Effect of timing of implementation of containment measures on Covid-19 epidemic. The case of the first wave in Italy"
Trivedi and Das (2020); "Effect of the timing of stay-at-home orders on COVID-19 infections in the United States of America"
Umer and Khan (2020); "Evaluating the Effectiveness of Regional Lockdown Policies in the Containment of Covid-19: Evidence from Pakistan"
VoPham et al. (2020); "Effect of social distancing on COVID-19 incidence and mortality in the U.S."
Wu and Wu (2020); "Stay-at-home and face mask policies intentions inconsistent with incidence and fatality during U.S. COVID-19 pandemic"
Xu et al. (2020); "Associations of Stay-at-Home Order and Face-Masking Recommendation with Trends in Daily New Cases and Deaths of Laboratory-Confirmed COVID-19 in
the UnitedStates"Yehya,
Venkataramaniand Harhay (2020); "Statewide Interventions and Coronavirus Disease 2019 Mortality in the United States: An Observational Study"
Ylli et al. (2020); "The lower COVID-19 related mortality and incidence rates in Eastern European countries are associated with delayed start of community circulation Alban
Ylli1 …"Uses modelling
Student paper
Synthetic control study
Social distancing (notlockdowns)
Uses a time series approach
Duplicate
Uses a time series approach
Uses modelling
Only looks at timing
Synthetic control study
Uses modelling
Duplicate
Uses a time series approach
Uses a time series approach
Uses a time series approach
Only looks at timing
Only looks at timing
Uses a time series approach
Only looks at timing
Only looks at timing
Too few observations
Do not look at mortality
Too few observations
Do not look at mortality
Only looks at timing
Not effect of lockdowns

7.2

Interpretation of estimates and conversion to common estimates

In Table 9, we describe for each study used in the meta-analysis how we interpret their results
and convert the estimates to our common estimate. Standard errors are converted such that the tvalue, calculated based on common estimates and standard errors, is unchanged. Whenconfidence intervals are reported rather than standard errors, we calculate standard errors using tdistribution with ∞ degrees of freedom (i.e. 1.96 for 95% confidence interval).

Table 9: Notes on studies included in the meta-analysis
1. Study (Author & title)

2. DatePublished

3. Journal

4. Comments regarding meta-analysis

Alderman and Harjoto(2020); "COVID-19: U.S.
shelter-in-place orders anddemographic characteristics
linked to cases, mortality,
and recovery rates"
Aparicio and Grossbard(2021); "Are Covid Fatalities
in the U.S. Higher than in theEU, and If so, Why?"

26-Nov20

TransformingGovernment:
People,Process andPolicyReview of
Economicsof theHouseholdWe use the 1% effect noted by the authors in "We find that the natural log of the duration (in days)
that the state instituted shelter-in-place reduces percentages of mortality by 0.0001%, orapproximately 1% of the means of percentages of deaths per capita in our sample. The standard error
is calculated on basis of the t-value in Table 3.

Ashraf (2020);"Socioeconomic conditions,
government interventions
and health outcomes during
COVID-19"

1-Jul-20

16-Jan-21

ResearchGate

We use estimates from Table 3, model 5. For each estimate the common estimate is calculated as(difference in COVID-19 mortality with NPI)/(difference in COVID-19 mortality without NPI)-1,
where (difference in COVID-19 mortality with NPI) is 237.89 (Table 2 states that deaths per million is406.99 in U.S. and 169.10 in Europe) and (difference in COVID-19 mortality without NPI) is estimatedas exp(ln(difference in COVID-19 mortality with NPI)-estimate).
It is unclear whether they prefer the model with or without the interaction term. In the meta-analysis,
we use an average of -0.326 (Table 3, without) and -0.073 (Table 6, with) deaths per million per
stringency point (i.e. -0.200). The common estimate is the average effect in Europe and United States
respectively calculated as (Actual COVID-19 mortality) / (COVID-19 mortality with recommendation
policy) -1, where (COVID-19 mortality with recommendation policy) is calculated as ((Actual COVID19 mortality) - Estimate x Difference in stringency x population). Stringencies in Europe and UnitedStates are equal to the average stringency from March 16th to April 15th 2020 (76 and 74
respectively) and the stringency for the policy based solely on recommendations is 44 following Haleet al. (2020).

48

1. Study (Author & title)

2. DatePublished

Auger et al. (2020);"Association between
statewide school closure and
COVID-19 incidence andmortality in the U.S."
Berry et al. (2021);"Evaluating the effects ofshelter-in-place policies
during the COVID-19
pandemic"

1-Sep-20

Bjørnskov (2021a); "Did
Lockdown Work? An
Economist's Cross-Country
Comparison"Blanco et al. (2020); "Do
Coronavirus Containment
Measures Work? Worldwide
Evidence"

29-Mar21
1-Dec-20

3. Journal

4. Comments regarding meta-analysis

JAMA

Estimate that school closure was associated with a 58% decline in COVID-19 mortality and that theeffect was largest in states with low cumulative incidence of COVID-19 at the time of school closure.
States with the lowest incidence of COVID-19 had a −72% relative change in incidence comparedwith −49% for those states with the highest cumulative incidence.

24-Feb-21 PNAS

CESifoEconomic
Studies

The estimated effect of SIPO's, an increase in deaths by 0,654 per million after 14 days (significant, cf.
Fig. 2), is converted to a relative effect on a state basis based on data from OurWorldInData. For
states which did implement SIPO, we calculate the number of deaths without SIPO as the number ofofficial COVID-19 deaths 14 days after SIPO was implemented minus 0,654 extra deaths per million.
For states which did not implement SIPO, we calculate the number of deaths with SIPO as thenumber of official COVID-19 deaths 14 days after March 31 2020 plus 0,654 extra deaths per million.
We use March 31 2020 as this was the average date on which SIPO was implemented in the 40 states
which did implement SIPO. Using this approximation, the effect of SIPO's in the U.S. is 1,1% moredeaths after 14 days. Common standard errors are not available.
We use estimates from Table 2 (four weeks). Common estimate is calculated as the average of theeffect in Europe and United States, where the effect for each is calculated as (ln(policy stringency) ln(recommendation stringency)) x estimate.

World BankGroup

The study is not included in the meta-analysis, as it looks at the effect of NPIs on growth rates anddoes not include an estimate of the effect on total mortality.

Bonardi et al. (2020); "Fast8-Jun-20and local: How did lockdown
policies affect the spread andseverity of the covid-19"0Bongaerts et al. (2021);"Closed for business: The
mortality impact of businessclosures during the Covid-19
pandemic"
Chaudhry et al. (2020); "A
country level analysis
measuring the impact ofgovernment actions, country
preparedness andsocioeconomic factors on
COVID-19 mortality andrelated health outcomes"
Chernozhukov et al. (2021);"Causal impact of masks,
policies, behavior on early
covid-19 pandemic in the
U.S."

14-May21

PLOS ONEFind that, world-wide, internal NPIs have prevented about 650,000 deaths (3.11 deaths wereprevented for each death that occurred, i.e. 76% effect). However, this effect is for any lockdown
including a Swedish lockdown. They do not find an extra effect of stricter lockdowns and state that“our results point to the fact that people might adjust their behaviors quite significantly as partial
measures are implemented, which might be enough to stop the spread of the virus.” Hence, whether
the baseline is Sweden, which implemented a ban on large gatherings early in the pandemic, or thebaseline is “doing nothing” can affect the magnitude of the estimated impacts. Since all Westerncountries did something and estimates in other reviewed studies are relative to doing less – and,hence not to doing nothing, we report the result from Bonardi et al. as compared to “doing less.”
Hence, for Bonardi et al. we use 0% as the common estimate in the meta-analysis for each NPI (SIPO,
regional lockdown, partial lockdown, and border closure (stage 1, stage 2 and full) because all NPIs are
insignificant (compared to Sweden’s “doing the least”-lockdown).
Business shutdown saved 9,439 Italian lives by 13th 2020. This corresponds to 32%, as there were20,465 COVID-19-deaths in Italy by mid April 2020.

1-Aug-20

EClinacalMedicine

Finds no effect of partial border closure, complete border closure, partial lockdown (physicaldistancing measures only), complete lockdown (enhanced containment measures including suspensionof all non-essential services), and curfews. In the meta-analysis we use a common estimate of 0%, asestimates and standard errors are not available.

1-Jan-21

Journal of
Econometrics

Chisadza et al. (2021);"Government Effectivenessand the COVID-19
Pandemic"

10-Mar21

MDPI

Dave et al. (2021); "WhenDo Shelter-in-Place Orders

3-Aug-20

Economic
Inpuiry

The study looks at the effect of NPIs on growth rates but does include an estimate of the effect on
total mortality at the end of the study period for employee face masks (-34%), business closure (29%). and SIPO (-18%), but not for school closures (which we therefore exclude). In reporting theresults of their counterfactual, they alter between "fewer deaths with NPI" and "more deaths without
NPI.” We have converted the latter to the former as estimate/(1+estimate) so "without businessclosures deaths would be about 40% higher" corresponds to "with business closures deaths would be
about 29% lower.”
The common estimate is the average effect in Europe and United States respectively calculated as(Actual COVID-19 mortality) / (COVID-19 mortality with recommendation policy) -1, where (COVID19 mortality with recommendation policy) is calculated as ((Actual COVID-19 mortality) - Estimate x
Difference in stringency x population). Stringencies in Europe and United States are equal to theaverage stringency from March 16th to April 15th 2020 (76 and 74 respectively) and the stringency
for the policy based solely on recommendations is 44 following Hale et al. (2020). In the meta-analysis
we use the non-linear estimate, but the squared estimate yields similar results.
The study looks at the effect of SIPO's on growth rates but does include an estimate of the effect on
total mortality after 20+ days for model 1 and 2 in Table 7. Since model 3, 4 and 5 have estimates49

1. Study (Author & title)
Fight Covid-19 Best? Policy
Heterogeneity Across Statesand Adoption Time"
Dergiades et al. (2020);"Effectiveness ofgovernment policies in
response to the COVID-19
outbreak"Fakir and Bharati (2021);"Pandemic catch-22: The
role of mobility restrictions
and institutional inequalities
in halting the spread of
COVID-19"
Fowler et al. (2021); "Stayat-home orders associatewith subsequent decreases
in COVID-19 cases andfatalities in the UnitedStates"Fuller et al. (2021);"Mitigation Policies and
COVID-19–AssociatedMortality — 37 European
Countries, January 23–June30, 2020"

2. DatePublished

3. Journal

4. Comments regarding meta-analysis
similar to model 2, we use an average of model 1 to 5, where the estimates of model 3 to 5 arecalculated as (common estimate model 2) / (estimate model 2) x estimate model 3/4/5.

28-Aug20

SSRN

The study is not included in the meta-analysis, as it looks at the effect of NPIs on growth rates anddoes not include an estimate of the effect on total mortality.

28-Jun-21

PLOS ONEThe study is not included in the meta-analysis, as it looks at the effect of NPIs on growth rates anddoes not include an estimate of the effect on total mortality.

10-Jun-21

PLOS ONEThe study looks at the effect of SIPO's on growth rates but does include an estimate of the effect on
total mortality after three weeks (35% reduction in deaths) which is used in the meta-analysis.

15-Jan-21

Morbidity
andMortality
WeeklyReportFor each 1-unit increase in OxCGRT stringency index, the cumulative mortality decreases by 0.55
deaths per 100,000. The common estimate is the average effect in Europe and United States
respectively calculated as (Actual COVID-19 mortality) / (COVID-19 mortality with recommendation
policy) -1, where (COVID-19 mortality with recommendation policy) is calculated as ((Actual COVID19 mortality) - Estimate x Difference in stringency x population). Stringencies in Europe and UnitedStates are equal to the average stringency from March 16th to April 15th 2020 (76 and 74
respectively) and the stringency for the policy based solely on recommendations is 44 following Haleet al. (2020).
New Zealand We use the two graphs to the left in figure 3, where we extract the data from the rightmost datapoint
Economic(I.e. % impact of county lockdowns on Covid-19 deaths by 1/06/2020). We then take the average ofPapersthe estimates found in the two graphs, because it is unclear which estimate the author prefers.

Gibson (2020); "Government
mandated lockdowns do not
reduce Covid-19 deaths:
implications for evaluating
the stringent New Zealand
response"
Goldstein et al. (2021);"Lockdown Fatigue: The
Diminishing Effects of
Quarantines on the Spreadof COVID-19 "18-Aug20

Guo et al. (2021); "Mitigation
Interventions in the UnitedStates: An Exploratory
Investigation of
Determinants and Impacts"
Hale et al. (2020); "Global
assessment of therelationship between
government response
measures and COVID-19
deaths"Hunter et al. (2021); "Impact
of non-pharmaceuticalinterventions against
COVID-19 in Europe: A
quasi-experimental nonequivalent group and timeseries"

21-Sep-20 Research on
Social Work
Practice

4-Feb-21

CID Faculty
Working

We convert the effect in Figure 4 after 90 days (log difference -1.16 of a standard deviation change)
to deaths per million per stringency following footnote 3 (the footnote says "weekly deaths,” but webelieve this should be "daily deaths"), so the effect is e^-1.16 − 1 = −0.69 decline in daily deaths per
million per SD. We convert to total effect by multiplying with 90 days and "per point" by dividing with
SD = 22.3 (corresponding to the SD for the 147 countries with data before March 19, 2020 - using alldata yields similar results) yielding -2.77 deaths per million per stringency point. The common
estimate is the average effect in Europe and United States respectively calculated as (Actual COVID19 mortality) / (COVID-19 mortality with recommendation policy) -1, where (COVID-19 mortality
with recommendation policy) is calculated as ((Actual COVID-19 mortality) - Estimate x Difference instringency x population). Stringencies in Europe and United States are equal to the average stringency
from March 16th to April 15th 2020 (76 and 74 respectively) and the stringency for the policy based
solely on recommendations is 44 following Hale et al. (2020).
We use estimates for "Proportion of Cumulative Deaths Over the Population" (per 10,000) in Table 3.
We interpret this number as the change in cumulative deaths over the population in percent and istherefore the same as our common estimate.

6-Jul-20

medRxiv

The study is not included in the meta-analysis, as it looks at the effect of NPIs on growth rates anddoes not include an estimate of the effect on total mortality. They ascertain that "sustained over threemonths, this would correspond to a cumulative number of deaths 30% lower,” however this is not acounterfactual estimate and three months goes beyond the period they have data for.

15-Jul-21

Eurosurveillance

The study is not included in the meta-analysis, as they report the effect of NPIs in incident risk ratio
which are not easily converted to relative effects.

50

1. Study (Author & title)

2. DatePublished

3. Journal

Langeland et al. (2021); "The
Effect of State Level COVID19 Stay-at-Home Orders on
Death Rates"
Leffler et al. (2020);"Association of country-wide
coronavirus mortality withdemographics, testing,lockdowns, and public
wearing of masks"
Mccafferty and Ashley(2021); "Covid-19 Social
Distancing Interventions by
Statutory Mandate and Their
Observational Correlation toMortality in the UnitedStates and Europe"
Pan et al. (2020); "Covid-19:
Effectiveness of nonpharmaceutical interventionsin the united states before
phased removal of social
distancing protections varies
by region"
Pincombe et al. (2021); "Theeffectiveness of nationallevel containment andclosure policies acrossincome levels during the
COVID-19 pandemic: ananalysis of 113 countries"
Sears et al. (2020); "Are we#stayinghome to Flatten theCurve?"
Shiva and Molana (2021);"The Luxury of Lockdown"

5-Mar-21

Culture &CrisisConference

Spiegel and Tookes (2021);"Business restrictions andCovid-19 fatalities"

18-Jun-21

Stockenhuber (2020); "Did
We Respond Quickly
Enough? How PolicyImplementation Speed in
Response to COVID-19
Affects the Number of Fatal
Cases in Europe"
Stokes et al. (2020); "Therelative effects of nonpharmaceutical interventions
on early Covid-19 mortality:
natural experiment in 130
countries"

10-Nov20

4. Comments regarding meta-analysis
The study is not included in the meta-analysis, as it looks at the effect of NPIs on odds-ratios anddoes not include an estimate of the effect on total mortality.

26-Oct-20 ASTMH

Their "mask recommendation" includes some countries, where masks were mandated and may(partially) capture the effect of mask mandates. However, the authors' focus is on recommendation,so we do interpret their result as a voluntary effect - not an effect of mask mandate. Using estimatesfrom Table 2 and assuming NPIs were implemented March 15 (8 weeks in total by end of study
period), common estimates are calculated as 8^est-1.

27-Apr-21 PragmaticandObservation
al Research

The study is not included in the meta-analysis, as it looks at the effect of NPIs on peak mortality anddoes not include an estimate of the effect on total mortality.

20-Aug20

medRxiv

The study is not included in the meta-analysis, as the cluster the NPIs (e.g. SIPO, mask mandata amd
travel restricions are clustered in Level 4).

4-May-21

Health Policy Policy implementations were assigned according to the first day that a country received a policy
and Planning stringency rating above 0 in the OxCGRT stay-at-home measure. As the value 1 is a recommendation"recommend not leaving house,” we cannot distinguish recommendations from mandates, and, thus,
the study is not included in the meta-analysis.

6-Aug-20

medRxiv

9-Apr-21

6-Oct-20

Find that SIPOs lower mortality by 29-35%. We use the average (32%) as our common estimate.
Common standard errors are calculated based on estimates and standard errors from (Table 4)
assuming they are linearly related to estimates.
The
The estimate with 8 weeks lag is insignificant, and preferable given our empirical strategy. However,Europeanthey use the 4-week lag when elaborating the model to differentiate between high- and low-income
Journal ofcountries, so the 4-week lag estimate for rich countries is used in our meta-analysis. Common
Develepmen estimate is calculated as the average of the effect in Europe and United States, where the effect for
t Researcheach is calculated as (policy stringency - recommendation stringency) x estimate.
The Review
We use weighted average of estimates for Table 4, 6, and 9. Since authors state that they place moreof Financial
weight on the findings in Table 9, Table 9 weights by 50% while Table 4 and 6 weights by 25%. We
Studiesestimate the effect on total mortality from effect on growth rates based on authors calculation
showing that estimates of -0.049 and -0.060 reduces new deaths by 12.5% 15.3% respectively. We
use the same relative factor on other estimates.
World
When calculating arithmetic average / median, the study is included as 0%, because estimates in Table
Medical &6 are insignificant and signs of estimates are mixed (higher strictness can cause both fewer and more
Health Policy deaths). We don't calculate common standard errors.

medRxiv

We use estimates from regression on strictness alone (Right panel in Table "Regression results, policy
strictness. Baseline is "policy not introduced within policy analysis period" in "Additional file"). We use
the average of 24 and 38 days from model 5. There are 23 relevant estimates in total (they analyze alllevels within the eight NPI measures in the OxCGRT stringency index). We calculate the effect ofeach NPI (e.g. closing schools) as the average effect in all of U.S./Europe. This is done by calculating
the effect for each state/country based on the maximum level for each measure between Mar 16 andApr 15 (e.g. if all schools in a state/country are required to close (school closing level 3) the relevant
estimate for that state/level is -0.031 (average of -0.464 and 0.402). We assume all NPIs are effective
for 54 days (from March 15 to June 1 minus 24 days to reach full effect). Standard errors areconverted to common standard errors following the same process (this approach is unique for Stokes,
as our general approach is not possible).

51

1. Study (Author & title)

2. DatePublished

3. Journal

Toya and Skidmore (2020);"A Cross-Country Analysis of
the Determinants of Covid19 Fatalities"

1-Apr-20

CESifoWorking
PapersTsai et al. (2021);"Coronavirus Disease 2019
(COVID-19) Transmission in
the United States Before
Versus After Relaxation ofStatewide Social Distancing
Measures"3-Oct-20

Oxfordacademic

4. Comments regarding meta-analysis
It is unclear how they define "lockdown.” They write that "many countries [...] imposed lockdowns ofvarying degrees, some imposing mandatory nationwide lockdowns, restricting economic and social
activity deemed to be non-essential,” and since all European countries and all states in the U.S.
imposed restrictions on economic (closing unessential businesses) and/or social (limiting large
gatherings) activity, we interpret this as all European countries and all U.S. states had mandatory
nationwide lockdowns. The effect of recommended lockdowns is set to zero in the meta-analysis, as
only one country was in this lockdown category (i.e. too few observations, cf. eligibility criteria). The
estimate for complete travel closure is -0.226 COVID-deaths per 100,000. Hence, if all of Europe
imposed complete travel closure, the total effect would be -0.266 * 748 million (population) * 10(100,000/1,000,000) equal to 1,690 averted COVID-19 deaths. However, according to OxCGRT-data
European countries only had complete travel bans (Level 4: "Ban on all regions or total borderclosure") in 11% of the time between March 16 and April 15, 2020. So the total effect is 1,690 * 11%= 194 averted deaths. During the first wave 188,000 deaths in Europe was related to COVID-19 (byJune 30, 2020), so the total effect is approximated to -0.1% in Europe and, following the same logic,
0% in U.S., where no states closed their borders completely. We use the average, -0.05%, in the metaanalysis. The estimate for mandatory national lockdown is 0.166 (>0) COVID-deaths per 100,000.
Since all European countries (and U.S. states) imposed lockdowns, the total effect is 1,241 (553) extra
COVID-19 deaths corresponding to 0.7% (0.4%). We use the average of Europe and the U.S., 0.5%, inthe meta-analysis. Calculations of the effect of "Mandatory national lockdown" follow the same logic,
but we assume 100% of Europe and United States have had "Mandatory national lockdown.”
The study is not included in the meta-analysis, as they report the effect of NPIs on Rt which are noteasily converted to relative effects.

52

8

References

Abadie, Alberto. 2021. “Using Synthetic Controls: Feasibility, Data Requirements, andMethodological Aspects.” Journal of Economic Literature 59 (2):391–425.
https://doi.org/10.1257/jel.20191450.
Alderman, Jillian, and Maretno Harjoto. 2020. “COVID-19: US Shelter-in-Place Orders and
Demographic Characteristics Linked to Cases, Mortality, and Recovery Rates.”
Transforming Government: People, Process and Policy ahead-of-print (ahead-of-print).
https://doi.org/10.1108/TG-06-2020-0130.
Alemán, Christian, Christopher Busch, Alexander Ludwig, and Raül Santàeulalia-Llopis. 2020.
“Evaluating the Effectiveness of Policies Against a Pandemic.” ZEW - Centre forEuropean Economic Research Discussion Paper.
Allen, Douglas W. 2021. “Covid-19 Lockdown Cost/Benefits: A Critical Assessment of the
Literature.” International Journal of the Economics of Business, September, 1–32.
https://doi.org/10.1080/13571516.2021.1976051.
An, Brian Y., Simon Porcher, Shui‐Yan Tang, and Eunji Emily Kim. 2021. “Policy Design for
COVID ‐19: Worldwide Evidence on the Efficacies of Early Mask Mandates and Other
Policy Interventions.” Public Administration Review 81 (6):1157–82.
https://doi.org/10.1111/puar.13426.
Aparicio, Ainoa, and Shoshana Grossbard. 2021. “Are COVID Fatalities in the US Higher thanin the EU, and If so, Why?” Review of Economics of the Household 19 (2):307–26.
https://doi.org/10.1007/s11150-020-09532-9.
Arnarson, Björn Thor. 2021. “Breaks and Breakouts: Explaining the Persistence of COVID-19.”
SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3775506.
Ashraf, Badar Nadeem. 2020. “Socioeconomic Conditions, Government Interventions and
Health Outcomes during COVID-19,” July.
http://dx.doi.org/10.13140/RG.2.2.21141.55520.
Atkeson, Andrew. 2021. “A Parsimonious Behavioral SEIR Model of the 2020 COVID
Epidemic in the United States and the United Kingdom,” February, w28434.
https://doi.org/10.3386/w28434.
Atkeson, Andrew, Karen Kopecky, and Tao Zha. 2020. “Four Stylized Facts about COVID-19.”
NBER Working Paper, August, 44. https://doi.org/10.3386/w27719.
Auger, Katherine A., Samir S. Shah, Troy Richardson, David Hartley, Matthew Hall, Amanda
Warniment, Kristen Timmons, et al. 2020. “Association Between Statewide School
Closure and COVID-19 Incidence and Mortality in the US.” JAMA 324 (9):1–13.
https://doi.org/10.1001/jama.2020.14348.
Bakolis, Ioannis, Robert Stewart, David Baldwin, Jane Beenstock, Paul Bibby, Matthew
Broadbent, Rudolf Cardinal, et al. 2021. “Changes in Daily Mental Health Service Useand Mortality at the Commencement and Lifting of COVID-19 ‘Lockdown’ Policy in 10
UK Sites: A Regression Discontinuity in Time Design.” BMJ Open 11 (5). British
Medical Journal Publishing Group:e049721. https://doi.org/10.1136/bmjopen-2021049721.
Berardi, Chiara, Marcello Antonini, Mesfin G. Genie, Giovanni Cotugno, Alessandro Lanteri,
Adrian Melia, and Francesco Paolucci. 2020. “The COVID-19 Pandemic in Italy: Policy
and Technology Impact on Health and Non-Health Outcomes.” Health Policy andTechnology 9 (4):454–87. https://doi.org/10.1016/j.hlpt.2020.08.019.
53

Berry, Christopher R., Anthony Fowler, Tamara Glazer, Samantha Handel-Meyer, and Alec
MacMillen. 2021. “Evaluating the Effects of Shelter-in-Place Policies during the
COVID-19 Pandemic.” Proceedings of the National Academy of Sciences 118(15):e2019706118. https://doi.org/10.1073/pnas.2019706118.
Björk, Jonas, Kristoffer Mattisson, and Anders Ahlbom. 2021. “Impact of Winter Holiday andGovernment Responses on Mortality in Europe during the First Wave of the COVID-19
Pandemic.” European Journal of Public Health, February, ckab017.
https://doi.org/10.1093/eurpub/ckab017.
Bjørnskov, Christian. 2021a. “Did Lockdown Work? An Economist’s Cross-Country
Comparison.” CESifo Economic Studies 00 (00):14.
https://doi.org/10.1093/cesifo/ifab003.
———. 2021b. “Born et al. Om Epidemien i Sverige – Hvad Er Der Galt Og Hvordan Ser Det
Ud Nu?” Punditokraterne (blog). June 14, 2021.
http://punditokraterne.dk/2021/06/14/born-et-al-om-epidemien-i-sverige-hvad-er-dergalt-og-hvordan-ser-det-ud-nu/.
Blanco, Fernando, Drilona Emrullahu, and Raimundo Soto. 2020. “Do Coronavirus Containment
Measures Work? Worldwide Evidence,” Policy Research Working Papers, , December.
https://doi.org/10.1596/1813-9450-9490.
Bonardi, Jean-Philippe, Quentin Gallea, Dimtrija Kalanoski, and Rafael Lalive. 2020. “Fast andLocal: How Lockdown Policies Affect the Spread and Severity of Covid-19.” CEPR
Covid Economics, 27.
Bongaerts, Dion, Francesco Mazzola, and Wolf Wagner. 2021. “Closed for Business: The
Mortality Impact of Business Closures during the Covid-19 Pandemic.” PLOS ONE 16(5). Public Library of Science:e0251373. https://doi.org/10.1371/journal.pone.0251373.
Book, Joakim. 2020. “Oxford’s Stringency Index Is Falling Apart – AIER.” December 24, 2020.
https://www.aier.org/article/oxfords-stringency-index-is-falling-apart/.
Born, Benjamin, Alexander M. Dietrich, and Gernot J. Müller. 2021. “The Lockdown Effect: A
Counterfactual for Sweden.” PLOS ONE 16 (4). Public Library of Science:e0249732.
https://doi.org/10.1371/journal.pone.0249732.
Brodeur, Abel, David Gray, Anik Islam, and Suraiya Bhuiyan. 2021. “A Literature Review of the
Economics of COVID‐19.” Journal of Economic Surveys, April, joes.12423.
https://doi.org/10.1111/joes.12423.
Cerqueti, Roy, Raffaella Coppier, Alessandro Girardi, and Marco Ventura. 2021. “The Sooner
the Better: Lives Saved by the Lockdown during the COVID-19 Outbreak. The Case of
Italy.” ArXiv:2101.11901 [Econ], January. http://arxiv.org/abs/2101.11901.
Chaudhry, Rabail, George Dranitsaris, Talha Mubashir, Justyna Bartoszko, and Sheila Riazi.
2020. “A Country Level Analysis Measuring the Impact of Government Actions, Country
Preparedness and Socioeconomic Factors on COVID-19 Mortality and Related Health
Outcomes.” EClinicalMedicine 25 (August):100464.
https://doi.org/10.1016/j.eclinm.2020.100464.
Chernozhukov, Victor, Hiroyuki Kasahara, and Paul Schrimpf. 2021. “Causal Impact of Masks,
Policies, Behavior on Early Covid-19 Pandemic in the U.S.” Journal of Econometrics,
Pandemic Econometrics, 220 (1):23–62. https://doi.org/10.1016/j.jeconom.2020.09.003.
Chisadza, Carolyn, Matthew Clance, and Rangan Gupta. 2021. “Government Effectiveness andthe COVID-19 Pandemic.” Sustainability 13 (6). Multidisciplinary Digital Publishing
Institute:3042. https://doi.org/10.3390/su13063042.
54

Cho, Sang-Wook (Stanley). 2020. “Quantifying the Impact of Nonpharmaceutical Interventions
during the COVID-19 Outbreak: The Case of Sweden.” The Econometrics Journal 23(3):323–44. https://doi.org/10.1093/ectj/utaa025.
Coccia, Mario. 2021. “Different Effects of Lockdown on Public Health and Economy ofCountries: Results from First Wave of the COVID-19 Pandemic.” Journal of Economics
Library 8 (1):45–63. https://doi.org/10.1453/jel.v8i1.2183.
Conyon, Martin J., Lerong He, and Steen Thomsen. 2020a. “Lockdowns and COVID-19 Deathsin Scandinavia.” CEPR Covid Economics. https://doi.org/10.2139/ssrn.3616969.
———. 2020b. “Lockdowns and COVID-19 Deaths in Scandinavia.” SSRN Electronic Journal,
June. https://doi.org/10.2139/ssrn.3616969.
Conyon, Martin J., and Steen Thomsen. 2021. “COVID-19 in Scandinavia.”
https://doi.org/10.2139/ssrn.3793888.
Dave, Dhaval, Andrew I. Friedson, Kyutaro Matsuzawa, and Joseph J. Sabia. 2021. “When Do
Shelter-in-Place Orders Fight Covid-19 Best? Policy Heterogeneity Across States andAdoption Time.” Economic Inquiry 59 (1):29–52. https://doi.org/10.1111/ecin.12944.
Dave, Dhaval, Andrew Friedson, Kyutaro Matsuzawa, Drew McNichols, and Joseph Sabia.
2020. “Did the Wisconsin Supreme Court Restart a Covid-19 Epidemic? Evidence from a
Natural Experiment.” SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3620628.
Dergiades, Theologos, Costas Milas, Theodore Panagiotidis, and Elias Mossialos. 2020.
“Effectiveness of Government Policies in Response to the COVID-19 Outbreak.” SSRNElectronic Journal, August. https://doi.org/10.2139/ssrn.3602004.
Doucouliagos, Hristos, and Martin Paldam. 2008. “Aid Effectiveness on Growth: A Meta
Study.” European Journal of Political Economy 24 (1):1–24.
https://doi.org/10.1016/j.ejpoleco.2007.06.002.
Duchemin, Louis, Philippe Veber, and Bastien Boussau. 2020. “Bayesian Investigation of
SARS-CoV-2-Related Mortality in France,” June.
https://doi.org/10.1101/2020.06.09.20126862.
ECDC. 2020. “COVID-19 in Children and the Role of School Settings in Transmission - First
Update.” https://www.ecdc.europa.eu/en/publications-data/children-and-school-settingscovid-19-transmission.
Fakir, Adnan M. S., and Tushar Bharati. 2021. “Pandemic Catch-22: The Role of Mobility
Restrictions and Institutional Inequalities in Halting the Spread of COVID-19.” PLOS
ONE 16 (6). Public Library of Science:e0253348.
https://doi.org/10.1371/journal.pone.0253348.
Ferguson, Neil M, Daniel Laydon, Gemma Nedjati-Gilani, Natsuko Imai, Kylie Ainslie, Marc
Baguelin, Sangeeta Bhatia, et al. 2020. “Impact of Non-Pharmaceutical Interventions(NPIs) to Reduce COVID- 19 Mortality and Healthcare Demand,” March, 20.
Flaxman, Seth, Swapnil Mishra, Axel Gandy, H. Juliette T. Unwin, Thomas A. Mellan, Helen
Coupland, Charles Whittaker, et al. 2020. “Estimating the Effects of Non-Pharmaceutical
Interventions on COVID-19 in Europe.” Nature 584 (7820):257–61.
https://doi.org/10.1038/s41586-020-2405-7.
Fowler, James H., Seth J. Hill, Remy Levin, and Nick Obradovich. 2021. “Stay-at-Home Orders
Associate with Subsequent Decreases in COVID-19 Cases and Fatalities in the UnitedStates.” PLOS ONE 16 (6). Public Library of Science:e0248849.
https://doi.org/10.1371/journal.pone.0248849.

55

Frey, Carl Benedikt, Chinchih Chen, and Giorgio Presidente. 2020. “Democracy, Culture, andContagion: Political Regimes and Countries’ Responsiveness to Covid-19.” CEPR Covid
Economics. https://cepr.org/sites/default/files/CovidEconomics18.pdf.
Friedson, Andrew I., Drew McNichols, Joseph J. Sabia, and Dhaval Dave. 2021. “Shelter-inPlace Orders and Public Health: Evidence from California During the Covid-19
Pandemic.” Journal of Policy Analysis and Management 40 (1):258–83.
https://doi.org/10.1002/pam.22267.
Fuller, James A., Avi Hakim, Kerton R. Victory, Kashmira Date, Michael Lynch, Benjamin
Dahl, and Olga Henao. 2021. “Mitigation Policies and COVID-19–Associated Mortality— 37 European Countries, January 23–June 30, 2020.” Morbidity and Mortality Weekly
Report 70 (2):58–62. https://doi.org/10.15585/mmwr.mm7002e4.
Ghosh, Subhas Kumar, Sachchit Ghosh, and Sai Shanmukha Narumanchi. 2020. “A Study on
The Effectiveness of Lock-down Measures to Control The Spread of COVID-19.”
ArXiv:2008.05876 [Physics], August. http://arxiv.org/abs/2008.05876.
Gibson, John. 2020. “Government Mandated Lockdowns Do Not Reduce Covid-19 Deaths:
Implications for Evaluating the Stringent New Zealand Response.” New Zealand
Economic Papers, November, 1–12. https://doi.org/10.1080/00779954.2020.1844786.
Goldstein, Patricio, Eduardo Levy Yeyati, and Luca Sartorio. 2021. “Lockdown Fatigue: The
Diminishing Effects of Quarantines on the Spread of COVID-19,” June.
https://doi.org/10.21203/rs.3.rs-621368/v1.
Goolsbee, Austan, and Chad Syverson. 2021. “Fear, Lockdown, and Diversion: Comparing
Drivers of Pandemic Economic Decline 2020.” Journal of Public Economics 193(January):104311. https://doi.org/10.1016/j.jpubeco.2020.104311.
Gordon, Daniel V., R. Quentin Grafton, and Stein Ivar Steinshamn. 2020. “Statistical Analysesof the Public Health and Economic Performance of Nordic Countries in Response to the
COVID-19 Pandemic,” November, 2020.11.23.20236711.
https://doi.org/10.1101/2020.11.23.20236711.
GRADEpro. 2013. “GRADE Handbook.” October 2013.
https://gdt.gradepro.org/app/handbook/handbook.html.
Guallar, María Pilar, Rosa Meiriño, Carolina Donat-Vargas, Octavio Corral, Nicolás Jouvé, andVicente Soriano. 2020. “Inoculum at the Time of SARS-CoV-2 Exposure and Risk of
Disease Severity.” International Journal of Infectious Diseases 97 (August):290–92.
https://doi.org/10.1016/j.ijid.2020.06.035.
Guo, Shenyang, Ruopeng An, Timothy D. McBride, Danlin Yu, Linyun Fu, and Yuanyuan
Yang. 2021. “Mitigation Interventions in the United States: An Exploratory Investigationof Determinants and Impacts.” Research on Social Work Practice 31 (1):26–41.
https://doi.org/10.1177/1049731520957415.
Gupta, Sumedha, Kosali Simon, and Coady Wing. 2020. “Mandated and Voluntary Social
Distancing During The COVID-19 Epidemic: A Review.” NBER Working Paper Series,
June, w28139. https://doi.org/10.3386/w28139.
Hale, Thomas, Noam Angrist, Rafael Goldszmidt, Beatriz Kira, Anna Petherick, Toby Phillips,
Samuel Webster, et al. 2021. “Variation in Government Responses to COVID-19.”
Nature Human Behaviour 5 (4):529–38. https://doi.org/10.1038/s41562-021-01079-8.
Hale, Thomas, Andrew J. Hale, Beatriz Kira, Anna Petherick, Toby Phillips, Devi Sridhar, Robin
N. Thompson, Samuel Webster, and Noam Angrist. 2020. “Global Assessment of the56Relationship between Government Response Measures and COVID-19 Deaths,” July,
2020.07.04.20145334. https://doi.org/10.1101/2020.07.04.20145334.
Helsingen, Lise M., Erle Refsum, Dagrun Kyte Gjøstein, Magnus Løberg, Michael Bretthauer,
Mette Kalager, Louise Emilsson, and for the Clinical Effectiveness Research group.
2020. “The COVID-19 Pandemic in Norway and Sweden – Threats, Trust, and Impact on
Daily Life: A Comparative Survey.” BMC Public Health 20 (1):1597.
https://doi.org/10.1186/s12889-020-09615-3.
Herby, Jonas. 2021. “A First Literature Review: Lockdowns Only Had a Small Effect on
COVID-19.” SSRN Electronic Journal. https://dx.doi.org/10.2139/ssrn.3764553.
Herby, Jonas, Lars Jonung, and Steve H. Hanke. 2021. “Protocol for ‘What Does the First XX
Studies Tell Us about the Effects of Lockdowns on Mortality? A Systematic Review andMeta-Analysis of COVID-19 Lockdowns.’” SSRN Electronic Journal.
https://doi.org/10.2139/ssrn.3872977.
Homburg, Stefan, and Christof Kuhbandner. 2020. “Comment on Flaxman et al. (2020, Nature:
The Illusory Effects of Non-Pharmaceutical Interventions on COVID-19 in Europe.”
Nature 584 (7820):257–61.
Hunter, Paul R, Felipe J Colón-González, Julii Brainard, and Steven Rushton. 2021. “Impact of
Non-Pharmaceutical Interventions against COVID-19 in Europe in 2020: A QuasiExperimental Non-Equivalent Group and Time Series Design Study.” Eurosurveillance
26 (28). https://doi.org/10.2807/1560-7917.ES.2021.26.28.2001401.
Irfan, Omar, Jiang Li, Kun Tang, Zhicheng Wang, and Zulfiqar A Bhutta. 2021. “Risk of
Infection and Transmission of SARS-CoV-2 among Children and Adolescents in
Households, Communities and Educational Settings: A Systematic Review and MetaAnalysis.” Journal of Global Health 11 (July):05013.
https://doi.org/10.7189/jogh.11.05013.
Jefferson, Tom, Chris B Del Mar, Liz Dooley, Eliana Ferroni, Lubna A Al-Ansary, Ghada A
Bawazeer, Mieke L van Driel, et al. 2020. “Physical Interventions to Interrupt or Reduce
the Spread of Respiratory Viruses.” Edited by Cochrane Acute Respiratory Infections
Group. Cochrane Database of Systematic Reviews 2020 (11).
https://doi.org/10.1002/14651858.CD006207.pub5.
Johanna, Nadya, Henrico Citrawijaya, and Grace Wangge. 2020. “Mass Screening vs Lockdownvs Combination of Both to Control COVID-19: A Systematic Review.” Journal of Public
Health Research, 9. https://dx.doi.org/10.4081%2Fjphr.2020.2011.
Jonung, Lars, and Steve H. Hanke. 2020. “Freedom and Sweden’s Constitution.” Wall Street
Journal, May 20, 2020. https://www.wsj.com/articles/freedom-and-swedens-constitution11589993183.
Jonung, Lars, and Werner Röger. 2006. ”The Macroeconomic Effects of a Pandemic in Europe.
A Model-Based Assessment”, European Economy, Economic papers, nr 251, juni, 2006.
European Commission. Brussels.
https://ec.europa.eu/economy_finance/publications/pages/publication708_en.pdf
Juranek, Steffen, and Floris T. Zoutman. 2021. “The Effect of Non-Pharmaceutical Interventions
on the Demand for Health Care and on Mortality: Evidence from COVID-19 in
Scandinavia.” Journal of Population Economics 34 (4):1299–1320.
https://doi.org/10.1007/s00148-021-00868-9.

57

Kapoor, Mudit, and Shamika Ravi. 2020. “Impact of National Lockdown on COVID-19 Deathsin Select European Countries and the US Using a Changes-in-Changes Model.”ArXiv:2006.12251 [Physics, q-Bio, q-Fin], June. http://arxiv.org/abs/2006.12251.
Kepp, Kasper Planeta, and Christian Bjørnskov. 2021. “Lockdown Effects on Sars-CoV-2
Transmission – The Evidence from Northern Jutland.” MedRxiv, January.
https://doi.org/10.1101/2020.12.28.20248936.
Laliotis, Ioannis, and Dimitrios Minos. 2020. “Spreading the Disease: The Role of Culture.”
CEPR Covid Economics, June. https://doi.org/10.31235/osf.io/z4ndc.
Langeland, Andy, Jose Marte, and Kyle Connif. 2021. “The Effect of State Level COVID-19
Stay-at-Home Orders on Death Rates,” March, 23.
Leffler, Christopher T., Edsel Ing, Joseph D. Lykins, Matthew C. Hogan, Craig A. McKeown,
and Andrzej Grzybowski. 2020. “Association of Country-Wide Coronavirus Mortality
with Demographics, Testing, Lockdowns, and Public Wearing of Masks.” The American
Journal of Tropical Medicine and Hygiene 103 (6):2400–2411.
https://doi.org/10.4269/ajtmh.20-1015.
Lemoine, Philippe. 2020. “Lockdowns, Science and Voodoo Magic.” Nec Pluribus Impar.
December 4, 2020. https://necpluribusimpar.net/lockdowns-science-and-voodoo-magic/.
Lewis, Nic. 2020. “Did Lockdowns Really Save 3 Million COVID-19 Deaths, as Flaxman et al.
Claim?” Climate Etc. June 21, 2020. https://judithcurry.com/2020/06/21/did-lockdownsreally-save-3-million-covid-19-deaths-as-flaxman-et-al-claim/.
Li, Yanni, Mingming Liang, Liang Gao, Mubashir Ayaz Ahmed, John Patrick Uy, Ce Cheng,
Qin Zhou, and Chenyu Sun. 2021. “Face Masks to Prevent Transmission of COVID-19:
A Systematic Review and Meta-Analysis.” American Journal of Infection Control 49(7):900–906. https://doi.org/10.1016/j.ajic.2020.12.007.
Lipp, Allyson, and Peggy Edwards. 2014. “Disposable Surgical Face Masks for Preventing
Surgical Wound Infection in Clean Surgery.” In Cochrane Database of Systematic
Reviews, edited by The Cochrane Collaboration, CD002929.pub2. Chichester, UK: John
Wiley & Sons, Ltd. https://doi.org/10.1002/14651858.CD002929.pub2.
Liu, Ian T., Vinay Prasad, and Jonathan J. Darrow. 2021. “Evidence for Community Cloth Face
Masking to Limit the Spread of SARS-CoV-2: A Critical Review,” November.
https://www.cato.org/working-paper/evidence-community-cloth-face-masking-limitspread-sars-cov-2-critical-review.
Mader, Sebastian, and Tobias Rüttenauer. 2021. “The Effects of Non-Pharmaceutical
Interventions on COVID-19-Related Mortality: A Generalized Synthetic Control
Approach across 169 Countries.” SocArXiv. https://doi.org/10.31235/osf.io/v2ef8.
Matzinger, Polly, and Jeff Skinner. 2020. “Strong Impact of Closing Schools, Closing Bars andWearing Masks during the COVID-19 Pandemic: Results from a Simple and Revealing
Analysis.” https://doi.org/10.1101/2020.09.26.20202457.
Mccafferty, Sean, and Sean Ashley. 2021. “Covid-19 Social Distancing Interventions by
Statutory Mandate and Their Observational Correlation to Mortality in the United Statesand Europe.” Pragmatic and Observational Research 12 (April). Dove Press:15–24.
https://doi.org/10.2147/POR.S298309.
Moher, D., A. Liberati, J. Tetzlaff, D. G Altman, and for the PRISMA Group. 2009. “Preferred
Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement.”
BMJ 339 (jul21 1):b2535–b2535. https://doi.org/10.1136/bmj.b2535.

58

Neidhöfer, Guido, and Claudio Neidhöfer. 2020. “The Effectiveness of School Closures andOther Pre-Lockdown COVID-19 Mitigation Strategies in Argentina, Italy, and South
Korea.” SSRN Electronic Journal, July. https://doi.org/10.2139/ssrn.3649953.
Nussbaumer-Streit, Barbara, Verena Mayr, Andreea Iulia Dobrescu, Andrea Chapman, Emma
Persad, Irma Klerings, Gernot Wagner, et al. 2020. “Quarantine Alone or in Combination
with Other Public Health Measures to Control COVID-19: A Rapid Review.” Edited by
Cochrane Infectious Diseases Group. Cochrane Database of Systematic Reviews, April.
https://doi.org/10.1002/14651858.CD013574.
Nuzzo, Jennifer B, Lucia Mullen, Michael Snyder, Anita Cicero, Thomas V Inglesby, Amesh A
Adalja, Nancy Connell, et al. 2019. “Preparedness for a High-Impact Respiratory
Pathogen Pandemic,” September, 84.
Paldam, Martin. 2015. “Meta-Analysis in a Nutshell: Techniques and General Findings.”
Economics 9 (1):20150011. https://doi.org/10.5018/economics-ejournal.ja.2015-11.
Pan, William K., Stefanos Tyrovolas, Giné-Vázquez Iago, Rishav Raj Dasgupta, Fernández
Daniel, Ben Zaitchik, Paul M. Lantos, and Christopher W. Woods. 2020. “COVID-19:
Effectiveness of Non-Pharmaceutical Interventions in the United States before Phased
Removal of Social Distancing Protections Varies by Region.”
https://doi.org/10.1101/2020.08.18.20177600.
Patel, Urvish, Preeti Malik, Deep Mehta, Dhaivat Shah, Raveena Kelkar, Candida Pinto, Maria
Suprun, Mandip Dhamoon, Nils Hennig, and Henry Sacks. 2020. “Early Epidemiological
Indicators, Outcomes, and Interventions of COVID-19 Pandemic: A Systematic Review.”
Journal of Global Health 10 (2):020506. https://doi.org/10.7189/jogh.10.020506.
Perra, Nicola. 2020. “Non-Pharmaceutical Interventions during the COVID-19 Pandemic: A
Rapid Review.” ArXiv:2012.15230 [Physics], December.
http://arxiv.org/abs/2012.15230.
Pincombe, Morgan, Victoria Reese, and Carrie B Dolan. 2021. “The Effectiveness of NationalLevel Containment and Closure Policies across Income Levels during the COVID-19
Pandemic: An Analysis of 113 Countries.” Health Policy and Planning 36 (7):1152–62.
https://doi.org/10.1093/heapol/czab054.
Poeschl, Johannes, and Rasmus Bisgaard Larsen. 2021. “How Do Non- Pharmaceutical
Interventions Affect the Spread of COVID-19? A Literature Review.” Nationalbanken
Economic Memo, no. 4:20.
https://www.nationalbanken.dk/en/publications/Documents/2021/04/Economic%20Mem
o%20nr.%204-2021.pdf.
Pozo-Martin, Francisco, Florin Cristea, Charbel El Bcheraoui, and Robert Koch-Institut. 2020.
“Rapid Review Der Wirksamkeit Nicht-Pharmazeutischer Interventionen Bei Der
Kontrolle Der COVID-19-Pandemie.” Robert Koch, September, 17.
https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Projekte_RKI/RapidReview-NPIs.pdf?__blob=publicationFile.
Reinbold, Gary W. 2021. “Effect of Fall 2020 K-12 Instruction Types on COVID-19 Cases,
Hospital Admissions, and Deaths in Illinois Counties.” American Journal of Infection
Control 0 (0). Elsevier. https://doi.org/10.1016/j.ajic.2021.05.011.
Rezapour, Aziz, Aghdas Souresrafil, Mohammad Mehdi Peighambari, Mona Heidarali, andMahsa Tashakori-Miyanroudi. 2021. “Economic Evaluation of Programs against
COVID-19: A Systematic Review.” International Journal of Surgery 85 (January):10–
18. https://doi.org/10.1016/j.ijsu.2020.11.015.
59

Robinson, Oliver. 2021. “COVID-19 Lockdown Policies: An Interdisciplinary Review.” SSRNElectronic Journal. https://doi.org/10.2139/ssrn.3782395.
Sears, James, J. Miguel Villas-Boas, Vasco Villas-Boas, and Sofia Berto Villas-Boas. 2020.
“Are We #Stayinghome to Flatten the Curve?,” August, 2020.05.23.20111211.
https://doi.org/10.1101/2020.05.23.20111211.
Sebhatu, Abiel, Karl Wennberg, Stefan Arora-Jonsson, and Staffan I. Lindberg. 2020.
“Explaining the Homogeneous Diffusion of COVID-19 Nonpharmaceutical Interventions
across Heterogeneous Countries.” PNAS, August, 202010625.
https://doi.org/10.1073/pnas.2010625117.
Shenoy, Ajay, Bhavyaa Sharma, Guanghong Xu, Rolly Kapoor, Haedong Aiden Rho, andKinpritma Sangha. 2022. “God Is in the Rain: The Impact of Rainfall-Induced Early
Social Distancing on COVID-19 Outbreaks.” Journal of Health Economics 81(January):102575. https://doi.org/10.1016/j.jhealeco.2021.102575.
Shiva, Mehdi, and Hassan Molana. 2021. “The Luxury of Lockdown.” The European Journal of
Development Research, April. https://doi.org/10.1057/s41287-021-00389-x.
Siedner, Mark J., Guy Harling, Zahra Reynolds, Rebecca F. Gilbert, Sebastien Haneuse,
Atheendar S. Venkataramani, and Alexander C. Tsai. 2020. “Social Distancing to Slow
the US COVID-19 Epidemic: Longitudinal Pretest–Posttest Comparison Group Study.”
PLOS Medicine 17 (8). Public Library of Science:e1003244.
https://doi.org/10.1371/journal.pmed.1003244.
Spiegel, Matthew, and Heather Tookes. 2021. “Business Restrictions and COVID-19 Fatalities.”
Edited by Itay Goldstein. The Review of Financial Studies, June, hhab069.
https://doi.org/10.1093/rfs/hhab069.
Stanley, T.D., and Hristos Doucouliagos. 2010. “PICTURE THIS: A SIMPLE GRAPH THATREVEALS MUCH ADO ABOUT RESEARCH.” Journal of Economic Surveys 24(1):170–91. https://doi.org/10.1111/j.1467-6419.2009.00593.x.
Stephens, M., J. Berengueres, S. Venkatapuram, and I. A. Moonesar. 2020. “Does the Timing ofGovernment COVID-19 Policy Interventions Matter? Policy Analysis of an Original
Database.” https://doi.org/10.1101/2020.11.13.20194761.
Stockenhuber, Reinhold. 2020. “Did We Respond Quickly Enough? How Policy-Implementation
Speed in Response to COVID-19 Affects the Number of Fatal Cases in Europe.” World
Medical & Health Policy 12 (4):413–29. https://doi.org/10.1002/wmh3.374.
Stokes, Jonathan, Alex James Turner, Laura Anselmi, Marcello Morciano, and Thomas Hone.
2020. “The Relative Effects of Non-Pharmaceutical Interventions on Early Covid-19
Mortality: Natural Experiment in 130 Countries,” October, 2020.10.05.20206888.
https://doi.org/10.1101/2020.10.05.20206888.
Subramanian, S. V., and Akhil Kumar. 2021. “Increases in COVID-19 Are Unrelated to Levelsof Vaccination across 68 Countries and 2947 Counties in the United States.” European
Journal of Epidemiology, September. https://doi.org/10.1007/s10654-021-00808-7.
Toya, Hideki, and Mark Skidmore. 2020. “A Cross-Country Analysis of the Determinants of
Covid-19 Fatalities.” CESifo Working Papers, April, 14.
Tsai, Alexander C, Guy Harling, Zahra Reynolds, Rebecca F Gilbert, and Mark J Siedner. 2021.
“Coronavirus Disease 2019 (COVID-19) Transmission in the United States Before
Versus After Relaxation of Statewide Social Distancing Measures.” Clinical Infectious
Diseases 73 (Supplement_2):S120–26. https://doi.org/10.1093/cid/ciaa1502.

60

Umer, Hamza, and Muhammad Salar Khan. 2020. “Evaluating the Effectiveness of Regional
Lockdown Policies in the Containment of Covid-19: Evidence from Pakistan.”
ArXiv:2006.02987 [Physics, q-Bio], June. http://arxiv.org/abs/2006.02987.
UNICEF. 2021. “In-Person Schooling and COVID-19-Transmission: Review of Evidence 2020.”
https://www.unicef.org/media/89046/file/In-person-schooling-and-covid-19transmission-review-of-evidence-2020.pdf.
World Health Organization Writing Group. 2006. “Nonpharmaceutical Interventions for
Pandemic Influenza, National and Community Measures.” Emerging Infectious Diseases
12 (1):88–94. https://doi.org/10.3201/eid1201.051371.
Wu, Samuel X., and Xin Wu. 2020. “Stay-at-Home and Face Mask Policies Intentions
Inconsistent with Incidence and Fatality during US COVID-19 Pandemic.”
https://doi.org/10.1101/2020.10.25.20219279.
Zhang, Mengxi, Siqin Wang, Tao Hu, Xiaokang Fu, Xiaoyue Wang, Yaxin Hu, Briana Halloran,
et al. 2021. “Human Mobility and COVID-19 Transmission: A Systematic Review and
Future Directions.” Preprint. Infectious Diseases (except HIV/AIDS).
https://doi.org/10.1101/2021.02.02.21250889.

61