West et al. (1) estimate the effectiveness of 24 REDD projects from six countries, representing around a quarter of projects registered under the Verified Carbon Standard (VCS). The core of their method is to match each REDD project area (PA) to synthetic control (SC) sites: circular proxies of the same size as the PA that are assumed to undergo the same trend in deforestation as the PA would have, if the REDD project had not been implemented. The area deforested in the PAs and SCs were estimated using a global scale forest loss dataset (2), and West et al. attempted to quantify the carbon credits successfully delivered according to the difference between deforestation in the PAs and the SCs. They concluded that the majority of projects did not generate any carbon benefits, and of those that did, most issued more carbon credits than were justified. However, this analysis did not provide an accurate estimate of carbon benefits of these projects for three reasons:

1. The satellite-derived data (2) they use in all analyses has errors and biases that its creators state make it inappropriate for this type of comparison. These led to effective projects being incorrectly labeled as ineffective; and a potential underestimate of avoided deforestation of 90% or more. 2. Their project-level SC and country-level Generalized Synthetic Control (GSC) methods failed to select appropriate donor areas, with the SC sets failing common tests in 24 of 30 cases. The resulting models are also highly unstable, with fine-tuning of the method causing wildly different results. This is likely due to omitting key variables, such as 'distance to roads'. 3. West et al. made two different calculation errors, that mean the proportion of credits they report that deliver real carbon benefits should be increased by 62%. West et al. further did not attempt to calculate the uncertainty on their estimates of carbon credits generated, nor did they adopt a formal hypothesis testing framework. These weaknesses have led to their results appearing more precise than they are, and may have led to a widespread misinterpretation that REDD projects are worthless (3). This is compounded by extending their conclusions beyond their dataset: they generalize about REDD success on the basis of a non-random sample of 24 projects from a potential dataset of 103 registered Verra REDD projects (4), itself a subset of 377 known REDD projects (5). Details on all these points follow below. West et al. based their analysis on the 'Global Forest Change' (GFC) dataset, a product designed for use as a "stratifier in targeting forest extent and/or change by a probability sample" (2), e.g. as used by Tyukavina et al. (6). The website where the data are accessed explicitly states that "definitive area estimation should not be made using pixel counts from the forest loss layers", and that the data does not meet standards for areal land-cover change estimation set by the Intergovernmental Panel on Climate Change (7). However, West et al. did not adopt a point-based sample approach, instead using the data directly. This inevitably adds great uncertainty to their analysis, and due to the nature of their comparisons (between REDD PAs and either control sites or REDD PAs before their start date) reduces the chances that a successful project will be identified as such, or its benefits fully quantified. The GFC data should not be used in this way because they contain false and missed detections that combine to reduce the apparent success of all projects. These errors can be large: for example, a study using field plots and Synthetic Aperture Radar data covering drier countries in sub-Saharan Africa, including all of Tanzania and Zambia, and the southern portion of DRC (3 of the 6 countries covered by West et al.) found the GFC data overestimated forest cover by a factor of two, but then underestimated forest loss rates by a factor of five, with these overall rates of error varying greatly by region and starting canopy cover (8). We calculate that in this region this error rate would lead to an 89% understatement of project carbon benefits, meaning a project that stopped all deforestation (100% successful) would appear to have only stopped 11% if the GFC data were used to do the assessment. The African regional analysis in West et al. found an overall effectiveness of 16% (414 ha year -1 or 423 ha year -1 effect size, depending on method, compared to 2700 ha year -1 baseline); the error rate in the GFC data used suggests this could really represent an effectiveness of well over 100%. We calculate this based on assuming the detection rate (Rd = proportion of true deforestation detected) and false alarm rate (Rf = proportion of stable forest marked as deforestation) were the same in the PA and the SC, then the area of avoided deforestation would be underestimated by a factor (Rd -Rf ). To understand this consider two sites with the same area A, of which Di is deforested: the predicted area of deforestation in site i will be:

The true difference in the areas of deforestation (DDif:1,2) predicted between sites 1 and 2 would therefore equal:

For study (8) Rd = 0.19 and Rf = 0.08, leading to the 89% underestimate of effectiveness. But even using the estimates for wet tropical regions by the providers of the GFC data (2), Rd = 0.83 and Rf = 0.002, leading to underestimation of real avoided deforestation by 17%. West et al could have corrected for this best case scenario by increasing their observed effectiveness in tables and figures by 17%, and recognising that the impact could be bigger in many areas. These errors are further compounded because the satellites used and methodology applied in the GFC data have changed significantly during the time period analyzed by West et al. In particular a change in method and the use of Landsat 8 caused a big increase in detections post 2012, with particularly increased sensitivity to degradation and small-scale deforestation (7,9).
This will have a spatially heterogeneous impact that may only become apparent while projects are in operation, but not during the training period of selecting Synthetic Controls. For example two sites, one with small-scale degradation and the other with no disturbance, might appear identical in GFC from 2000-2012, but diverge from 2013 onwards when the small-scale degradation starts being detected. This step change in sensitivity could significantly bias all the analyses in West et al. Leaving aside the quality of the underlying data, though it is itself sufficient to invalidate the results, we conducted an empirical evaluation of the synthetic control method (SCM) employed by West et al. SCMs were designed as a tool to understand the effects of state policy interventions (10), and in a conservation context it is vital that steps are taken to account for domain specific complexity, select appropriate covariates, and test that valid controls have been obtained (11). Here, we show that the approach taken by West et al. leads to inappropriate matching, ignores key drivers of deforestation, and that in most cases (24 of 30) the SCs set selected for PAs fail to pass standard validation tests. REDD baselines are created using reference regions that are located near to the project area, so that local laws, customs, forest characteristics, and pressures on forests are similar (though see Note 1). In contrast, West et al. draw potential donor areas from randomly generated circles over entire countries, in many cases leading to SCs that are incomparable. The covariates they use to generate SCs from the potential donors also lack critical variables: they do not include 'distance to roads' -identified by  as the single most important driver of deforestation risk. In fact, they only include two variables from the top five identified by Bush & Ferretti-Gallon, with 'Population' and 'Wealth' also omitted. The decision to include donor areas from entire countries leads to some poor comparisons. For example, two PAs in Peru and all Colombian PAs are compared to sites on the opposite side of the Andes (West et al. Fig. S4). Biomes, access to markets, diseases, and crops grown, are all vastly different on either side of the Andes in Peru and Colombia, all of which will influence deforestation dynamics. Elsewhere, West et al. make comparisons across huge distances, not least in the DRC -a country with a similar land area to western Europe, and corresponding huge variance in modes and types of agricultural production, transport networks, access to markets, laws and culture. A simple comparison to satellite data shows that only 1 of the 4 selected donor areas for site DRC 934 is in a closed canopy forest like the PA, with the other three in the savanna woodland ecosystem of southern DRC (Fig. 1). Even the single matched rainforest site is located over 600 km from the PA, at a much greater travel distance (multiple weeks by boat, the dominant means of transport in this region of DRC) from Kinshasa and its 17 million people. Even more concerning is that the chosen SCs for site DRC 934 all have 0% 'Logging concession cover', whereas the project has 100% 'Logging Concession Cover (West et al. Table A17). This variable was matched for, but is not a strict condition, so SCs were allowed if they fitted other variables better even if they failed entirely to match this condition. However, forests located in or out of a logging concession likely have a totally different risk profile: the REDD project activity here was to prevent a logging concession becoming active. This poor quality of matching is systematic: every PA in Peru that showed more deforestation than its SC was matched to an SC with less timber concession coverage; every Colombian PA that showed more deforestation than its SC was matched to an SC with a higher level of pre-existing legal protection; and over 80% of PAs were matched to an SC that had lower levels of deforestation in a 10 km buffer zone around it. Given these biases, a lack of significant difference in deforestation rates between PAs and SCs after project implementation is not surprising, and indeed could even itself indicate project success. In their Figure S6, West et al. show a "validation" in which they divide the pre-treatment period into two parts (training and validation periods). If the method is reliable, then the SC and the PA should show parallel deforestation trends in the validation period, but this requirement appears to be violated. The authors claim that 26 out of 30 PAs pass the validation by setting a threshold in terms of absolute deforestation by the final year, equal to a deviation of less than 0.5% of the total project area. Replicating this same test, we find that 24 of the PAs pass using normal SCM, and 25 pass using an Augmented synthetic control method (ASCM) proposed by Ben-Michael et al. (13). However, this "validation" cannot be considered a rigorous test. The absolute size of error allowed is very large, especially for PAs with low deforestation rates (10 of the 32 PAs had pre-project total deforestation less than 0.5%, meaning a deviation of over 100% is allowed in this test). The credibility of an SC estimator depends on its ability to track the trajectory of the outcome variable (in this case deforestation d) for an extended period in time t ( 14), so a better numerical test is given by:

< 0.2 (Equation 3)

In addition, following best practice (9) we applied a second test:

*+,-. (/"01&"'123) *+,-. ('4"13135)

< 5 (Equation 4)

These equations and the derivation of values 0.2 and 5 as test statistics follow normal practice in the SC literature, see Supplementary Information for more detail.
We find that only 7 out of 30 PAs pass the requirement in Equation 3, and that these results are also more consistent with the graphical trends: only PAs that exhibit a parallel trend with the SC pass the test (Fig S1  4, and suggest that if the authors truly wish to apply an "abundance of caution" (1), then they should apply both tests, which 24 out of 30 PAs would fail. Our results show the same pattern when using ASCM: 6 PAs pass both validation tests, 4 of which also passed the tests using SCM (see supplementary materials). Given this failure rate (and arguably even their stated failure rate of over 10% of PAs), it should have been concluded that their SC model was not reliable and required adjustments to its input datasets, rather than continuing with the analysis. It is likely that improving the number of variables known to be important for deforestation, using a more reliable remote sensing datasets, and putting in hard controls related to biomes or land use types available for SC matches, would have improved these results. Due to the flaws in the SCM approach discussed above, the results presented by West et al. are not robust to small changes in the algorithm used to generate SCs. This robustness testing is a common feature of SCM approaches, with reference 9 for example varying their models in many ways to see how the results change. To demonstrate this, we replicated their method with and without the application of a filter they used that prioritizes donor areas with similar rates of buffer deforestation. If the method were robust, we would expect the filter to have some effect, but for the overall results to be similar without it given that buffer deforestation remains one of the covariates. In fact, removing the filtering step leads to reversal of outcomes for 7 of the PAs. For example, Colombia 1566 has 7,000 ha less deforestation than its SC when the filter is used, but 2,500 ha more deforestation than its SC when the filter is removed. On average, these numbers (which West et al. carry forward into their calculation of carbon credits), differ by over 200%. Such discrepancies clearly point to a lack of robustness that should have been addressed through improving the models, before drawing any conclusions about the effectiveness of the REDD projects using these methods. This instability in results is seen to some degree in West et al's analysis. Their reported effectiveness for the same projects, using the same datasets, varies widely: for example overall effectiveness of all African projects covered varies from 0% (project-level SC analysis) to 16% (regional GSC) to 50% (regional GSC using cardinality matching by year 5, West et al. Table S12). The authors make two further errors. Firstly, they unfairly compare their estimates of avoided deforestation to the pre-project estimates of carbon credits. This ignores the fact that carbon credits are actually issued ex post following an independently verified assessment of performance, and the number is normally lower than the original estimate due to reductions for leakage and residual deforestation. In Secondly, when converting from avoided deforestation to carbon offsets (West et al. Table S1), the wrong numbers for "project baseline deforestation" are used. In all cases the deforestation between the year 2000 and the start of the project has been added to the baseline deforestation that occurred after the project started. This has the effect of inflating the baseline deforestation by a mean of 18.0% (range 0 -91% with bigger disparities for more recent projects), and means that the average project was given an "expected carbon offset per hectare of avoided deforestation" value of 207 Mg CO2 ha -1 , compared to a true value of 268 Mg CO2 ha -1 (Table S1). The exact magnitude of the error this causes varies by the set of projects West et al. consider to be successful in their different analyses, but we estimate it understates the carbon benefits of avoided deforestation by between 18 and 35%. These two errors are independent, and should be combined to understand the overall impact of these calculation errors. If this is done with West et al. 's individual project based analysis, the 'proportion of credits that deliver real carbon benefits' should be increased by 61.6% (Table S2), purely due to calculation missteps.While this appears to have only a small impact on West et al. 's estimate of the overall effectiveness of the REDD projects in their analysis (an increase in effectiveness from 6.1% to 9.8% is still far from 100%), when combined with uncertainties resulting from the input satellite data and poorly selected comparison sites, all of which likely decreased the observed estimated impact of a truly impactful project, they are significant. West et al. conclude that "only a minority of projects achieved statistically significant reductions in comparison with ex post counterfactuals", but we have shown that this result is highly uncertain given the lack of sensitivity of their approach given their input satellite data, use of inappropriate synthetic controls resulting in poor synergies between control and project sites and sensitivity of their results to small changes in methodology, and calculation errors. As such, we believe their paper should be retracted or heavily revised. It is certainly possible that some REDD projects have issued more credits than, in retrospect, was warranted: but that is not possible to prove using West et al. 's approach. We note that all these project areas have low rates of deforestation (average rate of 0.21%/yr according to West et al. figures, compared to an average of 0.47%/yr for forests in the six countries covered by West et al. ( 7)), despite often being located in higher risk areas. We call for future studies on REDD project effectiveness that use locally tuned forest change data with known accuracies or point-based sampling approaches to quantify deforestation.
Furthermore, analytical approaches must always lead to meaningful comparisons between forests of the same ecological type and legal status, and should consistently pass rigorous validation checks before conclusions are drawn from them. We suggest pixel based matching methods (e.g. 15, 16) may be more appropriate given the volatile nature of deforestation as an outcome variable. Finally, we suggest that broad conclusions about avoided deforestation projects as a whole should not only satisfy these requirements, but also require a formal hypothesis testing framework and feature a representative sample of projects. 1. Using spatially proximate reference regions can introduce another problem, that of spatial autocorrelation, but this problem has been known for many years, and techniques have been developed in spatial statistics to nullify its effect. Further, unaccounted for spatial autocorrelation will in the case of such post hoc analyses of the effectiveness of REDD projects act conservatively: it will decrease the likelihood that the null hypothesis that PAs and control areas are the same is rejected. We conducted an empirical evaluation of the synthetic control method (SCM) employed by West et al. (1). Our approach and the methodology are as close as possible, given the example scripts provided by the authors did not specify the years used to create training and validation periods or the exact cutoffs used for filtering potential donor areas by buffer deforestation. The analysis was conducted with the Synth and augsynth packages available for R software. We replicate the results in West et al. Figure S6 (our Figure S1) and Table S3 (Table S3) using the dataset made available by West et al. in each REDD+ project using both the SCM (that they adopt) and the ASCM. The simple SCM used in their analysis does not allow negative weights while ASCM relaxes this constraint allowing for negativity and extrapolation. ASCM is particularly useful for improving the covariate balance between treated and control groups and often leads to a better pretreatment fit (13). Weights are chosen to balance pre-treatment outcomes and other covariates as in the SCM. The first step when applying a SCM is to test the validity of the methodology. The approach consists of dividing the pre-treatment period in two parts (training and validation period) using a random year. This step is shown in fig. S3. We then apply three numerical tests to determine the quality of the fits in the validation period between project areas (PAs) and synthetic controls (SCs). In table S3 we replicate the same test used by West et al. We calculated the difference between project and synthetic control deforestation in the final year of the validation period and then we compared this number with the total project area. This is done for both the ASCM and the SCM. In table S4 we applied a variant of the validation test proposed by Abadie et al. (10). This test is based on the ratio between the root mean squared prediction error (RMSPE) for the validation period and for the training period. The smaller the value, the better is the validity of the SC. Based on experimentation and following guidance in the literature, we set 5 as a limit value for this ratio; if the post RMSPE / Pre RMSPE ratio is bigger than 5 the project fails the validation. The benefit of using this methodology is that it is not biased by the quality of the fit in the training period. In addition, we proposed a second more reasonable numerical test using the ratio between the maximum of the absolute value differences in the validation period and total deforestation. This approach not only allows us to capture the divergence between project and SCs across the entire validation period, but also eliminates the bias of preference of low deforestation areas. Empirically we choose 20% (of total deforestation in the last year of the validation period) as a threshold to pass the validation test, as this seemed to keep only projects where by eye the match appeared strong in the validation period. The results are shown in table S5. We continued to use ASCM as this algorithm led to better validation results and therefore should be more robust. Addressing the second concern around the sensitivity of SC results, we ran the analysis both with and without the filter applied by West et al. This filter selectively includes only those donors with deforestation patterns analogous to the project sites (ideally 10 percent deviation, but increasing this range to ensure an SC with a good match in the pre-implementation period is obtained). We used different percentage deviations (20 or 30 percent) for each country based on the ability of the SC to find enough controls for the analysis. Figure S2 shows the cumulative deforestation of the PAs relative to the SCs created with and without the filter. We replicate West et al. 's figures S7 and S8 in our figures S3 and S4 (without filtering) and figures S5 and S6 (with filtering). Finally, in Table S6 we show the average treatment effects on the treated (ATT) for each project when the filter is applied and when it is not, which we defined as the mean difference between the deforestation in the PS and the SC over the implementation period. *The large disparity in Peru 844 is due to the wrong number for baseline deforestation being taken from the project documents. Table S2: Corrected proportion of carbon credits linked to reduced deforestation using SC results. Replication of the result in (1) that 6.1% of carbon offsets are linked to reduced deforestation. Here, we make two main corrections: firstly we use the correct values for the number of offsets generated per hectare (Table S1); and secondly we compare to credits actually issued (using values from West et al. Table S4) instead of expected credits (as used in West et al. Table S5). Combined, this increases the proportion of "real" carbon credits according to their SC analysis by 62%. S3.
Synthetic control "validation": the difference between project and synthetic control deforestation in the final year of the validation period. The Difference in % is the % of the project area. Projects with a difference between project and synthetic control deforestation > 0.5% of the project area are assumed to have failed validation and are marked with an asterisk. Table S4: Post RMSPE / Pre RMSPE ratios. Validation test. ratio between the root mean squared prediction error (RMSPE) for the training period and for the validation period. Projects with a Post RMSPE / Pre RMSPE > 5 are assumed to have failed validation and are marked with an asterisk. Synthetic control validation: the maximum of the absolute value differences between project and synthetic control deforestation in each year of the validation period. The Difference in % is the % of the project deforestation in the final year of the validation period. Projects with a % difference > 20% are assumed to have failed validation and are marked with an asterisk. . "Validation" of the synthetic control method in each REDD+ project using an augmented synthetic control method (ASCM). Pre-REDD+ deforestation in "to-be" REDD+ project areas (red) versus synthetic controls (blue) Shaded blue areas represent the validation periods. Differences between the averages ATT for each project when the filter is applied (green line) and when it is not (orange line). The ATT is calculated as the difference between the deforestation in the project area and the synthetic control deforestation.