Addressing strengths and weaknesses of a multi ecosystem climate change experiment
Year: 2019

We thank M. Glenn for her critical review of our research (Mau et al. 2018) In response to her Letter to Editor entitled, "Sample Size in Linear Regression of the Effect of Priming on Soil Carbon and Nitrogen Across Ecosystems" (Glenn, 2018) we argue that using four ecosystems to investigate the impact of long term warming on priming is actually a strength of our study, as the majority of other priming studies to date focus on only one ecosystem. Across the four ecosystems we examined, mean annual temperature ranged from 6.6 degC to 12.8 degC, annual precipitation from 127mm to 543mm, and vegetation varied from high desert grassland to mixed conifer forest, together capturing an unusually broad sample of temperate terrestrial ecosystems for an empirical study. We agree that including more than four ecosystems would have further strengthened our ability to infer general patterns. However, replication at that level, and over a duration comparable to what we achieved, is logistically challenging for even the most generouslyfunded single empirical study. Meta analysis, which we certainly welcome, provides a way to overcome some of these challenges, but suffers from a different set of shortcomings, the most salient of which here is that it is premature: as far as we know, the four ecosystems we assessed were the first long term transplant studies to measure the appropriate variables to be able to compare concurrent changes in soil carbon and nitrogen as well as priming. We reiterate that each of the points displayed in the panels of Fig. 4 of Mau et al. (2018) represents the average of 1000 bootstrapped iterations of differences in carbon or nitrogen content and differences in priming between the ambient and transplanted soils. The bootstrap resampling technique was used to obtain 1000 Monte Carlo datasets, each of which yielded a linear regression and we calculated simple confidence intervals on the Monte Carlo slopes. Here, we have replicated the figure including the slopes of the resampled data to more thoroughly visualize uncertainty in the relationships  Because the ambient and transplanted soils were not replicated in a paired design, we resampled all individual measurements of ambient and transplanted soil nutrient content and priming to account for the full range of variability in our data. To determine the robustness of each of the relationships, we calculated the slope for each iteration and assessed the resulting distributions of slope parameters. When a null slope of zero fell outside the 2.5th and 97.5th percentiles of that distribution, we interpreted the overall linear relationship as different from zero. This is a standard statistical approach for propagating uncertainty in studies with unpaired observations (Manly, 2007) as well as for ecosystemlevel manipulation experiments, where strict guidelines of 50, 30, or even 20 truly independent replicates are rarely achieved. Glenn's critique also points to a feature of our experiment that is absolutely a limitation: the transplantation design requires different sites for the different treatments, a feature that challenges strong inference because any characteristic of the transplant site that differs from the control could contribute to the observed responses. Two design changes could have helped overcome this: 1) we could have used electrically powered infrared heat lamps to experimentally manipulate temperature within each experimental site (or some other in situ manipulation technique) avoiding the transplantation all together, or 2) we could have located other, nearby, mountain regions and transplanted plots independently from one region to another, with replicate control and transplanted sites across a regional suite of mountain gradients. Neither of these approaches was practical, the first requiring funds that were not awarded, the second requiring mountains that do not exist. One of the reasons our experiment has been able to persist for more than a decade on a tight budget is the low cost of transplantation as a way to simulate climate change. Although other site level differences besides temperature could in theory contribute to differences in soil carbon cycling between transplanted and ambient plots, evidence for such site level differences is weak. Rates of atmospheric nitrogen deposition are quite low in the region (NADP, 2018) We are also unaware of any empirical evidence for effects of variation in wind speed on soil carbon cycling, nor for the direct effects of the minor changes in solar radiation caused by elevation differences associated with the transplant design we implemented. Temperature is the strong and obvious main difference between the sites, though we agree with Glenn that we can never be sure. We disagree with Glenn that our study would be improved by recasting it as an investigation of the effect of 'elevation' In fact, elevation is only one of many differences between the sites, so this change offers no improvement over the fundamental problem of inference. Our designation of 'transplantation' throughout the manuscript is, in fact, more accurate, and more neutral. Still, our interpretation of the likely driver of observed differences as driven by warming is a simple application of Occam's razor, along with an acknowledgement that warming across the landscape is an observable, widespread, and rapid phenomenon with urgent implications for understanding soil carbon cycling. As scientists funded by public dollars, we are obligated to interpret our results logically, transparently, and by acknowledging their direct relevance to issues about which people need to make decisions. In this case, interpreting our experiment as an effect of 'elevation' on principles that amount to statistical and inferential minutiae effectively abnegates one of our fundamental obligations: to explain how results are