DOI: 10.5061/dryad.cz8w9gj8w
title: Data and code from: Coordinated distributed experiments in ecology do not consistently reduce heterogeneity in effect size

description: Ecological meta-analyses usually exhibit high relative heterogeneity of
 effect size: most among-study variation in effect size represents true
 variation in mean effect size, rather than sampling error. This
 heterogeneity arises from both methodological and ecological sources.
 Methodological heterogeneity is a nuisance that complicates the
 interpretation of data syntheses. One way to reduce methodological
 heterogeneity is via coordinated distributed experiments, in which
 investigators conduct the same experiment at different sites, using the
 same methods. We tested whether coordinated distributed experiments in
 ecology exhibit a) low heterogeneity in effect size, and b) lower
 heterogeneity than meta-analyses, using data on 17 effects from eight
 coordinated distributed experiments, and 406 meta-analyses. Consistent
 with our expectations, among-site heterogeneity typically comprised
 &lt;50% of the variance in effect size in distributed experiments. In
 contrast, heterogeneity within and among studies typically comprised
 &gt;90% of the variance in effect size in meta-analyses. However, this
 difference largely reflected the small size of most coordinated
 distributed experiments, and was no longer significant after controlling
 for size (number of studies or sites). These results are consistent with
 the hypothesis that methodological heterogeneity rarely comprises a
 substantial fraction of variance in effect size in ecology. We also
 conducted pairwise comparisons of absolute heterogeneity between
 coordinated distributed experiments and meta-analyses on the same topics.
 Coordinated distributed experiments did not consistently exhibit lower
 absolute heterogeneity in effect size than meta-analyses on the same
 topics. Our findings suggest that coordinated distributed experiments
 rarely increase uniformity of results by reducing methodological
 heterogeneity. Our results help refine the numerous distinct reasons for
 conducting coordinated distributed experiments.
description: # Coordinated distributed experiments in ecology do not consistently
 reduce heterogeneity in effect size Included here is a data file for a
 distributed experiment, and code which analyses the heterogeneity of many
 coordinated distributed experiments and meta-analyses. The R code file
 reproduces the results of this study, called meta-analyses vs distd expts
 - R code for sharing v 2.R. ## ## Description of the data and file
 structure Data File: rousk et al 2013 table 3 data - INCREASE.csv: data
 from the INCREASE distributed experiment by Rousk et al. (2013) All other
 data used in code is automatically sourced from URLs, but relevant
 variables are still described below. **Other variables in datasets were
 not used in our analysis, and so are not explained in this README file.**
 Cells with missing data have "NA" values. Variables used in
 code: Costello & Fox variables:  meta.analysis.id: Unique ID number
 for each meta-analysis eff.size: Effect size var. eff.size: Variance in
 effect size study: Unique names for all studies within all meta-analyses
 id.effect.within.study: Unique ID number for each effect within a study.
 Starts over at 1 for each study. PANAMEX variables: Sites: Sites where
 experiments were conducted Lat: Latitude coordinate Atl.Pac: Factor for
 whether site was in Atlantic (A) or Pacific (P) Treat: Treatments coded,
 1: open control, 2: full predator exclusion cage, 3: part cage, 4:
 exposure. 1 and 3 are considered the controls. Plate: Akin to an
 experimental plot, unique for each replicate Wt: Biomass, wet weight in g
 of each plate  Smith et al. 2024, DroughtNet variables: se: Sampling error
 site_code: Character code for each site mean_DS3: Mean drought response at
 each site, effect size n: Number of replicates at each site Rousk et al
 2013 Table 3 data, INCREASE variables: Ttmt: Control, warming, or drought
 treatment of plot bacterial.growth.mean: Mean estimated bacterial growth
 based on leucine incorporation bacterial.growth.sd: Standard deviation of
 estimated bacterial growth based on leucine incorporation
 fungal.growth.mean: Mean estimated fungal growth based on acetate
 incorporation into ergosterol fungal.growth.sd: Standard deviation of
 estimated fungal growth based on acetate incorporation into ergosterol
 fungal.PFLA.mean: Mean concentration fungal-biomass-specific PLFA markers
 fungal.PFLA.sd: Standard deviation of concentration
 fungal-biomass-specific PLFA markers site: Sites where experiments were
 conducted Fronhofer et al. 2018, Dispnet variables:  lab: Lab where
 experiments took place effect: Treatment, effect of either predation or
 resources on dispersal id.within.lab: Unique identifier for each
 experiment conducted at a single lab no_dispersers: Number of individuals
 who dispersed no_residents: Number of individuals who remained
 sp.lab.rsrc: Species name, lab name, and resource availability level
 together, separated by “.” PRED: Yes or no to presence of predators  RA:
 Resource availability level Collins et al. 2021 ITEX data variables:
 site_name: Name of site where experiments took place subsite: Subsites
 where experiments took place year: Year experiments took place phen_stage:
 Phenological stages  simple.phenophase: Simpler categories of phenological
 stages, ex. “Senesce” definition: Definitions of phen_stage values
 treatment: CTL control or OTC open top chamber (warming treatment)
 early.or.late: Early or late phenological events Spp: Species doy: Day of
 year Milcu et al. 2018 variables: Lab: “L#” Unique for each lab that
 conducted experiments Treat..legume: B means only Brachypodium grass and
 no legume, BM means grass with legume Medicago present
 Biom.dm..g...shoot.: Aboveground biomass of plot in g
 Method.comm..type.of.setup: Where the experiment took place, either in a
 greenhouse or growth chamber Spehn et al. 2005, BIODEPTH variables: plot:
 Unique to each experimental plot location: Site of experimental plot year:
 1 through 3, only year 3 was used legumes: Yes/no for legume
 presence/absence species.richness: Number of species in plot  biomass:
 Aboveground plant biomass in g/m2 Borer et al. 2020, Nutnet variables:
 site: Site name where experiments conducted  block: Block number of site
 where experiments conducted plot: Plot number where given treatment
 imposed  year: Year data collected from plot year_trt: Number of years
 treatment has been imposed at plot lm_lg: Log10 aboveground plant biomass
 in g in plot for a given year trt: Control (no treatment), NPK (nutrient
 addition), Fence (herbivore exclusion), or NPK+Fence (both) imposed on
 plot Guasconi et al. 2023 variables: shoot_biomass_control: Mean
 aboveground plant biomass in control plots 
 stand_dev_shoot_biomass_control: Standard deviation of aboveground plant
 biomass in control plots n_replicates_shoot_samples: Sample size of
 control plots shoot_biomass_treatment: Mean aboveground plant biomass in
 drought treated plots  stand_dev_shoot_biomass_treatment: Standard
 deviation of aboveground plant biomass in drought treated plots
 n_replicates_shoot_samples: Sample size of drought treated plots study:
 Unique number for each study included in the meta-analysis datapoint:
 Unique number for each effect size  Jia et al. 2018 variables: X: Unique
 number for each effect size Article: Unique number for each study included
 in the meta-analysis Vegatation_response: Response variable used for
 effect size, we only use “standing biomass” Unfenced_mean: Mean response
 variable (biomass) for unfenced plots Unfenced_se: Standard error for
 unfenced plots Unfenced_n: Number of unfenced plots  Fenced_n: Number of
 fenced comparator plots Fenced_se: Standard error of biomass for fenced
 plots Fenced_mean: Mean response variable (biomass) for fenced plots
 Cardinale et al. 2006 variables: Study: Unique for each published paper
 Entry: Unique for each experiment N1: Number of observations in a study
 with 1 species N2: Number of observations in a study with 2 species Y1:
 Mean aboveground plant biomass in g/m2 for observations with 1 species Y2:
 Mean aboveground plant biomass in g/m2 for observations with 2 species
 SD1: Standard deviation of aboveground plant biomass in g/m2 for
 observations with 1 species SD2: Standard deviation of aboveground plant
 biomass in g/m2 for observations with 2 species LnRR2: Log response ratio
 effect size of the presence of 2 species vs 1 species on aboveground plant
 biomass SV2: Sampling variance of each effect of the presence of 2 species
 vs 1 species on aboveground plant biomass Stuble et al 2021 variables:
 LastYear: Yes/no for last year of experiment EarlyLate: Early or late
 phenophase CitationID: Unique for each published paper effect.id: Unique
 for each experiment yi: Hedge’s g effect size of warming on timing of
 phenological event  vi: Sampling variance of each effect of warming on
 timing of phenological event ## ## Sharing/Access information Data used in
 code was derived from the following sources: Milcu et al. 2018:
 [https://doi.pangaea.de/10.1594/PANGAEA.880980](https://doi.pangaea.de/10.1594/PANGAEA.880980) Borer et al. 2020: [https://doi.org/10.6073/pasta/a318fe0fb11eb43c1a2c8233b2e3494f](https://doi.org/10.6073/pasta/a318fe0fb11eb43c1a2c8233b2e3494f) Stuble et al. 2021: [https://onlinelibrary.wiley.com/doi/abs/10.1111/gcb.15685](https://onlinelibrary.wiley.com/doi/abs/10.1111/gcb.15685) Costello & Fox 2022: [https://doi.org/10.5061/dryad.zkh1893b7](https://doi.org/10.5061/dryad.zkh1893b7) Collins et al. 2021: [https://github.com/cour10eygrace/OTC_synthesis_analyses.git](https://github.com/cour10eygrace/OTC_synthesis_analyses.git) Cardinale et al. 2006: [https://www.pnas.org/doi/10.1073/pnas.0709069104#supplementary-materials](https://www.pnas.org/doi/10.1073/pnas.0709069104#supplementary-materials) Rousk et al. 2013: Table 3 of Rousk et al. 2013 Spehn et al. 2005: [https://figshare.com/articles/dataset/Supplement_1_Data_from_the_BIODEPTH_project_15_ecosystem-process_variables_measured_at_eight_different_European_grassland_field_sites_over_three_years_together_with_metadata_and_a_table_with_site_information_/3565170/1?file=5640240](https://figshare.com/articles/dataset/Supplement_1_Data_from_the_BIODEPTH_project_15_ecosystem-process_variables_measured_at_eight_different_European_grassland_field_sites_over_three_years_together_with_metadata_and_a_table_with_site_information_/3565170/1?file=5640240) Ohlert and Collins 2021: doi:10.6073/pasta/5c2eb1c145a3babf399a0587d0189c2e Jia et al. 2018: [https://doi.org/10.5061/dryad.s7g70j0](https://doi.org/10.5061/dryad.s7g70j0) Ashton et al. 2022: [https://doi.org/10.25573/serc.19469900](https://doi.org/10.25573/serc.19469900). Fronhofer et al. 2018: [https://doi.org/10.5281/zenodo.1344579](https://doi.org/10.5281/zenodo.1344579) Guasconi et al 2023: [https://doi.org/10.1016/j.scitotenv.2023.166209](https://doi.org/10.1016/j.scitotenv.2023.166209) Smith et al. 2024: [https://datadryad.org/stash/dataset/doi:10.5061/dryad.3j9kd51rb](https://datadryad.org/stash/dataset/doi:10.5061/dryad.3j9kd51rb) ## ## Code/Software We conducted all statistical analyses using R 3.6.3 running within R Studio 1.3.1093 (R Core Team 2020). We fit all models using functions from the metafor package, version 2.4-0 (Viechtbauer, 2010). The likelihood ratio tests for differences in heterogeneity between meta-analyses vs. coordinated distributed experiments were based on the following example from metafor package author Wolfgang Viechtbauer: [https://www.metafor-project.org/doku.php/tips:different_tau2_across_subgroups](https://www.metafor-project.org/doku.php/tips:different_tau2_across_subgroups). If data does not read into R from the provided URLs, users can download the the data at these URLs personally.