Impact Evaluation Report 

EarthCorps Restoration Methods 

September 30, 2019 

Revised October 30, 2019 

Prepared for: 

Bill Brosseau 

Operations Director 

EarthCorps 

6310 NE 74th Street, Suite 201E 

Seattle, WA 98115 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Watershed Company 

RevisedOctober 2019 

T A B L E O F C O N T E N T S 

Page # 

1 Introduction......................................................................................... 1 

2 Purpose ............................................................................................... 1 

2.1 Theory of Change ................................................................................................... 1 

2.2 Outcome of Interest............................................................................................ 2 

2.3 Research Question............................................................................................. 3 

3 Methods ............................................................................................... 4 

3.1 Study Participants .............................................................................................. 4 

3.2 Plan Design ......................................................................................................... 4 

3.3 Treatment Sites................................................................................................... 5 

3.3.1 Site Selection...........................................................................................................5 

3.3.2 Plot Selection...........................................................................................................6 

3.4 Data collection .................................................................................................... 6 

3.4.1 Before (pre-restoration site conditions)........................................................... 7 

3.4.2 After (post-restoration site conditions)............................................................ 7 

3.4.3 Control Sites....................................................................................................... 7 

3.5 Monitoring Stations ............................................................................................ 8 

3.5.1 Sample Plots ...................................................................................................... 8 

3.5.2 Photo Points ............................................................................................................9 

4 Statistical Approach ........................................................................... 9 

5 Results............................................................................................... 10 

6 Discussion......................................................................................... 12 

Appendix A: Photos 

L I S T O F F IG U R E S 

Figure 1. Map of treatment site locations, in incorporated King and Pierce Counties............6 

Figure 2. Diagram of sample plot layout ...................................................................................8 

Figure 3. Comparison of invasive species cover before and after treatment in both control 

and treatment plots..................................................................................................10 

i 
 

L I S T O F TA B L E S 

Table 1. Treatment sites and associated characteristics ......................................................................5 

Table 2. Treatment sites and sample plots...........................................................................................9 

Table 3. Standard deviations and average cover of native and invasive species in control and 

treatment plots before and after restoration..........................................................11 

Table 4. Statistical analysis data on multiple comparisons of data. P Value used as final dataset is 

underlined .................................................................................................................11 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I M P A C T E V A L U A T I O N R E P O R T 

EA RTHCORPS RESTORAT ION METHOD S 

1INTRODUCTION 

EarthCorps is a Seattle-based non-profit that trains young adults in 

environmental restoration techniques along shorelines, forests, and trails in the 

Puget Sound area. Over the course of a year-long training program, volunteers 

learn practical skills that can be used to combat global warming, pollution, and 

environmental degradation. 

In order to comply with Corporation for National and Community Service 

(CNCS) grant requirements, EarthCorps has retained The Watershed Company 

(Watershed) to provide an independent evaluation of restoration outcomes. The 

purpose of this evaluation plan is to design an independent and statistically valid 

assessment of EarthCorps restoration projects. This impact evaluation report 

provides a description of the sampling approach and analysis. 

The purpose of this impact evaluation report is to assess the effectiveness of 

EarthCorps restoration methods. As the independent reviewer, staff from The 

Watershed Company, tabulated and analyzed the data EarthCorps crews 

collected. 

2PURPOSE 

2.1 Theory of Change 

EarthCorps’ Theory of Change engages AmeriCorps members to scale up green 

infrastructure in order to better manage stormwater, which will decrease 

pollution into Puget Sound watersheds. Deploying AmeriCorps members to 

increase the scale of "green" stormwater management is an innovative model, 

providing person power to a critically under-staffed profession. Across the US, 

it is estimated that 85% of water pollution comes from stormwater runoff. The 

Pacific Northwest is a national leader in applying green infrastructure to 

stormwater management, perfecting techniques that are needed here and 

elsewhere. Recent studies on the effectiveness of watershed restoration activities 

(e.g. restoring forests) and engineered green stormwater infrastructure 

1 
 

 

 

 

(e.g. rain gardens) indicate that the selected green infrastructure intervention can 

be expected to produce the proposed results of reducing polluted runoff. 

The urban tree canopy intercepts and slows precipitation 

(https://www.epa.gov/green-infrastructure/what-greeninfrastructure#raingardens). According to City of Seattle’s 2013 Urban Forestry 

Stewardship Plan, forests in the city remove 725 metric tons of pollution from the 

environment each year 

(http://www.seattle.gov/trees/docs/2013%20Urban%20Fores%20Stewardship%2 

0Plan%20091113.pdf). 

EarthCorps members spend approximately 70% of their time in watershed 

restoration activities, 10% of their time in installing and maintaining engineered 

green stormwater infrastructure; and 20% of their time in education, training, 

and reflection activities. Because most of the member service will be designated 

to watershed restoration activities, this evaluation will focus on the outcome of 

these activities. The intent is to monitor and track outcomes on the same set of 

watershed restoration project sites over the timeframe of approximately 10 

years, a timeframe that coincides with the long-term outcomes stated in our 

logic model. The current evaluation represents the first stage in our longitudinal 

study of these project sites. It focuses on the first phase in the restoration 

sequence: initial invasive plant removal. Future evaluations will focus on 

successive restoration phases: maintenance, native plant installation, and 

adaptive management. 

Outside of the framework of this current evaluation, EarthCorps plans to monitor 

the outcomes of the other member activities. They may focus on those activities in 

a future evaluation. 

2.2 Outcome of Interest 

The outcome of interest for this evaluation plan is the medium-term outcome 

listed in the attached logic model: "Natural areas are healthier." This outcome 

aligns with the theory of change because members will engage in watershed 

restoration activities, which will make natural areas healthier. Healthier natural 

areas will retain more stormwater, reducing runoff. The selected indicator for 

determining whether natural areas are healthier as a result of member activities 

is a decrease in invasive plant cover compared to baseline after 1 year. Invasive 

plant cover is a commonly used ecological measurement of the percentage of a 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

given area of land that is covered by invasive plant foliage. Invasive plants pose 

a threat to natural areas by out-competing native plants, damaging trees, and 

suppressing native plant and tree regeneration. Trees, especially conifers, are 

the most effective plants for stormwater capture, retention, evapotranspiration 

and filtration in the Pacific Northwest. To protect existing and future trees, it is 

vital to significantly reduce or eliminate invasive plants. Removing invasive 

plants and keeping them from taking over again is a medium-term outcome 

that can take three or more years. It is an essential step to making natural areas 

healthier, more sustainable, and most importantly, increasing their stormwater 

retention capacity. 

RESTORATION TREATMENTS 

Watershed restoration activities can include a wide range of intervention actions. 

Because the outcome of interest for this evaluation is decreased invasive plant 

cover, we observed the impact of AmeriCorps members’ actions aimed at 

eradicating or suppressing invasive plants by using power tools, hand tools, and 

herbicide. Treatment sites with Himalayan blackberry were treated with brush 

cutting (using a power tool to cut stalks) and foliar spray (spraying herbicide on 

the remaining foliage). Treatment sites with English ivy were treated through 

manual control and removal (grubbing out the stems and roots by hand or with 

hand tools). Initial treatment occurred in spring/summer 2018 and follow-up 

maintenance treatment occurred in spring/summer 2019. No invasive plant 

removal or herbicide application occurred on control sites. 

2.3 Research Question 

This evaluation seeks to answer the question, “Does the restoration action done 

by EarthCorps achieve the outcome of making natural areas healthier?” The 

evaluation of this outcome will be based on the indicator of decreasing invasive 

plant coverage. If AmeriCorps restoration actions result in a significant sustained 

reduction in invasive plant coverage, then restoration actions are deemed 

effective in making natural areas healthier. 

For each evaluated site, the evaluator will assess: 

1) What was the percentage of invasive cover on the site before the initial 

intervention? (baseline data from 2018) 

2) What is the percentage of invasive cover after the intervention? 

(intermediate-term indicator, after one growing season) 

The evaluation described in this report assesses invasive plant coverage before 

and after restoration actions. This evaluation represents monitoring performed 

one year after the initial invasive removal/treatment intervention in 2018. This 

3 
 

 

enables EarthCorps to assess the medium-term outcome of making natural areas 

healthier. EarthCorps intends to continue to monitor the site for up to 10 years 

after the initial intervention. 

In addition to invasive plant cover, data collection on native woody plant cover 

and conifer density on the sites will measure short-term gains toward the goal of 

restoring a native forest community. This data will enable the evaluation of 

relevant longer-term outcome indicators such as increase in native woody plant 

cover or increase in conifer density in future impact evaluations. 

3METHODS 

3.1 StudyParticipants 

This study was led by EarthCorps. Staff from EarthCorps and The Watershed 

Company collaborated on sample methodologies, study questions of interest, 

and study design. EarthCorps supervisors and crew members implemented the 

restoration treatments and collected and compiled field data. EarthCorps 

funded the restoration project through a cost-share partnership (75/25). 

EarthCorps funded the evaluation. Staff from The Watershed Company 

developed the Evaluation Plan, assisted with plot set up, analyzed data 

provided by EarthCorps, and summarized results in this report. 

3.2 Plan Design 

The sampling framework incorporates a Before, After, Control, Impact (BACI) 

design. BACI requires an examination of pre-restoration site conditions and 

post- restoration site conditions, as well as a comparison of the post-restoration 

site to a reference site. The before-after data shows how the sites changed over 

time. 

The control-impact data allows restoration actions to be differentiated from 

natural variability and stochastic events. 

Using permanent plots to track changes to the landscape over time will provide 

a repeatable measure of the resource condition at these locations. Using 

invasive species cover as an outcome indicator allows comparison across 

different habitat types and the varied restoration objectives of our agency 

partners. As shown on the program logic model, reducing invasive species is a 

key medium-term outcome of members' watershed restoration activities. 

In order to minimize the variability between treatment sites and control sites, 

control sites should be situated near the corresponding restoration site and 

they should experience similar environmental conditions but be independent 

of activities affecting the restoration site. Sampling techniques should be 

4 
replicable among all sites. Sampling periods should be identical between 

reference and restoration sites, to the extent possible. Since some time delay 

between field visits to various sites is inevitable, sampling periods are 

scheduled by season. 

3.3 Treatment Sites 

3.3.1 Site Selection 

EarthCorps identified seven sites for restoration treatment, considering factors 

such as land ownership, access, site characterization, restoration treatments, 

seasonality, and the evaluation timeline. The ability for EarthCorps to secure ongoing permission to access a site was an important consideration, to ensure that 

monitoring data can be collected on each site throughout the evaluation time 

period, and potentially in the future. 

A second important consideration for the initial site selection was to limit the 

variety of treatments and invasive plant species being evaluated, in order to be 

able to observe identical treatments across multiple sites. EarthCorps chose to 

focus on two of the most prevalent invasive plant species in our region: 

Himalayan blackberry (Rubus armeniacus) and English ivy (Hedera helix). 

EarthCorps identified the project sites that fulfilled the requirements of on-going 

access, having the invasive plant species, and with restoration actions scheduled 

in spring/summer 2018. Seven project sites met this requirement. There was no 

random selection of sites because these seven project sites represent 100% of the 

sites scheduled to receive invasive species treatment within the timeframe of the 

evaluation. Evaluating seven sites is within the capacity of EarthCorps and the 

independent evaluator, so it was not necessary to select fewer than seven sites. 

Sites are in the greater Seattle, Washington area (Table 1). Treatment sites are 

characterized by a prevalence of non-native, invasive vegetation, including 

Himalayan blackberry (Rubus armeniacus) and/or English ivy (Hedera helix). Sites 

with Himalayan blackberry were treated with brush cut and foliar spray. Sites 

with English ivy were treated through manual control and removal. Treatment 

sites range in size, with a minimum size of ~4,000 square feet. 

Four of the seven sites that EarthCorps has identified have additional areas 

where no restoration actions are currently planned to occur. These areas are 

proposed as control sites. 

5 
 

Table 1. Treatment sites and associated characteristics. 

Site Name / ID Primary Weed Control Site? 

Discovery Park (06-01), Seattle Himalayan blackberry No 

Discovery Park (17-13), Seattle Himalayan blackberry No 

Discovery Park (19-01), Seattle Himalayan blackberry No 

Hylebos Natural Area, Fife Himalayan blackberry Yes 

Lake Hills Greenbelt, Bellevue English ivy Yes 

Lewis Creek, Bellevue Himalayan blackberry Yes 

Nature Trails Park, Normandy Park English ivy Yes 

Figure 1. Map of treatment site locations, in incorporated King and Pierce Counties. 

3.3.2 Plot Selection 

Plot selection considered safety and accessibility for the people 

performing data collection. For example, a plot was not sited on a 

dangerously steep ravine. 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Stratified randomization was used to select plot center points on a map 

of each site in order to randomize the placement of the sample plots at 

each site to minimize sampling bias. Sample plot are described further in 

section 3.5.1 below. 

3.4 Data collection 

Each “Treatment” site was monitored three times: 1) immediately prior 

to restoration action in spring 2018; 2) immediately following restoration 

action in the spring/summer 2018; and 3) late spring/early summer 2019, 

one growing season after the restoration actions on the treatment sites. 

Each “Control” site was also monitored three times, on the same schedule as the 

treatment sites. Data collection in spring/summer 2018 was not required, but the 

field crews included controls in their data collection. 

3.4.1 Before (pre-restoration site conditions) 

Pre-restoration site conditions were recorded at each site prior to invasive species 

removal. Data collection included: 

 a description of the dominant invasive plants at each site (treatment 

species and non-treatment species) 

 visual estimates of invasive cover (treatment / non-treatment species 

estimates) 

 native woody cover, differentiating between evergreen and deciduous 

cover, and (non-native, non-invasive woody cover added) 

 conifer density. 

3.4.2 After (post-restoration site conditions) 

Post-restoration conditions were recorded following the restoration action 

(spring/summer 2018) and one full growing season later (late spring/early 

summer 2019), as described below. 

Spring/Summer 2018 

To capture initial treatment site conditions and document potential variability 

among sites, initial treatment site data collection documented restoration 

methods. Variables such as invasive plant removal/treatment methods, soil 

amendments, or any native planting, were recorded. 

Data collection included the same metrics identified above. 

Late Spring/Early Summer 2019 

In late spring/early summer 2019, one growing season after the initial restoration 

activities, each site was re-evaluated, using the same metrics identified above. 

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3.4.3 Control Sites 

Data collection at “Control” sites will follow the same protocol as the 

“Treatment” sites, except that no “After” monitoring will be conducted in 2018. 

All five control plots, spanning four sites, were visited in spring/summer 2018. 

One control plot at the Lewis Creek site was not revisited in spring/summer 2019. 

The control sites should not be directly impacted by restoration actions. 

However, some control plots experienced partial clearing due to work conducted 

by others not aware of EarthCorps’ work. 

3.5 MonitoringStations 

3.5.1 Sample Plots 

Prior to any restoration action, EarthCorps crews established monitoring plots 

for representative sub-sampling at each site (including both “Treatment” and 

“Control” sites). Monitoring plots were 33 feet by 33 feet. The center of each plot 

was marked in the field using a metal post wrapped with field flagging tape and 

metal identification tags and the corners will be marked with pin flag or field 

tape for visual reference. Plots were demarcated during monitoring assessment 

using two transect tapes arranged perpendicular to each other (Figure 1). Plot 

locations were marked on aerial photos or site maps. 

The number of sample plots varied by site size. The number of plots for each site 

is shown in Table 2. 

   Figure 2. Diagram of sample plot layo

8 
 

Table 2. Treatment sites and sample plots. 

Site Name / ID 

Primary 

Weed 

Number of 

Treatment Plots 

Number of 

Control Plots 

Site size 

(acres) 

Discovery Park (06-01), Seattle Himalayan 

blackberry 3 0 0.77 

Discovery Park (17-13), Seattle Himalayan 

blackberry 1 0 0.19 

Discovery Park (19-01), Seattle Himalayan 

blackberry 5 0 1.19 

Hylebos Natural Area, Fife Himalayan 

blackberry 3 1 0.34 

Lake Hills Greenbelt, Bellevue English ivy 0 1 0.11 

Lewis Creek, Bellevue Himalayan 

blackberry 6 1 1.38 

Nature Trails Park, Normandy 

Park English ivy 7 2 2.86 

Invasive species composition, cover, native woody cover (evergreen and 

deciduous), and conifer density was visually estimated within each sample plot. 

In all cases, cover was assessed based on the leafed-out condition. 

Areas not directly covered by the monitoring plots will be visually assessed for 

general conditions and any anomalous areas not represented by the monitoring 

plots will be noted. 

3.5.2 Photo Points 

Photo points include before-after photo points at each plot. Photos were taken 

from each plot corner towards the center of the plot. Four photos were taken at 

each plot during each of the three visits. 

“Before” and “After” pictures were taken from the same locations at each 

treatment site. “After” pictures were taken in spring/summer 2018 and again in 

spring/summer 2019. 

Control site photos were taken from the same locations at each reference site in 

spring/summer 2018 and again in spring/summer 2019. 

4STATISTICAL APPROACH 

The study uses a quasi-experimental design with randomized control to ascertain 

correlations between site treatments and plant cover. The study consists of 

9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

randomized plots, including untreated control (reference) plots as detailed in 

Section 3.5 above. 

Data collected on percent cover of native and invasive plants were analyzed 

using one and two-tailed t-test. The approach tested the null hypothesis that 

there is no difference between invasive plant cover before and after restoration 

actions. 

Statistical significance was based on an alpha of 0.05. With seven treatment sites, 

the power to detect a 30 percent difference in invasive cover, given a standard 

deviation of 20 percent is 0.85. The standard deviation of control sites was below 

7 percent, allowing for the use of a smaller number of samples for the control 

sites. 

5RESULTS 

Plots treated with restoration exhibited significantly decreased invasive species 

cover after treatment (Figure 3, p=2.4e-22). There was not a significant change in 

invasive species cover in control plots (p=0.4) 

Treatment 

Figure 3. Comparison of invasive species cover before and after treatment in both control 

and treatment plots. 

10 
 

Table 3. Relative Standard deviations and average cover of invasive species in control and 

treatment plots before and after restoration. 

Control 

Before 

Control 

After 

Treatment 

Before Treatment After 

Standard 

Deviation (%) 

Average Cover 

Invasive (%) 

2.2 

98.8 

5.1 

96.2 

11.2 

93.2 

24.8 

16.3 

Table 4. Statistical analysis data on multiple comparisons of data. P Value used as final dataset is 

underlined. 

Invasive Plant Cover Analysis T-Stat P Value 

One Tail 

P Value Two 

Tail 

Degrees 

of 

Freedom 

Significant 

result? 

Control Before Compared to 

Control After 

0.871437 0.20106755 0.402135099 11 No 

Treatment Before Compared to 

Treatment After 14.21542 2.39131E-22 4.78262E-22 68 Yes 

Treatment After Compared to 

Control After 

9.444397 2.9865E-13 5.97299E-13 53 Yes 

Average percent cover of invasive species in the control plots was 98.8 percent 

at the beginning of the study and 96 percent at the end of the study, with 

standard deviations of 2 and 5 percent respectively. After restoration, average 

percent cover of invasive species dropped from 93 percent to 16 percent, with a 

standard deviation of 24.8 percent (Table 3). Reduction in invasive cover at the 

treatment sites over the study period was statistically significant. Minor 

variation in invasive cover in the control plots over the study period was not 

statistically significant (Table 4). 

Additional data, including native cover and conifer density, were collected and 

may be utilized in further studies. Average percent cover of native woody plants 

remained substantially constant during the monitoring period, with average 

cover at 68 percent before restoration and 67 percent cover after. Conifer density 

at study plots ranged from zero to 19 conifers per 1,089 square foot plot. On 

average there were three conifers per sample plot. 

11 
6DISCUSSION 

Based on the results of this experiment, the restoration techniques used by 

EarthCorps crew members are an effective way to decrease invasive species at 

both sites dominated by Himalayan blackberry and English ivy in one growing 

season. While this result is positive, it leaves many longer-term questions 

unanswered. 

Subsequent studies may monitor invasive plant cover at these study plots over 

additional growing seasons. If more plots dominated by English ivy are added 

to the next study, it may be possible to compare the impact of restoration work 

on English ivy relative to Himalayan blackberry. The level of effort required to 

displace invasive plants is expected to vary by species. 

Another potential study question for further monitoring is: How many growing 

seasons a site must be maintained before a significant increase in native plant 

cover is realized? To assess this question, restoration efforts, including planting 

densities, would need to be standardized across treatment and control plots. 

In addition to decreasing invasive cover, increasing conifer density and cover is a 

long-term goal of EarthCorps’ work. The baseline data collected during this 

study may be utilized to track increases in conifer density and cover over future 

growing seasons. Since baseline data collected during this study includes 

deciduous and conifer cover in each sample plot, later studies may compare 

conifer success under differing canopy conditions. 

While the short-term impact of EarthCorps’ work to remove invasive plants is 

evident both in photographic documentation and statistical data analysis, 

EarthCorps seeks to learn more through additional longer-term studies. The 

outcome of future studies may improve adaptive management strategies 

implemented by EarthCorps members. 

12 
Appendix A 

ReferencePhotographs 

 

 

A-1 
 

 

A-2 
 

2018 March 15 – Nature Trails Park A Plot 5 SW 

A-3 
 

 

A-4