# Presentation: 209825

## Measuring the Emissions Impacts of Ozone Action Programs

- Integrating Air Quality and Transportation Planning Workshop
*Using Outreach to Improve Air Quality*
- NARC Workshop – February 24, 2004
- Eric N. Schreffler, ESTC

## INTRODUCTION

- How do you measure the actual impact of public education programs?
- How do you get beyond awareness and recall?
- What about episodic versus seasonal programs?

## TOPICS

-   General Evaluation Issues
-   Ozone Action Quantification Method
-   Voluntary TDM Program Evaluation
-   General Guidance and Tools

## Evaluation Issues

-   Projections rely on intended outcome, not 	planning 	strategies
-   Methods used to project results differ from 	those used to measure impacts
-   Surveys focus on awareness, not behavior
-   Too many assumptions...not empirical

## Public Education Programs

-   Most common are ozone alert or seasonal 	clean air 	programs
-   Often under control of public information
-   Rely on mass media and outreach
-   Use marketing and advertising specialists
-   Little experience with impact quantification

## Public Education Programs

-   So, how do you make leap from marketing and education...
-   		to travel behavior and trip reduction...
-   			to emission reduction?

## Why Evaluate?

-   Quantify actual emission reductions for SIP credit
-   Satisfy measurement requirements in Early Action 	Compacts
-   Evaluate cost effectiveness of program
-   Measure travel behavior changes
-   Document impacts for CMAQ reporting
-   Help demonstrate conformity
-   Provide decision-makers with feedback on results

## Range of Evaluation Approaches

-     **Depends on program objectives and funder ****	****expectations:**
-   track awareness of program
-   measure recall of message and call to action
-   evaluate “precursors” to behavior change
-   assess stated preferences
-   estimate travel behavior change
-   convert travel behavior to emission reductions

## EPISODIC PROGRAM

- Spare the Air Programs in
- Sacramento and San Francisco

## ARB/EPA METHOD

- “A Method to Measure the Travel and Emissions Impacts of Ozone Action Public Education Programs”
-   developed in Sacramento
-   implemented in SF Bay Area

## Research Sponsors

-   California Air Resources Board
-   US Environmental Protection Agency
-   Federal Highway Administration
-   Sacramento Metro AQMD
-   Bay Area AQMD
-   San Joaquin Valley AQMD

## Research Objectives

-   Method to Quantify Trip and Emission Reductions
-   Method for Episodic Education Programs
-   Affordable for Use by Air Districts
-   Accurate and Rigorous
-   Develop Correction Factors to Adjust Survey Findings
-   Acceptable to EPA for SIP Credit

## Research Definitions

- STA			=	Spare the Air
- Reducer		=	Driver who purposely 						reduces trips because 						of STA
- Non-Reducer	=	Driver who did not respond 					to STA message

## Research Design

-   Track Behavior of Reducers and Control Group
-   Compare STA Behavior to Other Days
-   Compare Reducer Behavior to Non-reducers
-   Develop Estimate of Actual Trip Reduction
-   Compare Actual to Reported Trip Reduction
-   Produce Correction Factor for Over-reporting of 	Trip Reduction

## What Do You Need to Know?

-   Proportion of drivers who reduce travel
-   Self-reported number of trips reduced
-   Proportion of work and non-work trips
-   Knowledge of ozone message
-   Average trip lengths
-   Regional emission factors

## Key Comparisons

-   Travel behavior (trips) of respondents who“purposely reduced” trip on STA days versus non-STA days (Treatment)
- and
-   Travel behavior of all drivers on STA days versus non-STA days (Control)

## Sacramento Surveys

-   Developed “Reducer” and “Standard” surveys
-   Fielded surveys evening after STA alert
-   Followed-up on Non-STA day
-   Surveyed Summer of 1999 and 2000 in Sacramento
-   Called almost 4,000 people
-   Resulted in 134 Reducers and 177 Non-reducers 	with paired surveys

## Net Average Trip Reduction

-  Reducers made *0.4478* fewer trips on STA days as compared to Non-STA days
-  Control group made *0.6497* MORE trips on STA days
-  Net average trip reduction = 0.4478 –(-0.6497)
- 	= ***1.0975*** trips reduced by “reducers”

## Self-Reported Trip Reduction

-    Summary question about how many trips reducers eliminated equaled:
- 				 2.2 fewer trips

## Correction Factor

-   				   Net Measured Average Trip Reduction
- Correction Factor =   _________________________________
- 				    Average Self-Reported Trip Reduction
- Correction Factor =    _1.0975_	=	**0.50**
- 				     2.2
- So, people actually reduce 1/2 of a trip for every reported trip reduced

## Summary of ARB Method

- Ask if they purposely reduced driving
- Ask how many trips they reduced
- Ask what kind of trips they reduced
- Ask if they knew it was an ozone action day
- Apply correction factor to reported reduction
- Apply reduction to population of reducers
- Apply VMT and emission factors

## Recommended Method

- 	Step 1 - Modify Survey and Sample Size
- Add questions for % of reducers, reported trip reduction, and type of trip reduced
- Sample size of about 1,000 to get acceptable range of error for estimate
-  Step 2 – Field Survey right after STA Day

## Survey Topics

- 1.		Did you _purposely_ increase or decrease the 		amount of driving you did today?
- 2a.		How many trips did you decrease?
- 2b. 	How did you decrease each trip?
- 2c.		What kind of trip did you decrease?
- 2d. 	Why did you reduce the trip?
- 3.  		Are you aware of Spare the Air or AQ ads?

## Survey Tips

- Need random sample of drivers
- Utilize RDD telephone survey
- Avoid self-selection – e.g., alert recipients
- Sample size depends on “incidence” and 	trips reduced
- Don’t ask leading questions
- Ask about campaign awareness at end
- Be aware of survey biases

## Recommended Method

- 	Step 3 – Tabulate Results
- Proportion of reducers
- Reported average number of trips reduced
- Type of trip reduced (work vs. non-work)
-  Step 4 – Estimate Total Reducers
- Apply proportion of reducers to driving population (equal to RDD sample)

## Recommended Method

- 	Step 5 – Estimate Average Trip Reduction
- Derive self-reported trip reduction from survey
-  Step 6 – Apply Correction Factor
- Multiply correction factor (0.5) to average self-reported trip reduction from Step 5

## Recommended Method

- 	Step 7 – Estimate Total Trip Reduction
- Multiply adjusted trip reduction (Step 6) by total reducers (Step 4)
-  Step 8 – Determine Proportion of Work and Non-Work Trips Reduced
- Apply proportions of work and non-work trips reduced to adjusted total trip reduction

## Recommended Method

- 	Step 9 – Estimate VMT Reduction
- Multiply trips reduced by type (Step 8) by average trip length by type (work and non-work)
-  Step 10 – Estimate Emission Reduction
- Apply emission factors to total trip (by type) and VMT reduction

## Summary of Method 

-   Run measurement plan by USEPA
-   Add three revised question strings to 	regional follow-up survey
-   Adjust and readjust sample size
-   Apply correction factor to self-reported trip 	reduction
-   Calculate emission reductions

## Preliminary Impact Findings 

-   About 5% of drivers reduce trips for STA
-   They report reducing 2.2 trips
-   They actually reduce 1.1 trips
-   They eliminate or postpone trips
-   They reduce non-work trips
-   Reduced 0.35 tons/day NOx; 0.37 tons/day ROG 	and 0.06 tons/day PM10

## Multivariate Analysis Findings

-   Those working at employer sites that give air 	quality alerts about 1.6 x more likely to be 	reducers
-   Women about 1.6 x more likely to be reducers 	than men
-   Those with larger families (3 or more children) 	about 1⁄4 as likely to be reducers

## Suggested Use of Results

-   Emission factors are getting smaller
-   Better to also track proportion of reducers and 	average 	trips reduced
-   Report total number of drivers affected
-   Monitor changes in these indicators over time
-   Modify program accordingly to influence these
-   Calculate cost effectiveness ($/driver, $/trip, $/mile 	or $/pound of pollution (≤ $10/lb.)

## VOLUNTARY TDM PROGRAM

- Framework for Cooperation to Reduce Traffic Congestion and Improve Air Quality
- Atlanta, Georgia

## ATLANTA TDM Evaluation 

-   Georgia seeking SIP credit for all voluntary TDM	trip reduction programs in Atlanta (VMEP)
-   1.5% of emission target or 4.28 tpd of NOx 	and 6.51 tpd of VOC in 2003
-   Equates to 4.4 million miles of travel
-   90% from CAC participating employees (132,645)
-   10% from collateral influence of campaign (14,739)

## ATLANTA TDM Framework

-   Clean Air Campaign/Private – employer outreach
-   Clean Air Campaign/Public - state worksites
-   Media Campaign – public education
-   Support from Framework Partners
- regional rideshare program
- network of TMAs – transit pass discounts
- vanpool program
- cash for commuters incentive

## TDM Evaluation

-   State funds large-scale evaluation for VMEP SIP
-   Performed by Center for Transportation and the 	Environment
-   Assisted by advisory group and consultants
-   Annual regional survey
-   Annual evaluation of partner programs
-   2003 test of emission impacts toward attainment; 	2004 refinement being implemented

## Evaluation Plan

-   Annual “bottom-up” evaluation of partner programs
-   count placements into new alternative modes
-   avoid double counting
-   sum by mode, not partner	= 0.73 ton of NOx; 0.84 VOC
-   Annual “top-down” evaluation of regional impacts
-   are commuters and others shifting modes?
-   developed “switcher” survey to find mode switchers

## Evaluation Issues

-   SIP forecast backed into target impact
-   Assumes every “participant” reduced a trip 	every day
-   Did not account for occupancy or part-time 	use
-   Did not account for prior mode or access mode

## Refinements for 2004

-   Annual survey of regional travelers will be repeated
-   Will account for all mode switching (net impact)
-   Will try to directly link switch to influence of media 	campaign and/or TDM programs
-   Georgia DNR-EPD will decide what to report to EPA
- 		**www.tdmframework.org**

## General Guidance

-   Think through forecasting and measurement 	issues when designing program
-   Maintain consistency in approach/methods
-   Develop measurement plan – get help first time
-   Don’t just do it for SIP or Early Action Compact
-   Minimize assumptions and borrowed factors
-   Minimize self-reporting bias

## Guidance Document

-  Developed Guidance 	Manual for Air Districts
-  Reasons for Evaluating 	Program
-  Steps for Using Method
-  Tips on Planning Evaluation
-  Case Study from Bay Area
-  Sample Survey from Bay 	Area

## Additional ARB Tools

- Quantification Method for Ozone Action Programs
- _[www.arb.ca.gov/research/abstracts/98-318.htm](http://www.arb.ca.gov/research/abstracts/98-318.htm)_
- Automated Methods to Find the Cost Effectiveness of Funding Air Quality Projects
- Determining the Cost Effectiveness of Employer TDM Programs
- _[www.](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[arb](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[.ca.](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[gov](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[/planning/](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[tsaq](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[/](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[eval](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[/](http://www.arb.ca.gov/planning/tsaq/eval/htm)__[htm](http://www.arb.ca.gov/planning/tsaq/eval/htm)_