# Presentation: 751383

## Measuring ROI on ACSI Surveys

- Tim Evans
- Social Security Administration
- Office of Electronic Services
- tim.evans@ssa.gov

## Introduction

- ROI on your surveys not directly attributable to your survey expenditures—you may be validating prior work
- Ways of using ACSI data for ROI purposes
- Identify user behavior that explicitly shows your site’s value
- Identify data that shows how you can improve your site to increase its value
- Identify and focus your (limited) resource priorities

## SSA and ACSI

- Began Q1/04
- Four current surveys
- Main site
- FAQ’s site
- Retirement Planner site
- Internet Social Security Benefit Application--ISBA (not published)
- Two more surveys planned
- Business Services On-line (employer wage reporting)
- Disability (largest single group of surveyed visitors say they came for disability info)

## SSA ROI for ACSI Surveys--Summary

- Main survey shows high value + return on 2003 site redesign
- FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ’s availability
- Retirement Planner survey shows high value of Planner + the opportunity to focus our resources
- ISBA Survey helps clarify resource priorities in limited-budget environment

## Main Site Survey, Q3/04http://www.socialsecurity.gov

- Overall					75
- Content					80
- Functionality				76
- Look & Feel				79
- Navigation				73
- Search					74
- Site Performance			83

## Compare to ForeSee Results Benchmarks

- Overall Satisfaction: +5 *vis a vis* Q2/04 Gov’t benchmark; +4 *vis a vis *overall
- Look & Feel: +4/+3
- Content: +2/+2
- Navigation: +5/+3

## 2003 Site Redesign

- Site-wide adoption of CSS Style Sheets for common look & feel (colors, fonts, etc.)
- Added common header with top nav-links
- Primarily implemented via common *DreamWeaver* template for easy propagation site wide
- Context: ~80 SSA webmasters with central, but informal governance

## SSA FAQ’s Survey, Q3/04http://ssa-custhelp.ssa.gov/cgi-bin/ssa.cfg/php/enduser/entry.php

- Overall Score			74
- Content					81
- Functionality				77
- Look & Feel				79
- Navigation				75
- Search					75
- Site Performance			84

## Custom Questions Results Reveal ROI--#1

- If there were no FAQ’s, how would you have contacted Social Security to ask your question? (N=1273)
- Send e-mail					16%
- Browse web site				   8
- Call 800 number				39
- Call local SSA office				22
- Visit local SSA office				10
- Write SSA					  1
- Would not have contacted SSA		  4

## Custom Questions Results Reveal ROI--#2

- After reading our answer(s), what do you plan to do next?  (N=487)
- Nothing; question answered			36%
- Nothing; question not answered 	  	  4
- Send SSA e-mail			  	  6
- Call SSA 800 number			14
- Call SSA local office			  	  9
- Visit local SSA office				22
- Try different question			  	  4
- Look elsewhere on site		  	  3

**Notes:**
Question re-worded, 8/25/04, to better focus it; this explains lower number of respondents. Prior wording had less precise answer choices, but trend was the generally clear

## Key Comparisons for 2 Questions

- Send SSA e-mail		16%		  6%	  -62%
- Call SSA 800 number	39		14	  -65
- Call SSA local office	22		  9	  -59
- Visit local SSA office	10		22	+120

**Notes:**
Refer to Pew Study to be discussed later

## ROI—Reduction Use of High-Cost Support Channels

- Forester Research Estimates Cost of Services
- Knowledge-Based Self Service: $1.17/incident
- Telephone support: $33/incident
- E-mail support: $10/incident
- SSA Results, shown by ACSI data
- 65% reduction in calls to 800 number
- 59% reduction in calls to local office
- 62% reduction in e-mails
- 120% increase in local office visits

**Notes:**
54 million 800 calls in FY 2003

## SSA Retirement Planner Survey, Q3/04http://www.socialsecurity.gov/r&m1.htm

- Overall Score			77
- Content					82
- Functionality				80
- Look & Feel				81
- Navigation				77
- Site Performance			85
- Tasks/Transactions		78

## SSA Retirement Planner Survey, Q3/04--Futures

- Likely to Return		85
- Recommend			83
- File on-line			61
- What’s wrong with this picture?
- Is it good news or bad?

## Retirement Planner—Custom Question

- How do you plan to file for retirement benefits? (7/12-8/9 data; N=1127)
- On line					39%
- Face to face				37
- Telephone			  	  4
- On line w/SSA help	  	  	  2
- More recent data shows ‘on line w/help’ running ~13%, ‘on line’ ~30%--totaling _more than 40%_ (not enough data for conclusions)

**Notes:**
On line w/help option added 8/4

## Pew Internet Studyhttp://www.pewinternet.org/PPF/r/128/report_display.asp

*How Americans Get in Touch with Government* (May, 2004)
- Telephone			40%
- Web				24
- In person			13
- E-mail				11
- Write a letter		10

## Pew Internet Study—Cont’d

- Why Americans contact Government
- Conduct Transactions		30%
- Get specific information		25
- Register opinions			19
- Help w/specific problems	11
- For complex, problem-solving contacts, 62% prefer phone/in person

**Notes:**
Refer back to FAQ increase in local office visits; complex problem-solving contacts

## What Does This Tell Us About ROI for Retirement Planner?

- With ~40% interested in filing on line, SSA is doing quite well (compare Pew 30% transactions figure)
- Futures Score shows opportunity to improve on-line application, as Boomers age and interest in on-line filing rises
- New ‘On-line w/Help’ Numbers strong indicator for future, bearing out Pew conclusions on complex contacts

## Internet Social Security Benefit Application (ISBA)

- Retirement/spouse applicants vastly more satisfied (>2:1) than disability applicants
- Far more (5:1) retirement/spouse applicants successfully complete on-line application
- Lowest scores in Functionality, Tasks/Transactions; cut across board, *especially among disability applicants*
- Disability applicants have far lower level of Internet experience (5:1 “beginner”), and may need additional help
- Expectations count: level of Internet experience inversely proportional to satisfaction

**Notes:**
Pew study:  only 40% of disabled use the Internet; 38% say their disability makes using the Internet harder or impossible; only 9% see Internet as preferred channel

## ISBA and ROI Lessons

- Retirement/spouse applicants >80% of ISBA customers; universe will grow as Boomers age and demand service (Pew data cited earlier)
- Disability applicants have lower Internet experience, need more support via high-cost channels
- The whole disability application process differs substantially from retirement application process
- With resources limited, best use lies in improving overall Functionality, Tasks/Transactions
- All applicants will benefit from general improvements in the on-line application

**Notes:**
Again, Pew survey on disabled Americans’ reluctance to use the Internet

## Wrap

- ROI on your surveys not directly attributable to your survey expenditures
- Ways of using ACSI data for ROI purposes
- Identify user behavior that explicitly shows value of your site
- Identify data that shows how you can improve your site to increase its value
- Identify and focus your resource priorities

## SSA Conclusions

- Main survey shows high value + return on 2003 site redesign
- FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ’s availability
- Retirement Planner survey shows high value of Planner + the opportunity to focus our resources
- ISBA Survey helps clarify resource priorities in limited-budget environment