# Presentation: 633036

## Now You've Built It...Does It Work? 

- Susan Fariss, National Library of Medicine Web Management Team
- Cindy Love, National Library of Medicine Specialized Information Services

**Notes:**
Thank you for your introduction.

It’s a pleasure to be here today to give you an idea of how the National Library of Medicine (or NLM) evaluates its web sites. Susan and I work 3 miles apart in 2 very different divisions and have had a range of experiences with both evaluation methods and with different types of web resources and the variety of teams that manage them. I’ll give you a little background on what we’re up to with the ACSI and Susan will review the various other evaluation methods used at NLM. And then we’ll have time for questions.

## National Library of Medicine

- World’s largest medical library
- One of the Institutes of NIH
- Over 20 major Web resources

**Notes:**
The National Library of Medicine is the world’s largest medical library. We’re on the campus of NIH north of downtown Bethesda and you may have noticed as you drive by on Rockville Pike - we’re the building with the wings on top. We’re one of the Institutes of the National Institutes of Health, even though we don’t have ‘institute’ in our name and date back to the mid-1800s. Our mission is to collect, organize and disseminate medical information and the primary method we use for dissemination is the Internet. We have many major web resources ranging from PubMed or MEDLINE which is the premier index to the medical literature to MedlinePlus, our resource for patients and health consumers, and including databases on the history of medicine, human and animal genetics, toxicology, public health, HIV and AIDS, clinical trials and, of course, a home page that attempts to bring this all together.

## NLM Web sites using ACSI

| Web site | Latest score |
| --- | --- |
| MedlinePlus
   -Consumer health, drug info  | 85 |
| MedlinePlus en español  | 84 |
| Main web site | 77 |
| TOXNET		
   -Toxicology databases | 75 |
| AIDSinfo	
   -HIV/AIDS treatment | 77 |

**Notes:**
We’ve been using the ACSI on 5 of our web sites, listed here with their most recent scores. 
We selected the NLM home page, partly because it’s very difficult to design a page that tries to be all things to all people and we wanted to see how we’re doing.

We’re also measuring MedlinePlus in both the English and Spanish versions. MedlinePlus provides high-quality, carefully-selected health information for the public and has very broad appeal. It’s probably the most heavily-evaluated site NLM has ever produced. 

We decided to include two databases that have more ‘niche’ appeal and provide unique, critical information to their mostly specialist users. TOXNET provides detailed information on over 300,000 chemicals and drugs from the point of view of their possible harmful or toxic effects.  AIDSinfo creates and makes available the definitive federal guidelines for AIDS treatment in the US, as well as information on approved drugs and clinical trials for AIDS treatment. 

## How it all began

- FY 2004 - NIH evaluation set-aside funds awarded to survey 5 NLM sites using the ACSI
- FY 2005 - We’re going trans-NIH with 60 NIH Web sites to be surveyed

**Notes:**
We started last October, after receiving funding to survey 5 sites. NIH has an Office of Evaluation that administers funds that can be used for a wide range of evaluation projects, many of which have nothing to do with the Internet. We went through their application process and were awarded $100,000 to evaluate these web sites using the  ACSI.  This sparked a lot of interest at NIH and we’re now moving to incorporate the ACSI on 60 web sites representing virtually every NIH institute. We expect this will be a major undertaking and are looking forward to working with both the NIH institutes and ForeSee and the Federal Consulting Group to make this a success.

We get asked one question a lot...

## Why is the MedlinePlus score so high?

- NLM evaluates its Web sites a lot!
- Content meets very high selection standards
- Look, feel, navigation, and search are consistent
- Site is constantly updated and maintained
- AND
- We believe people are grateful for quality health info

**Notes:**
Why is the MedlinePlus score so high?

I think there are lots of reasons starting with the uniquely compelling interest people have in their own and their loved one’s health. When people can find the health information they need and trust that it is from a reliable source, they are grateful!

Also, there is very tight editorial control over the entire MedlinePlus web site and highly qualified medical librarians do most of the work. MedlinePlus represents what most of us would love to do with our websites if we only had the resources and budget. 

We license content from the best – we have licensed drug information from 2 sources, daily news, patient education tutorials that talk to you in either Spanish or English, and directories.

The site is under constant review and is in a continual improvement cycle.

AND MedlinePlus uses every evaluation method we can think of that has proven useful. Susan is going to give you a run through of the methods we’ve used.   Susan....

## Why NLM studies its Web sites

- To better understand our users and our position in the health information arena
- Why do users come to NLM Web sites?
- What do they do with the info they find?
- How satisfied and likely to return are they?
- How can we do better?

## Multi-dimensional approach

- Multi-pronged approach, using several evaluation methods
- All methods have relative strengths and limitations, still both art and science

## Types of evaluation

- Focus groups
- Heuristic review
- Usability lab testing
- Randomized online user surveys
- Online “external” user panel surveys
- Internet audience measurement
- Unsolicited user feedback
- Web log data analysis
- Search log analysis

## Focus groups

- Small group of users (typically 6 to 10) provides feedback about a Web site, with a moderator following a prepared script of queries about the site.
- Session lasts 45 minutes to an hour and can be in-person or via online chat or telephone.

***Typical cost. ***$10,000 in-person focus group; $5,000 online.

## When and why

- During site development or after release
- Assess the need for new or modified features and functions
- Evaluate the impact of Web site usage on user knowledge and behavior

## Heuristic review

- A Web usability expert reviews your Web site, compares it against generally accepted Web design and functionality principles and standards and suggests design improvements.
- May include site layout and structure, navigation tools, search function, and  fonts and colors.

***Typical cost. ***$5,000 to $10,000.

## When and why

- During development and to improve site
- Improve layout, navigation, and search functions (usability)
- Assure compliance with Web usability and accessibility best practices

## Usability lab testing

- Structured testing of Web site.
- Users perform a series of tasks using the Web site, and test facilitators monitor and record user behavior and cognitive processes  (think-aloud test).

***Typical cost. ***$10,000 to $25,000

## When and why

- During development and to improve site
-  Improve layout, navigation, and search functions (usability)
- Gauge the success and relative importance of new features and functions
- Assess the need for new or modified features and functions

## Randomized online user surveys

- A random selection of Web site users have the opportunity to respond to a pop-up survey of 5 to 20 questions when they visit the site.
- Collect sample of 2,000 to 3,000 in a year.

***Typical cost. ***$20,000 to $30,000

## When and why

- Improve site
- Gauge the success and relative importance of new features and functions
- Assess the need for new or modified features and functions
- Identify under-served or under-represented user groups
- Evaluate the impact of outreach and promotional activities
- Compare Web site usage and impact against agency mission and performance goals
- Evaluate the impact of Web site usage on user knowledge and behavior
- Diagnose and improve technical performance and download times

## Internet audience measurement

- Private companies collect usage data from large panels of Web users whose Web surfing is monitored constantly. Panel sizes range from  50,000 to 1.5 million and cover usage in US homes and, in some cases, office, school, and international sites. Companies extrapolate usage data from demographics and census data. Provides comparative data on usage of your Web site versus competitive Web sites in a defined market.
- Data collection and extrapolation methods vary by company. Differences in panel composition and methodologies mean that usage data from different companies are not strictly comparable. Should consider these results as estimates, not precise measures.

***Typical cost. ***$35,000 to $40,000 per year.

## When and why

- Ongoing
- Gauge the success and relative importance of new features and functions
- Assess the need for new or modified features and functions
- Identify under-served or under-represented user groups
- Evaluate the impact of outreach and promotional activities
- Compare Web site usage and impact against agency mission and performance goals
- Evaluate the impact of Web site usage on user knowledge and behavior
- Diagnose and improve technical performance and download times
- NLM has used comScore/MediaMetrix and Nielsen/NetRatings private Internet user panels

## NIH.gov in the health information arena

- Top 5 general health information web sites:
- WebMD - 3 Million unique visitors
- NIH.gov - 2.7 M (inclusive of NLM)
- AOL Health (powered by WebMD) - 1.8 M
- MSN Health w/ WebMD - 1.6M
- Yahoo! Health - 1.3 M
- Based on Nielsen/NetRatings Sept. 2003 US Home market, domain reporting

## Unsolicited user feedback

- User feedback via an e-mail box, a link on the homepage, or a help desk phone number.
- Categorize data and analyze.

***Typical cost. ***Staff time, which can be extensive.

## When and why

- Ongoing
- Fill needs of intended audience more effectively
- Gauge the success and relative importance of new features and functions
- Assess the need for new or modified features and functions
- Identify under-served or under-represented user groups
- Evaluate the impact of outreach and promotional activities
- Compare Web site usage and impact against agency mission and performance goals
- Evaluate the impact of Web site usage on user knowledge and behavior
- Diagnose and improve technical performance and download times

## Web log analysis

- Analyze Web usage and search log data gathered by Web log analysis software installed on the Web site server to collect usage data.

***Typical cost. ***$500 to $100,000 for software site license

## When and why

- Ongoing
- Gauge the success and relative importance of new features and functions
- Evaluate the impact of outreach and promotional activities
- Diagnose and improve technical performance and download times

## History of online user surveys at NLM

- First generation - posted surveys, non-random, no benchmarking
- Second generation - snap shot surveys at single point in time, randomized sample, 2,000 to 4,000 respondents, custom surveys, not able to benchmark
- Third generation – ACSI

**Notes:**
Back to the ACSI. We’ve used online pop-up surveys before and found that the results from 2 years ago correlate very nicely with this year’s results from the ACSI. Prior to the ACSI, the surveys were one-shot, snapshot surveys that covered the time necessary to collect at least 2000 responses without unduly annoying our repeat users. We consider the ACSI to be our “third generation” survey type because it takes us to the next level...

## Why ACSI?

- Ability to survey year round to track how changes to the site or other factors affect satisfaction scores
- Ability to benchmark using model questions

**Notes:**
...By providing the ability to  post the survey year-round and thus compare changes over time. We can see how changes to the site that we’ve introduced can change satisfaction scores. And we can also see if outside factors affect scores. 

Also with the ACSI, we can benchmark our scores to compare ourselves with federal or private sector web sites and give us an idea of where we stand.

## ACSI as a part of the big picture

- Use data to focus on areas needing improvement
- Choose appropriate evaluation methods to improve these areas, such as usability testing

**Notes:**
We use the ACSI as one part of the evaluation picture at NLM. It helps us identify areas where we need improvement as well as audiences we’re not quite reaching. As Larry has said, “It’s diagnostic, not prescriptive”.  Once we’ve identified trouble spots we can turn to other appropriate evaluation methods to pinpoint changes we need to make. And then after making the changes we can see users’ reactions as summed up in the ACSI survey results. We don’t have to make our best guess about what the problems are and how to change them to improve user’s experience with our web sites. We don’t have to guess what our users think, they tell us every day!

## Contact us:

- Susan Fariss
- 301-435-7102
- susan_fariss@nlm.nih.gov
- Cindy Love
- 301-496-5306
- cindy_love@nlm.nih.gov

- Reference:
- Wood, Siegel, LaCroix, Lyon, Benson, Cid, and Fariss. A Practical Approach to E-Government Web Evaluation. IT Pro: May/June 2003, p. 22-8.

## NLM Web Sites

## http://toxnet.nlm.nih.gov

## http://tox.nlm.nih.gov

## New look?

## http://aidsinfo.nlm.nih.gov

## New look?

## Previous look

## http://www.nlm.nih.gov/

## http://medlineplus.gov/

## http://medlineplus.gov/spanish/