**Analysis of NIH Stem Cell Web Site Usability Testing**

Eagle Design & Management, Inc., ran usability tests of the proposed NIH
Stem Cell Information Web site on May 30--June 1, 2006. Our three target
audiences were the general public, college students, and health
researchers; for each group, we interviewed five representatives.

**Summary Impressions**

Researchers were much more proficient at finding information on the
site, probably because they had more specific needs. They quickly
realized that the Research Topics section was intended for their use and
zeroed in on particular details. They appreciated having the left-hand
navigation bar, and their familiarity with the subject helped them find
the correct page easily.

> ***[Recommendation:]{.underline}*** Maintain the left-hand list of
> researcher-specific links.

The general public and students quickly found information if it was
presented directly to them (e.g., as a Fast Answer link) but were unsure
where to find other topics. They would look over every topic area,
clicking promising links that often did not have the information they
sought.

According to the American Customer Satisfaction Index (ACSI) results,
users seeking policy information make up less than 1% of site visitors.
Although policy information is crucial to the site, it probably does not
need to be featured as prominently as general information and research
topics.

> ***[Recommendations:]{.underline}*** Expand the prominence of Fast
> Answers and tie in with FAQs. Rename misleading navigation links
> (e.g., Policy & Guidelines, which does not spell out U.S. Policy in a
> simple statement) and/or eliminate from navigation links. Deemphasize
> organization around four topic sections (Info Center, Research Topics,
> Federal Policy, News & Announcements) to focus on the two different
> navigation styles (general public vs. researchers).

**Satisfaction Measures**

The proposed redesign scored higher than the live site on ACSI
satisfaction measures \[rated from 1=poor to 10=excellent\]. No doubt
anonymous survey takers find it easier to assign lower ratings than live
respondents, so scores for the redesign are likely to be artificially
high. Still, their relative rank provides valuable information about the
new user experience:

  ---------- --------------- ----------------- ---------------- ---------- -------------- --------------- --------------
             **Quality of    **Convenience**   **Clarity of     **Design   **Ease of      **Ability to    **Overall
             Information**                     Organization**   and        Navigation**   Find            Satisfaction
                                                                Layout**                  Information**   with Use**

  Live Site  7.4             7.3               7.6              7.3        6.9            7.5             7.2

  Redesign   8.5             8.4               8.6              8.0        9.1            8.6             8.9
  ---------- --------------- ----------------- ---------------- ---------- -------------- --------------- --------------

Since the redesign was intended primarily to improve ease of navigation
(the lowest of these measures on the live site), it is very encouraging
to see that Ease of Navigation and Overall Satisfaction with Use showed
the greatest gains. Design and Layout did not improve much, probably due
to the focus on functionality and the fact that the redesign was not
intended to be a final release. One user specifically mentioned that the
home page had too much white space on the right-hand side.

Convenience was the other low scorer, for both the live site and the
redesign. This suggests that although the redesign makes significant
improvements to navigation, users are still having occasional
difficulties finding information.

> ***[Recommendations:]{.underline}*** Keep elements of the redesign
> that helped users find information quickly, but rework the aesthetics.
> Streamline navigation options to steer users quickly to the correct
> topic area and improve the site's convenience. Move to centered,
> fixed-width layout to avoid excessive white space.

**Responses to Overview Questions**

Responses were as expected, although many users did not realize the
"Locate Eligible Cell Lines," "Student Resource Guide," and "New Center
Grants" images were hyperlinked. Likewise, most users felt the section
headers in the navigation were not linked.

> ***[Recommendation:]{.underline}*** Avoid using images as links unless
> clearly marked.

**Responses to Targeted Questions**

[General Public]{.underline}

1.  *Can you find something to help a regular person understand what
    stem cells are?*

> Although this was one of the "Fast Answers" highlighted on the center
> of the home page, 3 of 5 respondents went immediately to Stem Cell
> Basics on the left-hand navigation.
>
> **Recommendation:** As above, maintain left-hand navigation.

2.  *Why has stem cell research generated so much debate?*

> 3 respondents tried News & Announcements first. Most finally found the
> Stem Cells and Research Ethics page, but one respondent felt it did
> not answer the question.
>
> **Recommendation:** Rename News & Announcements to just
> "Announcements" to avoid confusion. Rewrite Research Ethics page to
> clarify NIH position?

3.  *Is there any information on the site about what disorders stem cell
    research might be able to help?*

> Again, only 2 respondents used the Fast Answers section. Others found
> Stem Cells & Diseases in the left-hand navigation. One respondent felt
> the page did not address many specific diseases.
>
> **Recommendation:** Expand content of Stem Cells & Diseases page? The
> Fast Answers help get users to the correct page quickly, but move them
> to the navigation.

4.  *Given the nature of stem cell research, some material is very
    technical. Are there any resources to help you understand it?*

> Respondents correctly turned to the Glossary and FAQs.

5.  *Can you use the site to find out the current US policy on stem cell
    research?*

> Respondents understandably clicked on "Policy & Guidelines," but this
> page does not clearly spell out U.S. policy; instead, it provides a
> list of links to complex policy documents.
>
> **Recommendation:** Add eligibility criteria to the Policy &
> Guidelines page. In general, the root-level index pages (e.g., the
> main Info Center page) contains only links to interior pages and is a
> waste of users' time. Add content to help orient users and explain
> what information they will find in that section.

6.  *What stem cell research in being conducted in the United States?*

> Respondents clicked all over the Research Topics links in the
> navigation looking for a simple answer to this seemingly simple
> question.
>
> **Recommendation:** Create a "Current Research" page that will
> summarize efforts and direct users to the correct interior page
> (Research at NIH, Universities and Institutions, International
> Research, etc.).

7.  *What stem cell research in being conducted in other countries?*

> Respondents correctly found International Research in the navigation.

8.  *Can the public get involved in clinical trials involving stem
    cells?*

> Respondents used the Fast Answers link to Stem Cells & Diseases or had
> seen the question in the FAQs; however, they found both pages
> confusing. On Stem Cells & Diseases, the link to clinicaltrials.gov
> was at the bottom of the page and not immediately apparent. The FAQs
> page was hard to read because every question is a link.
>
> **Recommendation:** Link specifically to the clinical trials paragraph
> on Stem Cells & Diseases. Unlink questions on FAQs; add
> ([Answer]{.underline}) or some other link after each instead.

9.  *Can a woman who has just given birth donate the umbilical cord
    blood so that new stem cells can be harvested?*

> 3 users had seen this in the FAQs; the other 2 tried various Research
> Topics links or thought about searching.
>
> **Recommendation:** Although the Research Topics section is primarily
> aimed at researchers, the general public expects to find information
> on research there. As above, creating a "Current Research" page may
> give them a better starting point. Reducing the number of top-level
> navigation links will also cut down on trial-and-error clicking.

10. *Can you find links to related issues/research or resources?*

> All eventually found Other Online Resources, but many users thought
> the News & Announcements or Media Center pages would lead to non-NIH
> information.
>
> **Recommendation:** As above, rename News & Announcements to just
> Announcements. Rename Media Center to "Downloads."
>
> [Students]{.underline}
>
> You have been assigned a research paper on stem cells. Your professor
> recommended the NIH Stem Cell Information web site as a resource, and
> you are visiting it to get answers to the following questions.

1.  *What are the differences between embryonic and adult stem cells,
    and how are each used in research?*

> Only one user clicked on "What Are Stem Cells?" in the Fast Answers.
> One other student went directly to Stem Cell Basics. The others
> considered searching, tried the Glossary, or scanned links under
> Research Topics.
>
> **Recommendation:** As above, move Fast Answers to navigation and
> clarify/streamline the Research Topics navigation links.

2.  *What are the pros and cons of stem cell research?*

> Only one student clicked on "Stem Cells and Research Ethics." Others
> tried Stem Cell Basics, the scientific report, or the FAQs.
>
> **Recommendation:** Rename link to Research Ethics page.

3.  *What testimony has Congress heard on stem cell research?*

> Most users found the "Statements and Testimony" link in the
> navigation, although it took time because it was so far down the list.
> One user tried News and Announcements.
>
> **Recommendation:** Keep important navigation elements "above the
> fold" by reducing the number of top-level links in the navigation bar.

4.  *What is the current US policy about stem cell research?*

> Every user understandably tried the Policy & Guidelines page to no
> avail.
>
> **Recommendation:** As above, add eligibility criteria to main Federal
> Policy and/or Policy & Guidelines page(s). Since this is such an
> important part of the site, add prominent link on home page.

5.  *How does US policy compare to the policies of other countries?*

> Only 2 students found the "International Research" link in the
> navigation. Two others never found this information; the final student
> used the student guide, which linked to International Research.
>
> **Recommendation:** As above, shorten the navigation bar to keep
> elements above the fold.

6.  *What are some recent developments in stem cell research?*

> Only 1 student found the Highlights of Stem Cell Research page, which
> is buried in the Scientific Literature page. The others unanimously
> felt the News section was the proper place to look for recent research
> developments.
>
> **Recommendation:** As above, rename News & Announcements to just
> Announcements. Use the new Current Research page to steer users to the
> Highlights of Stem Cell Research page.

7.  *Does the site provide any diagrams or illustrations you could use
    in your paper?*

> Although most students eventually found the Media Center, no one
> actually expected to find pictures there based on its name. Most tried
> the NIH Libraries and Projects page first, finding an extraneous but
> prominent link to the IMAGE consortium.
>
> **Recommendation:** Rename Media Center to "Downloads." Deemphasize
> IMAGE consortium link.

8.  *How should you reference this information in your paper?*

> Only 2 students found this information, which was in the Student
> Resource Guide. Most immediately checked the page's footer, where they
> expected to find citation information.
>
> **Recommendation:** Remove Student Resource Guide image and distribute
> relevant links throughout FAQs/Fast Answers or Current Research. Add
> citation info to footer of each page.

9.  *Where can you go to find more information?*

> Most students had no trouble finding the Other Online Resources link;
> 1 student used the FAQs instead.

10. Who *manages this site? Is there any indication about how to contact
    them if you have a question about the site?*

> Students also had no difficulty finding the Contact Us link in the
> footer.
>
> [Researchers]{.underline}

1.  *How can scientists obtain stem cells for research?*

> Unsurprisingly, researchers had no trouble finding the Registry.
>
> **Recommendation:** Maintain prominence of Registry on home page.

2.  *Who holds the intellectual property rights to those cells?*

> However, none of the researchers recognized the MOU/MTA links on the
> Registry outlined property rights. All suggested using the contact
> info to get in touch with providers directly.
>
> **Recommendation:** Explain MOU/MTA links on Registry home page.
> Perhaps include Research Protocols so it doesn't have to be a
> top-level navigation link.

3.  *What are the specific limitations on the kinds of experiments that
    can and can not be federally funded?*

> Only 1 researcher found the clearest delineation of policy, on the
> Eligibility Criteria page, by using the Fast Answers link. Like the
> general public, most researchers turned first to the Policy &
> Guidelines page; some tried Research Funding.
>
> **Recommendation:** As above, add prominent link to current policy on
> home page and succinct statement to Policy & Guidelines page.

4.  *Where can you find descriptions of current funding opportunities
    (RFAs/PAs)?*

> All researchers quickly found the Research Funding and Funding
> Annoucements pages.

5.  *Does NIH have any kind of advisory group or panel that guides the
    decisions on stem cell research?*

> Most easily located the Task Force page, although they weren't certain
> about about its role.
>
> **Recommendation:** Adding explanatory content to root-level pages can
> help users make sense of short or cryptic link text.

6.  *Does NIH have any public databases or scientific resources related
    to stem cells?*

> The researchers usually zeroed in on NIH Libraries and Projects
> quickly. Finding the databases under Scientific Literature was harder;
> one user expected to find only articles there.
>
> **Recommendation:** As above, add explanatory content to root-level
> pages.

7.  *Can you find out the critical characteristics of the existing cell
    lines? (What markers do they carry?)*

> No researcher found the NIH Stem Cell Unit pages with this
> information. All found the Registry's Tables of Cell Characteristics.
>
> **Recommendation:** Add notice of Stem Cell Unit's work to Registry.
> Use the new Current Research page to explain the Stem Cell Unit's
> mission.

8.  *Does the site provide information about current stem cell
    research?*

> Only one researcher found the Highlights of Stem Cell Research page;
> most tried News & Announcements, Stem Cell Research at NIH, or the
> Research Protocols pages.
>
> **Recommendation:** As above, use new Current Research page to steer
> users to Highlights of Stem Cell Research and rename News &
> Announcements to just Announcements.

9.  *What training opportunities are available for researchers?*

> Every respondent found the Training Programs link in the navigation.

10. *Are there any conferences or talks on stem cells for researchers to
    attend?*

> One researcher expected to find this under Training Programs, but all
> quickly found the Upcoming Events page.
>
> **Recommendation:** As above, maintain left-hand navigation,
> particularly for links aimed at researchers.

**Debriefing Responses**

Some respondents could not tell that this was an NIH Web site, or
wondered whether it was part of a particular institute.

> ***[Recommendation:]{.underline}*** Add NIH logo to header.

One user thought "What Are Stem Cells?" should be the first item on the
home page.

> ***[Recommendation:]{.underline}*** Since few respondents used the
> topic images at the top of the home page to navigate, and since they
> take up valuable vertical space, remove them and replace with
> important links and explanation: Promise of Stem Cells, with link to
> Stem Cell Basics; U.S. Policy on Stem Cell Research, with link to
> Eligibility Criteria; and The NIH Stem Cell Registry.

Students would like more videos and illustrations.

> ***[Recommendation:]{.underline}*** Include more PowerPoint slideshows
> from Dr. Battey's lectures; convert to HTML as well for easier
> accessibility.

Many respondents wanted a page detailing the NIH-funded laboratories
and/or scientists working on stem cell research. (Uses typically thought
the Meet Our Scientitst page would have this information.) One user also
felt the Highlights of Scientific Research should include breaking
research as well as significant developments.

> ***[Recommendation:]{.underline}*** Currently, the site simply links
> to institute pages on stem cell research. Although this approach would
> require more maintenance, it also would enhance the site's content.
> Recommend adding a page listing current labs' research if possible.

One respondent specifically lauded the color palette.

> ***[Recommendation:]{.underline}*** Maintain color scheme.