# Presentation: 005475

## Qualitative Research StudyonToxSeek

*Prepared for:*
- _[ ](http://www.markitecture.com/home.htm)_
- April 28, 2006

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  ToxSeek
- Recommendations
- Appendix

## Background

- As part of an ongoing qualitative evaluation of NLM websites, the National Library of Medicine conducted one online focus group on ToxSeek – a Web interface that performs a metasearch (or a “federated search”) and then clusters results from many sites and databases used for environmental health and toxicology information.
- This qualitative study had as its objective understanding the strengths, weaknesses, value, perceptions of its comprehensiveness, and other perceptions of the database for its potential primary user base (medical researchers, academics, librarians, and students.).
- The online focus group on the ToxSeek proposed database was conducted on April 18, 2006.

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  ToxSeek
- Recommendations
- Appendix

## Study Objectives

- The main objectives of this qualitative study on ToxSeek were to evaluate the strengths, weaknesses, value, and usefulness of ToxSeek to potential users.  Additional objectives included:
- Gauging the strengths and weaknesses of the content
- Gaining feedback on the organization of search results
- Understanding how respondents anticipate using Toxseek
- Gathering reactions to the comprehensiveness of the resources provided on ToxSeek
- _Target respondents_.  Screening parameters for respondents were as follows:
- Employed as professionals in the field of medical sciences, environmental health, library studies, and toxicology.
- All respondents were recruited from listserv subscriptions.
- All were asked to spend approximately 20 minutes navigating and using ToxSeek and completing a pre-group assignment prior to the discussion.

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  ToxSeek
- Recommendations
- Appendix

## Methodology:  Online Focus Group*

- Testing consisted of ONE online focus group of professionals in the ToxSeek community.
- Respondents were recruited via listserv subscription.
- The session lasted approximately 90 minutes and was comprised of 13 participants.  (See Appendix.)
- The group was asked to visit ToxSeek and given a user name and password prior to the session.  They were also asked to complete a series of tasks using the site.
- All participants received an incentive payment of $45.
- * *The online focus group represents a qualitative methodology used for the purposes of ideation, brainstorming, and evaluation.  Qualitative methodologies are based on a small sample size, and the findings are intended to be directional only, not projectable to the larger population.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings: ToxSeek
- Recommendations
- Appendix

## Summary Conclusions

_**Impressions of ToxSeek and Its Strengths**_
- ToxSeek's **initial impact** drew qualified reactions:  Some felt that the amount of information might be overwhelming (something which eventually became a benefit), while others found the interface initially confusing.
- With use, however, confusion dissipated, and ToxSeek's metasearching strengths became apparent.
- ToxSeek's biggest **strengths** are clearly its **clustering** capabilities, combined with being a **single destination** for environmental health and toxicology information.
- Additionally, the ability to **sort the search results** -- alphabetically, by relevance, and by source --  stood out as a unique and distinguishing feature of the site.
- Also outstanding as ToxSeek strengths are the scope and comprehensive nature of the site.
- The organization of categories on the ToxSeek home page, seen as logical and intuitive, is a clear strength as well.
- The "alternate spelling" and "preview" features also add to the value of ToxSeek.

## Summary Conclusions (cont’d.)

_**ToxSeek Area for Improvement**_
- Areas for improvement were primarily related to improving upon what ToxSeek already offers.
- For instance, **improvements** that may benefit the ToxSeek user experience include increasing sorting functionality to the pull-down menu (e.g., author, journal title, publication date); providing more aids to acronym definition (e.g., an acronym glossary, on-screen descriptors); and possibly making the "help" link in the top navigation more prominent.
- Another area of ToxSeek that could be given greater **prominence** is the ability to **add/remove** listings from the clustered search results.

## Summary Conclusions (cont’d.)

_**ToxSeek**_
- There is definitely a need for other syndrome-specific metasearches like ToxSeek, particularly in the area of disaster recovery and emergency response.
- Overall, respondents perceive ToxSeek as being a site for **advanced** or **skilled researchers** (including themselves), and is not a site for the general public.
- Across the board, ToxSeek was seen as a valuable **complementary** **resource **for information on environmental health and toxicology, with some users stating that they would be likely to use it **exclusively**.
*Following are additional findings from the online focus group testing on ToxSeek.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings: ToxSeek
- Recommendations
- Appendix

## Overall Findings: ToxSeek

- Current information gathering patterns and preferences
- Where and how respondents currently gather information on environmental health and toxicology was largely dependent on the information they're seeking.
- Resources mentioned for environmental health and toxicology searches included some general search sites (e.g., Google or Yahoo), and some specific-to-health sites, such as **PubMed**, **MedLine Plus**, the **CDC**.
- The most commonly-mentioned resources for environmental health and toxicology searches, however, were **PubMed** and **MedLine Plus**.
- Also mentioned were the NLM's MEDLARS system, ToxNet, EPA, and FDA.
**Google** was named as the most commonly-used search tool on the Internet by far (for any type of information), often being used as a starting point.

## Overall Findings: ToxSeek

- Current information gathering patterns and preferences (cont'd.)
- Respondents offered a broad range of examples of recent searches they've conducted online, including a search for **sludge processing and mercury output**; **melanoma investigation and treatment**; **lead poisoning and retardation**; **calcium diet supplements and osteoporosis**; diagnosing **chemical allergies**, and secondary exposure to **pesticides**.
**Government** information was not high on the list of type of information, but when necessary, respondents mentioned periodically visiting **FirstGov.gov**, and for **science** information, **Web of Science**, in addition to the usual PubMed, ToxNet, and CDC.
- Metasearch sites that were routinely visited for compiling information from a variety of sources included **Dogpile**; **Metacrawler**; **Webcrawler**; and **Clusty**.
- Other metasearch sites mentioned included Questia, ChemFinder, and Cochrane DataBase.
- Respondents noted that metasearching is particularly useful for circumstances such as:
- Accessing information that is either highly focused or very broad (one extreme or the other)
- Casting broader nets
- Definitions from a variety of sources
- Information that is not necessarily found in journals or other literature

## Overall Findings: ToxSeek

- Challenges in Finding Information
- The biggest challenge reported in using metasearch tools was getting "too much information" -- which one respondent described as "staggering" -- and, as a result, having to sift through results that include multiple listings of the same data (redundancy) and to separate the results by various criteria, such as format, source, etc.
- Respondents address these challenges in a variety of ways:  Sorting through the results, verifying the data with colleagues and other sites, and refining the search or conducting an entirely new search.

- "Challenges that I encounter is finding out too much information and then having to choose what to include in the literature search that I have to give to the library patron.”  (Darlene)

- “Challenges: too much redundancy in commercial sites, advertising, inability to separate search results by format.” (Pamela)

- "I search multiple sites several times each to ensure I've tracked down as many sources of info as possible.” (Allison)

- “relevancy of the results and my students are sometimes overwhelmed by the staggering numbers of results.” (Walter)

## Overall Findings: ToxSeek

- Impressions of ToxSeek
- The majority of respondents were previously unfamiliar with ToxSeek.
- Respondents would expect to use ToxSeek, however, for information such as **pollution**, **lead paint**, **chemicals**, **pesticides**, and the effects of **laboratory waste disposal**.

- “We get health services students asking about various chemicals and their regulation. I think I might try ToxSeek first next time.” (Terry)

- "I had a pregnant patient who worked as a dental technician and was concerned about workplace exposures.” (Paul)

## Overall Findings: ToxSeek

- Impressions of ToxSeek (cont'd.)
- Respondents' *initial* impressions of ToxSeek were mixed.  Several liked the "broad coverage" of information ToxSeek provides, while others felt that it was "too much information."
- Still others found the results useful but *initially* somewhat confusing.
- With increasing use, however, these issues became considerably less confusing to respondents.

- “I appreciated its initial clean interface, and the results of the practice searches seemed adequate. I was hesitant about the organization of the results once they appeared.” (Ely)

- "Easy and fast but display of results could use some improvement, e.g., telling me more what the document is without having to click on it.” (Terry)

## Overall Findings: ToxSeek

- Strengths of ToxSeek
- ToxSeek performed very strongly in the areas of the **clustering** feature, its **one-stop** quality, and providing the user the ability to **sort** results in a variety of ways.

- “I loved being able to group results by product type.” (Pamela)

- "I could do one search and cover many different databases.” (Martina)

- "I liked the clustering tool ... It was an immediate way to view how all the items were sorted, and another method of narrowing my search.” (Ely)

- "One stop shopping w/ fully stocked shelves.” (George)

## Overall Findings: ToxSeek

- ToxSeek Areas for Improvement
- While the three methods of sorting were useful (alphabetic, relevance, and source), some also suggested being able to sort by **chronology** and and other variables.
-  One respondent questioned the relevance order of the results.

- “I was disappointed that I could not sort by chronology.” (Elmer)

- “Sometimes the first few hits on the search were not always the most relevant.” (Paul)

- “... I just wished I could have sub-divided the clustered results by literature type.” (Pamela)

## Overall Findings: ToxSeek

- Suggestions for Change
-  **Suggestions for change** to ToxSeek ranged from adding more sorting features to spelling out the acronyms on-screen.
-  One user also experienced difficulty finding the "help" link in the top navigation.

- “I would add descriptors to each of the database selection in the initial menu ... Simplify the results screen to ease readability. I would also like to see the information divided by doc. type.”  (Ely)

- "It might be useful to expand the acronyms so that the researcher knows what they are for. I have learned about many of them, but more through a process of trial and error.”  (Paul)

- “sort by date of pub or entry and a boolean search capability.”  (Elmer)

## Overall Findings: ToxSeek

- Scope of Resources on ToxSeek
-  Reactions to the list of resources on ToxSeek **varied**.  Some felt that the scope of the list was fully **comprehensive** or could even stand for additional resources.
-  Others, however, felt **overwhelmed** by the number of resources, suggesting that the search start small and expand if necessary.

- "Scope is pretty comprehensive--not too long unless you decide to select everything.  Still, I'm not sure if sources such as ToxFAQs were included..." (Pamela)

- “I like it.  Appropriate.”  (PCYuan)

- “excellent scope. more sources would be even better." (Elmer)

- “I was a bit overwhelmed, but that is b/c I don't frequently use some of the resources ... The listing of the initial menu was a bit much." (Ely)

- “I think the scope of this list is too long. I felt there were too many choices at once. Maybe it can change to half the list and when the results come back then it can ask if you would like to expand your searching with more databases." (Darlene)

## Overall Findings: ToxSeek

- Organization of the ToxSeek Categories
- Respondents consistently stated, however, that the **organization** of categories on the ToxSeek home page were **intuitive** and **logical**, suggesting only that the acronyms be clarified on screen (or that the rollovers be more obvious, as several respondents overlooked the rollovers).
- Other suggestions included adding resources to the international list and a category titled "Other Resources."
-  One respondent would like the site to be compatible with EndNotes.

- “I have no problem with the organization, just want the acronyms identified more clearly.”  (Martina)

- “The international list is a bit short ...The Brits and Canada for instance also have great sites which should be included ... Health Canada, the centre for occupational health are excellent.  Cochrane reviews are also good.”  (Paul)

- “You left out the UN's Envir Programme - UNEP.”  (George)

## Overall Findings: ToxSeek

- Incorporating other Search Tools
-  The concept of adding **Ask.com**, **Google.com**, **MSN.com**, and/or **Yahoo.com** to ToxSeek received a lukewarm response.  Most felt that merging the two types of searches would be unnecessary and possibly create redundancy.

- "It would not be helpful IMO." (Allison)

- “Might be too much info and a great deal of junk, unless Toxseek could filter out the redundancy--I'd vote against it." (Pamela)

- "Only if you can include or exclude them at will." (Terry)

- “My sense of the ask.com and the others is that they are all basically the same if not identical. I don't think they would contribute to ToxSeek and to a great extent would probably make it worse." (Paul)

## Overall Findings: ToxSeek

- ToxSeek Search Results
- Generally, respondents felt that the results layout (in three groupings) took some adjustment.  Initially, some were confused but figured it out, and others jumped straight to the familiar-looking listings results at the bottom right.
-  The clustering of results was a very strong feature of the search results page.

- "confusing at first but after i looked at the results they made sense and I was ok with the display. it is a bit much at first." (Martina)

- "A bit overwhelming but manageable. I would tend to scroll down to the bottom section and start looking at the previews to see if they were relevant." (Paul)

- “I had a tendency to ignore that data at the top and check out the groupings and the top few results. Loved the groupings." (Jane)

- "I liked the clustering. Some of the clusters I just ignored, such as the ones that only explained one of my search terms but not the interactions between the two." (Pamela)

## Overall Findings: ToxSeek

- ToxSeek Search Results (cont'd.)
- The results at the top (where sites and databases were listed) was clear and useful to some but of limited use to other respondents.
- It was also clear to respondents that clicking on one of the links in the top section would take them directly to a site.
- The results on the lower left side of the page (Result Clusters), on the other hand, were extremely helpful, particularly in narrowing the search results.  Few, however, automatically noticed that it was possible to **add** and **remove** results from the list.

- "It is not info that I need to know." (Jane)

- "they were not useful to me at present but might be at some later date." (Martina)

- “I would use the Result Clusters to focus my results, that is zoom in on a specific aspect much the way MeSH subheadings work." (Terry)

- "very helpful when I narrow my study." (PCYuan)

## Overall Findings: ToxSeek

- ToxSeek Search Results (cont'd.)
- The results on the lower right side of the results page was the most familiar-looking of the three groupings to respondents.
- The **sort** feature of this results section was one of the strongest points of ToxSeek, but some would like to see other sort categories added.

- "I found the result listing typical of other engines. Nothing unexpected." (Ely)

- "I like the re-order feature.  I can pick and choose the way I want the results to display." (Martina)

- “Would just as soon keep: sort by author, date publ, title of journal, etc..." (George)

- "... for me, this is a great feature, as sometimes I want to see them by what the search engines thinks is relevant and sometimes I need to see the source. Hardly ever is alphabetical important to me..." (Jane)

## Overall Findings: ToxSeek

- Other Features
- The **preview** feature was also highly appealing and very useful.
- Few noticed the **alternative spelling** feature but like the fact that it would catch any potential misspellings.

- "I didn't notice this but it is always a great idea" (Ely)

- “I can imagine that it will be useful" (Raj)

- "Preview is good; that way you can determine if you should add a term." (Darlene)

- “I did take a look at it, the preview feature is good." (Paul)

## Overall Findings: ToxSeek

- Target Audience
- Target audience for ToxSeek was seen primarily as being academic -- researchers, students, librarians -- and those in the environmental health and toxicology field.
- Reactions were mixed as to whether a general public might use ToxSeek.

- "I can see students using it as well as the general public but I think they might need more intervention and help ... the interface is a bit overwhelming so I think that the general public would have difficult with this search tool.”  (Martina)

- “regulators like EPA, toxicologists, graduate students, etc.  I suspect that some knowledgeable general public uses it too.”  (Raj)

- “Toxseek is for the more experienced researcher ... I would say it is not for the more casual general public researcher.  I think you need to be a well informed individual, who knows what they are looking for.”  (Paul)

- “It seems on a higher level than Google and lower than some scientific databases. Perhaps the allied health professions? General researchers? Students?”  (Jane)

## Overall Findings: ToxSeek

- Anticipated Usage
- Anticipated usage of ToxSeek was high.  Most respondents reported that they expect to use ToxSeek as a supplementary resource, though some might use it exclusively.
- Additionally, the majority of respondents either had already or intend to bookmark the ToxSeek site and are very likely to recommend the site to others.

- "Would be more likely to use it for a search where I know nothing about the subject (or very little) to get broad overview and not know where to start. If I want a particular fact, I'll start with a single source.”  (Terry)

- “I'll use it. i really have not used metasearch engines before, but will from now on ... I'll probably still use my other search engines...old habits die hard.”  (Allison)

- “I would simply because I can do a more powerful search than I can using individual databases.”  (Martina)

## Overall Findings: ToxSeek

- Other Needs for "Seek"
- The areas of **emergency response** and **disaster recovery**, as well as **nutrition** and other **syndrome-specific topics** could benefit from a metasearch site similar to ToxSeek.

- “Yes, emergency make it available for PDA. Like when the Hurricane hit in New Orleans and computers were not available.”  (Darlene)

- “As a subject specialist librarian, subject specific resources are always valuable...our patrons don't have to dig through unrelated material to find what they want.”  (Ely)

## Overall Findings: ToxSeek

- The Uniqueness of ToxSeek
- Clearly, ToxSeek derives its uniqueness from its clustering feature and comprehensive nature.
- As a result of this unique feature, Toxseek has set itself apart from other metasearch tools in the environmental health and toxicology arena.

- "The clustering is definitely unique. It's an interesting way to narrow a search.”  (Pamela)

- “It seemed more 'professional' and scholarly b/c of its interface. As a librarian or researcher, I would recommended ToxSeek rather than Dogpile, just so the public can get used to a more 'intelligent' interface. I also appreciated the clustering tool, compared to others that did not have one.”  (Ely)

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  ToxSeek
- Recommendations
- Appendix

## Recommendations

- A few recommendations emerged from this focus group, including the following:
-  Clarify the mouse rollovers on the acronyms; they may be easily overlooked in their current form.
-  Lengthen the list of international resources, including sources from the U.K. and Canada.
-  Consider the addition of more sorting categories, such as chronology, author, journal title, and publication date.
-  Highlight the ability to add and remove results from the list in the lower left grouping of search results; currently it may be getting lost within the text.
-  Make ToxSeek compatible with EndNotes.
-  Highlight or make prominent the "Help" link that currently appears in the top navigation.
*Following is a respondent profile of participants in this online focus group.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings: ToxSeek
- Recommendations
- Appendix:  Respondent Profile

## Appendix:  Respondent Profile

## Appendix:  Respondent Profile - ToxSeek

| # | Name | Gender | City, State | Occupation |
| --- | --- | --- | --- | --- |
| 1 | Allison | F | Round Rock, TX | Toxicologist |
| 2 | Darlene | F | Chicago, IL | Medical librarian |
| 3 | Elmer | M | Freeburg, IL | Consulting toxicologist |
| 4 | Ely | M | Chicago, IL | Medical librarian |
| 5 | George | M | Atlanta, GA | CDC librarian |
| 6 | Jane | F | Philadelphia, PA | Writer and editor of CDC/NCEH web content |
| 7 | Martina | F | Slippery Rock, PA | Assistant professor, Slippery Rock University |
| 8 | Pamela | F | Chicago, IL | Biosciences librarian |
| 9 | Paul | M | Bethesda, MD | Occupational medicine physician |
| 10 | PCYuan | M | Jackson, MS | Professor |
| 11 | Raj | M | Chicago, IL | Research Associate professor, Pharmacology Dept. |
| 12 | Terry | F | Seattle, WA | Librarian |
| 13 | Walter | M | Bronx, NY | Library director |

## Contact:

**Contact:**
***Mary Beth Solomon***
***For inquiries and capabilities****** ***
***on***
***Qualitative Analytics***
***201.434.0404***
***917.601.1273***