# Presentation: 120322

## Qualitative Research StudyonWorld Library of Toxicology

*Prepared for:*
- _[ ](http://www.markitecture.com/home.htm)_
- March 7, 2006

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Background

- As part of an ongoing qualitative evaluation of NLM websites, the National Library of Medicine conducted one online focus group on the World Library of Toxicology, Chemical Safety, and Environmental Health.
- This qualitative study had as its objective understanding the value, usefulness, strengths, and weaknesses of the proposed portal for potential users of the site (professionals in the field of toxicology and occupational health).
- The online focus group on the World Library site was conducted on February 23, 2006.

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Study Objectives

- The main objective of this qualitative study on the World Library of Toxicology (WLT) was to evaluate the value and usefulness to potential users of a portal that provides free global information resources in toxicology and ancillary disciplines.  Additional objectives included:
- Gauging feedback on navigational strengths and weaknesses of the current portal
- Understanding the value of the current content
- Soliciting feedback on additional content users would like to see
- Understanding how users anticipate using the site
- _Target respondents_.  Screening parameters for respondents were as follows:
- Mix of potential users who are physicians and/or toxicologists
- Individuals who are employed globally as professionals in the fields of toxicology and occupational health
- Recruited from listserv subscriptions and personal contacts who shared the posting
- All were asked to spend approximately 20 minutes viewing/navigating the password-protected database and completing brief tasks prior to the discussion as an aid to familiarizing themselves with the portal.

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Methodology:  Online Focus Group*

- Testing consisted of ONE online focus group of professionals in the toxicology and occupational health community.
- Respondents were recruited via listserv subscription and personal contacts at relevant organizations.
- The session lasted approximately 90 minutes and was comprised of nine (9) participants.  (See Appendix.)
- The group was asked to visit WLT and given a user name and password prior to the session.  They were also asked to complete a series of tasks using the portal.
- Participants in this group received no incentive payment.  All participants were active volunteers in the discussion.
- * *The online focus group represents a qualitative methodology used for the purposes of ideation, brainstorming, and evaluation.  Qualitative methodologies are based on a small sample size, and the findings are intended to be directional only, not projectable to the larger population.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Summary Conclusions

_**Impressions of WLT**_
- There were a variety of reactions to the overall WLT portal:  While respondents were pleased to have such a portal, there was a general sense that it **lacked a "defined" audience** and had **some navigational obstacles**.
- Respondents were very pleased that there will be a comprehensive international portal on toxicology resources.
- Common current challenges include finding the right site with accurate information, followed by the issue of language (e.g., not finding the information in their native language).
- Some of the navigational difficulties respondents experienced were related to categorical searches.  Some suggested adding the ability to search by chemical, by subject, and by sector.
- While a "star system" and user reviews of the content was not highly appealing, the idea of expert reviews (by professionals in the field) was, in contrast, seen as a very useful addition to the site.

## Summary Conclusions (cont’d.)

_**WLT**_ (cont'd.)
- Respondents appreciated WLT for its potential as a one-stop portal and international resource for information on toxicology and chemical safety.
- Respondents did make the fair assumption that the site was geared toward a "professional" target audience.
- Respondents agreed that the site could be used in an educational setting, for students and teachers as a training or educational tool.
- Some also felt that the site could be useful to legislators in the field of toxicology.
- Additionally, teaming with professional scientific societies would lend a great deal of validity to the content on the site.
- Anticipated usage was high, given that the portal continues to develop and grow according to the suggestions herein.

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Overall Findings: World Library

- Current information seeking
-  Respondents in the online focus group discussion on the World Library of Toxicology ranged from a Medical toxicologist from San Antonio, Texas to a physician from the Ukraine.
-  Current information sources include both online and hard-copy texts.
-  Online resources cited for getting information on toxicology issues included PubMed, Toxline, ToxNet, Micromedex, HSDB, and ChemID.
-  Respondents also rely on international journals.
-  Specifically, respondents seek out information on **chemical exposure**, **scientific consensus** on specific topics, **current legislation** and **regulations**, and other aspects of **chemical safety**.
-  Additionally, many respondents stated that they seek out information of global concern, such as international regulations on chemicals.

## Overall Findings: World Library

- Challenges in Finding Information
-  Language differences comprised the main challenges respondents encountered in finding the information they needed on the Internet.  Even once they find the site they need (or information they need), the obstacle of language/translation is a common hurdle.
-  Reliability of the information is another occasional obstacle.

- "Of course the other issue, once you find the right site, is language.”  (Stephen)

- “language and reliability of information/data are major issues in many developing countries.” (Jan)

- “Reliability of test results and obtaining an overview of how various professional judge the adequacy of the information and draw conclusions.” (Barbara)

## Overall Findings: World Library

- Challenges in Finding Information (cont'd.)
-  Typically, respondents address these obstacles by verifying information with colleagues overseas (or locally), and by checking the information against a variety of sources.
-   In other cases, navigational difficulties on a site can result in the respondent leaving the site to find the information elsewhere.
-  Another common challenge is finding online (free) access to journal articles.

- “If the Web site is difficult to navigate, I tend to abandon it and look elsewhere.” (Barbara)

- “journals don't always have free electronic access to articles and so there is a temptation to use the most easy to get information.” (Lesley)

## Overall Findings: World Library

- Impressions of World Library of Toxicology site
-  Respondent reactions to the World Library site were that there was a "lot of information" which yielded many "questions."
-  Others commented initially on the navigational aspects of the site.

- “A lot of information, easy to find, meet some difficulties to find the information.” (Irina)

- "It was reasonably tricky to navigate - and the sources included were a mixture of different types all presented with the same importance, It seemed better when you drove it by the visual aid i.e., the world map.” (Lesley)

- "much information noise ... too many narrow issues, but insufficient generalization.” (Yuriy)

- "maybe not as user friendly as it could be.” (Herman)

## Overall Findings: World Library

- Impressions of World Library of Toxicology site (cont'd.)
-  **Strengths** of the World Library site included the "wealth" of information available, links to other sites, and contact information.
-  Some respondents also felt the categories were appropriate, though it might be useful to have alternative approaches to category searches as well.
-  Navigationally, respondents called for more "panel buttons" on the home page, the removal of the "three buttons" at the bottom of the screen, the inclusion of a site map, and references to other databases.

- "There are other searches to look for specific information. I guess the country-based information is new and useful.” (Amalia)

- "There should be other ways to approach it, rather than just by country or continent ... I might want to search for pesticide regulations in all countries.” (Stephen)

- "It would be nice to find risk assessments and exposure information easily regardless of country.” (Herman)

## Overall Findings: World Library

- World Library Areas for Improvement
-  A reorganization of the category structure was called for, with alternatives such as searching by occupation, by sector, by subject, by chemical, and across sources.
-  Additionally, a variety of formats is critical, considering the breadth of the target audience.
-  Other useful improvements to the site include adding **introductory paragraphs** for each site, using a **"questionnaire approach"** that helps direct the user to the right place, a **site map** or other navigational aids, and **additional content** such as full texts of review articles.

- “Also nice to have a search tool that searched across sources so you didn't have to categorise.” (Lesley)

- “we have to offer in different formats and forms for different users. As pointed out, the target audience is a key factor.” (Abey)

## Overall Findings: World Library

- User/Reader ratings
-  The idea of incorporating User Ratings was generally perceived as being of "little value."  The "star system" (drawing analogies to Amazon.com) was not seen as a highly useful tool in validating site information.
-  On the other hand, reviews by experts (professionals in the field) would be extremely valuable as an assessment of the site content, as would coordinating efforts with professional societies.

- "I don't think such ratings are that useful.  They are very dependent on the people who take the time to do the rating.” (Herman)

- "Not interested - just as not interested in Amazon reviews ... Generally think the number of stars useless.” (Lesley)

- "Well, I doubt I would find a review by a layperson as extremely useful.” (Stephen)

- "Scientific societies could help to assure reliability.” (Amalia)

## Overall Findings: World Library

- Target Audience
-  Generally, respondents recognized the target group for the World Library site as being professionals like themselves.
-  Mores specifically, some respondents felt that the World Library site could be useful to legislators and regulators in the field of toxicology.

- I really couldn't tell - it seemed to be an international directory of resources." (Lesley)

- “as the TARGET we see scientists and national organizations on chemical safety”  (Yuriy)

- “I see professionals as the primary audience" (Amalia)

- "I'm sure it will be of interest to labor, legislators, and probably lawyers...." (Stephen)

## Overall Findings: World Library

- Anticipated Usage
-  Respondents definitely expected to use the World Toxicology site on a regular basis, for "country-specific queries" and as a "worldwide portal."  The only caveat was that quality control of the content be stringently maintained.
-  One respondent in particular finds most value in the World Toxicology site if it leads *directly* to content, rather than a series of links.

- “it is a site very useful to us. It can be made useful to many groups by zoning information for different targets." (Abey)

- “... if the portal delivers you the actual information you seek and does not deliver you at the door of yet another Web site.”  (Lesley)

- “I will use it too but quality control is very important." (Jan)

- “we need information on cardiotoxicology, immunotoxicology and age toxicology.”  (Yuriy)

## Overall Findings: World Library

- Other anticipated uses
-  Respondents also agreed that the World Toxicology site could be a useful tool for students and teachers.

- "a one stop source for practice guidelines (position papers, expert panels) in toxicology would be useful as well." (Stephen)

- “I think the site could be very helpful.  I think it will be a site used by students, attorneys, governments, etc." (Herman)

## Overall Findings: World Library

- Discovery of the World Toxicology portal
-  Respondents stated that they would expect to find links to the World Toxicology site on several NLM sites and on other international/global sites, as well as professional organization sites.  They would also expect it to be easily found via a Google search.

- "I think many will find it via an NLM portal... toxnet ... I think many professional organizations would add a link if allowed to do so." (Stephen)

- “I believe many key sites will link to it. We will know from the NLM site and through web searches." (Abey)

- "Google search, those who know about the NLM portal already will easily find this site." (Barbara)

## Overall Findings: World Library

- Key Next Steps
-  Suggested "next steps" included reassessing and defining target audience, reorganizing the site structure to accommodate the target, and creating search categories that are intuitive and relevant.

- “A dedicated search engine, more ways to approach the data other than by country ... Sites must be categorized so that searching across them by specific criteria is easy" (Stephen)

- “Define target audience; decide on referral role or DB role; investigate possibilities for teaming with IOMC, etc... professionals in toxicology and chemical safety are quite different animals" (Jan)

- “seek further feedback on key specific aspects from the target audience and then respond accordingly" (Abey)

## Overall Findings: World Library

- Key Next Steps (cont'd.)
-  Then, linking to relevant organizations (including other NLM sites), adding peer reviews or expert ratings, and teaming with professional societies will add to the validity and accuracy of site content and information.
*Following is a series of recommendations that emerged from the results of this online focus group.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix

## Recommendations

- A few recommendations that emerged from this focus group included the following:
- Define target audience more specifically and clearly.
- Reorganize the navigation, according to different categories, such as chemicals, sectors, occupations (in addition to country).
- Add introductory paragraphs or statements to each section or country, as a "lead in."
-  Pursue the joining of forces with other professional societies to help validate accuracy of content.
-  Provide a "peer review" or "expert evaluation" of content.
-  Link to other relevant sites (including NLM sites and in education-based contexts) to increase awareness of the portal and encourage traffic and site visitation.
*Following is a respondent profile of participants in this online focus group.*

## Background

- Study Objectives
- Methodology:  Online Focus Group
- Summary Conclusions
- Overall Findings:  World Library
- Recommendations
- Appendix:  Respondent Profile

## Appendix:  Respondent Profile

## Appendix:  Respondent Profile - World Library

| # | Name | Gender | City, State | Occupation |
| --- | --- | --- | --- | --- |
| 1 | Abey | M | Ontario, Canada | CCOHS (Canadian Centre for Occupational Health and Safety) |
| 2 | Amalia | F | Montevideo, Uruguay | Department of Toxicology. Poison Control Center (CIAT) |
| 3 | Barbara | F | Washington, DC | Environmental Protection Agency |
| 4 | Herman | M | Alexandria, VA | Epidemiologist |
| 5 | Irina | F | Minsk, Belarus | Deputy Director of Hygiene Center |
| 6 | Jan | M | Oostkapelle, the Netherlands | U.N. Chemical Safety expert |
| 7 | Lesley | F | Geneva, Switzerland | World Health Organization |
| 8 | Stephen | M | San Antonio, TX | Medical toxicologist |
| 9 | Yuriy | M | Kiev, Ukraine | Physician; Toxicologist; Institute for Occupational Health |

## Contact:

**Contact:**
***Mary Beth Solomon***
***For inquiries and capabilities****** ***
***on***
***Qualitative Analytics***
***201.434.0404***
***917.601.1273***