# Presentation: 555363

## Developer:
University of Pittsburgh Medical Center

Phase 1 Adopter:
University of Pennsylvania

- University of Pittsburgh Medical Center
- Phase 1 Adopter:
- University of Pennsylvania

**cancer Text Information Extraction System** **(caTIES)**

- caBIG TBPT Face to Face Meeting

- 10/27/05

## caTIESWhat Happened in Phase 1?

- Usability Tasks
- Training Tasks
- Bug Testing and Reporting Tasks
- End User Documentation Tasks
- Quality Assurance Tasks
- CAP Cancer Checklist Tasks
- HL7 Feed Tasks
- Internal Review Board Tasks

- caBIG TBPT Face to Face Meeting

- 10/27/05

**Notes:**
USABILITY TASKS
The Penn team conducted usability tests to gain insight into what parts of the caTIES user interface needed to be modified to be more intuitive.  The first step in this process was to create a few tutorials to provide the testers with all the information they needed to participate in the tests. Something that the University of Pittsburgh team recommended to us that we found to be a huge help in this process was the use of Camtasia software.  By using Camtasia we created a video for our tutorials so at the beginning of each Usability Test the tester would put on headphones and watch and listen to the training video.  After the video was shown, we provided tasks for the testers to perform while “thinking aloud”.  During these tasks we used the Camtasia software to record the tester’s computer screen and the audio of the tester’s talking through his/her thoughts and actions for the tasks. 
The audio & video created while the testers walked through each task proved to be a valuable resource in observing challenges and issues the users encountered with the query interface. I highly recommend this technique for usability testing.  I believe the recordings allowed us to create a more thorough Usability Test Report because we could stop, rewind, and replay the videos to ensure we discovered and documented each challenge we observed.
TRAINING TASKS
After the usability tests were performed, training sessions were conducted to teach the “resident testers” how to use the caTIES system so they could search for coding and retrieval bugs.  We trained the same testers who participated in the usability testing.  They were already somewhat familiar with the system and therefore easier to train.  We started by answering any outstanding questions from the usability testing.  We then provided both an interactive, live training demo and a printed training guide to help the “resident testers” get a good grasp of how to use the query interface.  The printed guide included examples that gave a complete step by step walk through with screen shots for every type of query the testers would need to perform.
I feel that both the live demo and the printed guide were necessary in order to provide enough information for the testers to be able to use caTIES on their own.  I would recommend the use of both techniques as supplements for each other.
BUG TESTING AND REPORTING TASKS
The Penn caTIES Team conducted tests to find bugs in caTIES and reported these bugs to the developers through Bugzilla.  I feel that this reporting method worked well. We could query the bug reports, check their status, and add comments or add screen shots to aid in explaining the errors and bugs. I would recommend this technique for adopters reporting bugs to the developers.
Another part of the bug reporting at Penn was to have “resident testers”, who are domain experts, report coding and retrieval bugs.  Since each “resident tester” was not aware of what bugs the other testers had reported it was necessary for the Penn team to screen the bugs to eliminate duplicates and combine similar issues. We created a standardized Bug Report Template that allowed consistent bug reporting by the “resident testers”.  We used an electronic format to ensure there would not be problems understanding a tester’s handwriting and so we could copy & paste the “resident tester’s” bug into the report we entered into Bugzilla.
END USER DOCUMENTATION TASKS
An important adopter tasks for Phase 1 of caTIES was the creation of an End User Guide.  This is a document that provides relevant information for a new caTIES user. (Which would include new adopters.)
The purpose of this document was to:
Educate a new user on the background and purpose of caTIES.
Provide detailed instructions on how to use the query interface.
and
Provide explanations and resolutions for common errors.
Since users of the system will have various levels of computer competencies the user guide should not assume the user has previous knowledge of the caTIES system, and should provide enough explanation that a novice user will not struggle with the document.
Something I highly recommend to be included in the user guide is step by step examples that contain screen shots. This provides an effective way to “walk the user through” the query interface.
At the time the Penn team developed the caTIES end user guide the template that is now available had not yet been created. The template that was available for our reference was more of an outline than a template, and while it did help structure the document, it left a great deal of the document up to our own interpretation.
Additionally, no user guides had been created for the caBIG project yet, so our team did not have an example to follow.  These factors made our experience challenging.  Hopefully writing the user guide for Phase 2 will be more straightforward now that there is a template to follow and now that we have received recommendations and comments on ways to improve our Phase 1 user guide.
 QUALITY ASSURANCE TASKS
The Penn caTIES team conducted quality assurance testing on the ten thousand Surgical Pathology Reports (SPRs) coded through the caTIES beta release.  One percent of these SPRs were inspected to evaluate the accuracy of:
Coding
De-identification
and
Negation
The testers had a checklist of specific issues for which to look, and were asked to examine the entire report for any type of error.   A query set was developed and the testers used both the caTIES dashboard and diagram interfaces to perform the queries.
The query set was just a starting point; the testers modified the queries while testing to ensure a good number of results were returned for evaluation.  SPRs from each of the result sets returned for these queries where then examined and issues where documented.  Most of the examples of errors recorded in the QA report were found repeatedly throughout numerous SPRs.  
Common errors discovered included:
Incorrect/Inconsistent Codings
Presence of False Positives
Presence of False Negatives
Partial Codings
Problems Pertaining to the De-identification Tool (De-Id)
and
Problems Pertaining to the Negation Engine.
CAP CANCER CHECKLIST TASKS
The Penn team assisted in quality assurance work regarding the Breast Cancer CAP Protocols UML Model,  the Melanoma CAP Protocols UML Model, and the Prostate Cancer CAP Protocols UML Model.  Our participation in this task included Reviewing and commenting on the Mapping of each model entity to the corresponding Object classes, properties and qualifier concept(s) and the Mapping of each of the enumerated value domain meaning representations to the corresponding ValueMeaningCode concept.
HL7 FEED TASKS
Additionally, the Penn Team set up an HL7 feed of anatomic and laboratory results to be run through caTIES.  This process involved meeting with HUP IT management numerous times to discuss the process for implementing this feed.  It also involved the installation of eGate software, the performance of network configurations, and the working through of firewall issues and issues related to egate.  Penn’s caTIES machine is now receiving data from the HL7 feed and earlier this week began loading the caTIES schema with this data.  Soon this data will be run through the caTIES pipeline.
INTERNAL REVIEW BOARD TASKS
One last category of tasks that I’d like to mention is the work involved to gain IRB approval.  All of our caTIES IRB papers are available for reference in CVS.  A recommendation I would make to anyone starting the IRB process is to communicate with your IRB early on in the project.  Allow the IRB time to investigate the project, the issues that need to be resolved and encourage communications between your IRB and the developer’s IRB.

## caTIESWhat Are The Preparations For Phase 2?

- Currently Working on Phase 2 SOW
- F2F For Phase 2 Requirements Gathering Has Already Occurred
- Early Coordination With New Adopters
- Bi-Weekly Teleconferences Have Already Started

- caBIG TBPT Face to Face Meeting

- 10/27/05