# Presentation: 027805

## Information Discovery, Use and Sharing

**Mary Brady**
**Program Manager**
**Information Technology Laboratory**
- _[mbrady@nist.gov](mailto:mbrady@nist.gov)_
**(301) 975-4094**

## Information Discovery, Use and Sharing

***Address National Priority areas by applying ITL’s core competencies in ****measurement and testing, mathematical and statistical analyses, modeling and simulation, and standards development and deployment**** to enable:  ***
* **Scientific / Knowledge discovery *
***Develop methods to characterize, analyze, and extract knowledge from information***
*Widespread, seamless, and secure information exchange, and*
***Develop methods for acquiring, defining, securing, accessing, integrating and preserving information (data, metadata)***
*Usability of vast stores of information*
***Improved methods for turning data, metadata into information– search & retrieval***
***Improved accessibility and usability***

- Turn data into information

## Barriers to Innovation

- Massive amounts of data must be gathered, integrated and analyzed
- Human annotation and metadata methods do not scale
- Robust computational methods are needed to extract relevant features from huge data repositories
- Methods are needed for finding, fusing and visualizing highly complex data to make experimental inferences
- Highly sophisticated computer modeling is needed to enable adequate comparisons and drive potential hypotheses

## Information Discovery, Use, and Sharing

- Medical Imaging (Change analysis in Lung Cancer)
**Develop measurement infrastructure to improve methods of measuring change in lung cancer tumors **
- Computational Biology
**Computational science capability to support high throughput measurements; Focus on techniques to convert physical cell measurements into quantitative descriptors that can be analyzed statistically. **
- Core Infrastructure
**Apply automated test methods, coupled with combinatorial testing approaches, to XML-based languages used in manufacturing and electronics industry supply chains; investigating potential measurement science role in web-based exchanges (reg/rep, ontologies, soa’s, web services, XML, etc).**
- Shape Searching
**3D shape searching techniques for protein structures within the HIV AIDS Database and for 3D CAD parts in a supply chain.**** **** **
- Multimodal
**Develop and apply metrics and testing to advance the state-of-the-art in human language processing, including text, video, and speech**

**Notes:**
*Medical Imaging (Change analysis in Lung Cancer)  * Phantoms and standard protocols to address uncertainty in physical 
measurement (PL, hopefully MSEL to analyze material properties of phantoms))
  * New statistical model to quantify deltas in radiologist annotations
  * New volumetric approaches to determining tumor size; density-based 
approaches will be pursued as well
 FYI, some tidbits that might be useful if questios arise for the IDUS Program:
* Currently 54% of the program includes redirected staff
* Working with Ram to bring his bio-related staff under the IDUS umbrella
* Working with Steve Ray to transfer core IT research to IDUS (where it makes 
sense -- primarily testing and semantics of web-based exchanges).
What is the Problem?
Our ability to generate data has outstripped out ability to turn it into useful information.  Current approach of annotating with metadata does not scale – need improved methods to enable automatic capture, searching, extraction, fusing, and visualizing information.

Why is it Hard?
Massive amounts of data must be gathered, integrated and analyzed, including video, speech, text (unstructured, structured), images, over widely varying spatiotemporal scales
Computational methods to extract and compare relevant features from datasets are increasingly dependent on complex geometries and relationships
Move towards fusing and visualizing highly complex data to make experimental inferences
Complex models that combine both theoretic and experimental data are increasingly used to enable adequate comparisons and drive potential hypotheses

NIST Role
Enable Scientific / Knowledge discovery by developing and applying computational science techniques to measurement science problems
Develop and apply metrics and testing to advance the state-of-the-art in core underlying technology areas => text, video, speech, image-based processing
Collaborate with broader community to develop standard methods for acquiring, defining, securing, accessing, integrating and preserving information

## Medical Imaging:  Change Analysis in Lung Cancer

- Detecting and analyzing changes in lung tumors is useful in disease diagnosis and therapy evaluation for lung cancer patients.
- High growth rate => indicative of disease onset
- Exponential growth rate => cancer
- Change of tumor size over time => response, stable, progressive disease
** ****BUT,**** **change analysis is difficult due to the lack of quantitative measurements.

**Sources of Measurement Error**
- CT Scanner - not calibrated, changes in parameters effect tumor image
- Lesion - Size, texture (density variation), margins, complexity
- Patient - cardiac motion, respiration, involuntary movements, and overall health
**NIST Role**
- Address Scanner and patient issues through the development of phantoms and standard protocols with traceability to NIST
- Develop methods to effectively characterize tumors with known uncertainties
- Conduct round-robin evaluations of reconstruction and analysis algorithms
**Collaborators**
- NCI, FDA, RSNA, CRPF, GE, Siemens, Phillips, Cornell, Stanford

**Notes:**
Medical Imaging – with Cornell and FDA

Program Manager invited to participate as part of a Federal Panel at the Cancer Research Prevention Foundation Lung Cancer Workshop IV Application of High Resolution CT Imaging Data to Lung Cancer Drug Development:  Measuring Progress.

Program Manager Co-hosted with PL a follow-up meeting with Cornell and FDA.  Discussions included the development of physical phantoms to calibrate the medical imaging device to the patient, development of standard imaging protocols, and improved algorithmic methods to detect medically relevant change in tumor size.  Initial industry efforts at developing physical phantoms are rudimentary at best and might benefit from additional collaboration with MSEL. 

Initial discussions with PL, ITK/VTK, Tom Baer, OSA, Cornell, and FDA in the development of phantoms and standards to support the medical imaging research community.  Ultimate goal is to create a validated research network, linking image data from researchers across the globe. 

Change Analysis for Lung Cancer Research with NCI and FDA  (redirected staff = 4.1)

Established a collaboration with NCI and FDA based on NCI’s “RIDER” collection of 85,000 images of tumors in the DICOM image file exchange format.

IDUS research indicates that volumetric measures may be more helpful than diameter in determining changes in tumor size. Merged the annotation data with the DICOM images from RIDER to create a visual overlay of RIDER images with RECIST annotation.  RECIST data provides endpoints for a line across the largest diameter of tumor, and is currently used as the method for determining tumor size.  

Loaded RIDER annotation and image data (85,000 images) into a mySQL database and conducted a number of consistency checks.  Identified several apparent problems in RIDER data, including a failure to completely anonymize the data and inconsistent approaches to populating the DICOM metadata.  This data is in turn used for analysis, and can have dramatic effects on the image data itself.  Documented issues with RIDER data and will forward to NCI.  Extracted relevant subsets and distributed.

Initiated a sensitivity analysis of the metadata associated with the DICOM images in the RIDER database.  The ultimate goal is to establish which parameters have an effect on interpretation of the image slices. 

## Computational Biology:  Single Cell Analyses

**Computational Biology:  *****Single Cell Analyses***

- Intracellular molecular reactions and interactions that control the response and fate of cells and organisms cannot be unambiguously compared and combined due to a lack of standards and validated protocols.
** ****NIST Role**
- Provide the measurement tools and standards that enable quantifiable and reproducible measurements of cells and their interactions through:
**Standard data/metadata** format for image capture, storage, retrieval, analysis.
- Software to **enable high throughput cell image analysis and interoperability**.
- Standards and validation required to **ensure reproducible image analysis**.
- M**ulti-site experiments to test software and validation protocols**.
**Technical Approach**
- Create and evaluate an integrated data collection, organization and analysis infrastructure for cellular imaging.
- Experimentalists and computational scientists - focus on the physical standards and protocols for data collection, image processing and analysis, storage of data and metadata, and evaluation of results.

**Notes:**
Medical Imaging – with Cornell and FDA

Program Manager invited to participate as part of a Federal Panel at the Cancer Research Prevention Foundation Lung Cancer Workshop IV Application of High Resolution CT Imaging Data to Lung Cancer Drug Development:  Measuring Progress.

Program Manager Co-hosted with PL a follow-up meeting with Cornell and FDA.  Discussions included the development of physical phantoms to calibrate the medical imaging device to the patient, development of standard imaging protocols, and improved algorithmic methods to detect medically relevant change in tumor size.  Initial industry efforts at developing physical phantoms are rudimentary at best and might benefit from additional collaboration with MSEL. 

Initial discussions with PL, ITK/VTK, Tom Baer, OSA, Cornell, and FDA in the development of phantoms and standards to support the medical imaging research community.  Ultimate goal is to create a validated research network, linking image data from researchers across the globe. 

Change Analysis for Lung Cancer Research with NCI and FDA  (redirected staff = 4.1)

Established a collaboration with NCI and FDA based on NCI’s “RIDER” collection of 85,000 images of tumors in the DICOM image file exchange format.

IDUS research indicates that volumetric measures may be more helpful than diameter in determining changes in tumor size. Merged the annotation data with the DICOM images from RIDER to create a visual overlay of RIDER images with RECIST annotation.  RECIST data provides endpoints for a line across the largest diameter of tumor, and is currently used as the method for determining tumor size.  

Loaded RIDER annotation and image data (85,000 images) into a mySQL database and conducted a number of consistency checks.  Identified several apparent problems in RIDER data, including a failure to completely anonymize the data and inconsistent approaches to populating the DICOM metadata.  This data is in turn used for analysis, and can have dramatic effects on the image data itself.  Documented issues with RIDER data and will forward to NCI.  Extracted relevant subsets and distributed.

Initiated a sensitivity analysis of the metadata associated with the DICOM images in the RIDER database.  The ultimate goal is to establish which parameters have an effect on interpretation of the image slices. 

## CORE INFRASTRUCTURE

## Conformance Test Development

**W3C -Specification**

**Test Assertions**

**Semi-formal Assertions**

**Spec Processing**

**Test Development**

**Labor Intensive**

**Compute Intensive**

**Comprehensive, coherent tests**
**Common test description and reporting**
** ****Problems identified early in life cycle**
** ****Requirement for Recommendation**
** ****Better specifications, better products**

**NIST Tests **

**Improved W3C, Vendor QA**

## XML Technologies

- XML is critical enabling technology
**Family of standards for describing and manipulating information**
**Describes data in an intelligent format that can be exchanged over incompatible applications, systems, and output media **
- To achieve the promise of XML - need to ensure the usability, correctness and quality of XML technology standards and their implementation
- NIST Role
**Provide technical leadership and conformance guidance**
**Partner with industry to develop a set of conformance test suites **
**Provide technical direction regarding use of XML technologies to others (e.g., NIST researchers, gov’t agencies)**

## XML Testing

**Expected Outcomes**
- Lead efforts for all W3C efforts
- Test suites used by many, results in bug fixes to all XML implementations and specs
- Tests are used as part of the process – 2 implementations must pass tests to exit CR
- Test methodology influences others – open source Linux, OASIS, SourceForge, Apache
- Over 30,000 tests developed

**Collaborators/Customers**
- IT Industry:
- IBM, Sun, Microsoft, Netscape, X-Hive, Arbortext, Oracle, Lucent Technologies, DataDirect, AT&T, Software AG, BumbleBee
- W3C, OASIS, University of Edinburgh
- Vertical Industries – Health Care, automotive, aerospace, construction, financial, travel, etc
- Consumers of XML technologies

**Drivers/Motivation**
- XML software not working correctly – inaccurate data, no interoperability
- Need for conformance tests
-  early in the WG process
-   adaptable to spec changes
-   use to improve spec
- Accelerate the development of specifications and implementations

**Staffing (FTEs)**
*(FTE’s, includes students, GRs)*
- 1      XML Core
- 1      XML Schema
- 1      XML Query
- .5     DOM, XSLT, XSL-FO (maintenance)
**Exit Strategy**
- Successfully applied to applications
- Technology transfer enriches communities

**XML Testing**

## ebXML Conformance Testing

**Expected Outcomes**
- Uniform method for testing ebXML standards
- Test Framework and Suites used to verify conformance and interoperability
- Test Framework and Suites used by
- Regional IT consortia OASIS (US), EAN-USS and ETSI (Eur), KorBIT and ECOM (Asia)
- Test companies, ebXML vendors and users
- Demo’ed at Reliable Infrastructures for XML, ETSI Plugtest, OAG meeting

**Collaborators/Customers**
- OASIS, Sun Microsystems, Drake Certivo, Fujitsu America,  Cyclone Commerce, Adobe, KorBit
- Vertical industries
-  Automotive, Aerospace, Healthcare, Construction
-  OAG, HIMSS/IHE
- ebXML vendors and testers, small/medium businesses using ebXML

**Drivers/Motivation**
- Interoperability is key to Internet B2B, ebXML provides the foundation
- Need to consolidate disparate testing efforts for ebXML family of standards
- A globally recognized, open testing architecture and test suite promotes global ebXML interoperability through a standard set of testing tools
-  reduces development efforts and duplicative test suites and testing

**ebXML Conformance Testing**

**Staffing (FTEs)**
*(FTE’s GRs)*
- 2 (1 FTE, 1 GR)
**Exit Strategy**
- Successfully applied to applications
- Technology transfer enriches communities - spurs global usage and contribution to a common set of ebXML test suites

**Notes:**
ebXML standards define an open XML-based business infrastructure that enables enterprises of any size and in any geographic location to conduct business over the Internet
ETSI – European Telecommunications Standards
KorBIT – Korean eBusiness Testbed

## ebXML Conformance Testing 

- Led OASIS Registry TC
- Developed initial Spec and tests
- Lead Testing effort in OASIS Implementation, Interop, Conformance (IIC) TC
- Developed Test Framework Software and Specification
- Developed and published Messaging Test Suite
- Revising Test Framework Software and Specification/Implementation, extend use to other ebXML specs
- Collaborate with other ebXML TCs (message, registry, business process, collaboration Protocol Profile to use Test Framework to produce test suites
- Work with Registry TC, developing test requirements and core test suite

- Test Framework
- Components

- OASIS Technical
- Committees (TC)

- Regional Consortia,
- Vendors, Users

## Health Information Technology

**Expected Outcomes**
- Lead and participate in standards and  conformance efforts
- partners with industry standards groups
- chair conformance efforts
- consultant to federal agencies with healthcare missions
- Develop conformance tests, tools and prototypes
- based  on industry priorities
- prototypes fill in industry gaps
- Conformance definitions included in standards

**Collaborators/Customers**
- American Telemedicine Association (ATA)
- ANSI HC Informatics Standards Board (ANSI HISB)
- ASTM Committee E31 on HC Informatics
- Consolidated Health Informatics (CHI)
- Health Level Seven (HL7)
- IEEE Medical Device Communications Group
- Integrating the HC Environment (IHE)
- PITAC - President’s Information Technology Advisory Committee

**Drivers/Motivation**
- Fragmentation in our health care system
- Demand for online access to medical info and services
- Improve patient care, reduce medical errors and costs (98,000 deaths/yr)
- “build a standardized platform ...all can communicate electronically”
*	**	**(HHS Secretary Thompson, 2003)*
- Vision of an interconnected electronic health infrastructure

**Staffing **
*(FTE’s, includes students, GRs)*
- 5.5  Mess. Conformance, EHR, Infrast. Integration
- 2     IEEE Medical Device Comm
- 3     Standards Landscape
- .25  Telemedicine
**Exit Strategy**
- Tests, tools and approaches successfully applied
- Technology transfer enriches community:
-  continued use of products
-  ability to build upon products
-  rely on NIST for consultation

**Health Information Technology**

**Notes:**
ASTM International, originally known as the American Society for Testing and Materials (ASTM), was formed over a century ago, 

## Putting standards together for effective information exchange
seamless information flow from application to application across and between HC enterprises
comprehensive view of patient information results in quality care decisions
Define and build standard-based profiles to address specific problems
NIST Activities
work with HIMSS IHE to define, test, and implement integration profiles
develop measurements and tools to validate that approaches are technically and economically feasible 
Co-authored IHE Enterprise Cross Document Sharing standard (XDS)
Develop XDS reference implementation and test tool 
NIST components featured at HIMSS Showcases 

- Putting standards together for effective information exchange
**seamless information flow from application to application across and between HC enterprises**
**comprehensive view of patient information results in quality care decisions**
- Define and build standard-based profiles to address specific problems
- NIST Activities
**work with HIMSS IHE to define, test, and implement integration profiles**
**develop measurements and tools to validate that approaches are technically and economically feasible **
**Co-authored IHE Enterprise Cross Document Sharing standard (XDS)**
**Develop XDS reference implementation and test tool **
**NIST components featured at HIMSS Showcases **

**Infrastructure Integration**

- Integrated HC environment = HC standards + IT standards + glue to bridge gaps

**Notes:**
designing distributed models and architectures for healthcare information systems, developing reference implementations for these models and architectures, and participating in the development of standards for healthcare information systems 
Healthcare is a particularly challenging environment for information systems. Healthcare information systems must be distributed and interactively accessible, and must provide interoperability between new and legacy technologies. 

## Cross-Enterprise Document Sharing

- Enterprise

- Enterprise

- Enterprise

**Imaging Center**

**Hospital B**

**Hospital A**

**Emergency Room**

**Physician**

**Patient**

**Admin**

**Repository**

**Repository**

**Repository**

**Repository**

- Cross-Enterprise Document Registry
- (XDS)

**Cross-Enterprise Document Sharing**

## Standards Landscape

- Plethora of standards – even within a discipline (e.g., clinical standards)
**Who’s doing what? How are standards being used? Deployed?**
**monitoring the relevant standards is arduous and labor-intensive effort**** **
- Repository of information on HC standards, organizations, and initiatives.
**knowledge of who’s doing what**
**minimize overlap and duplication of standards development**
**facilitate collaborative standards work**
**foster use and adherence to standards**
- NIST Activities
**develop an interactive web prototype to demonstrate the concept**
**collaborate with partners such as ANSI HISB, eGov’s CHI, and AHRQ**
**develop a tool set that can be adopted and extended to enable widespread implementation**
**initiated pilot focused on HISB information  -- adapted for use in NHIN**** **

**Notes:**
Description of standards, type of use, maturity level, interrelationships, who’s doing what, etc.

## 3D SHAPE SEARCHING

## Research Challenge

**Research Challenge**

- Need shape descriptor that is:
**Discrimiminating**
**Quick to compute**
**Concise to store**
**Pose-independent **
**Efficient to match**

**Shape descriptor**

**3D CAD database**

- Nearest Neighbor

- Rank List

## Shape Searching structural bioinformatics

## MULTIMODAL ROUND-ROBINS

## TREC

- Workshop series for large-scale evaluation of (text) retrieval technology
**realistic test collections**
**uniform, appropriate scoring procedures**
**a forum for the exchange of research ideas**
- Started in 1992 as an evaluation vehicle for DARPA Tipster project
-  http://trec.nist.gov

## TREC Tracks

**TREC Tracks**

- Retrieval in a domain

- Ad Hoc, Robust

- Interactive, HARD

- X{X,Y,Z}
- Chinese
- Spanish

- Video
- Speech
- OCR

- Enterprise
- Terabyte
- Web
- VLC

- Novelty
- Q&A

- Filtering
- Routing

- Legal
- Genome

- Static text

- Streamed text

- Human-in-the-loop

- Beyond just English

- Beyond text

- Web searching, size

- Answers, not docs

- Blog
- Spam

- Personal documents

## NIST ASR Benchmark Test History – May. ’06

**1988    1989     1990     1991    1992      1993     1994     1995    1996     1997     1998     1999      2000    2001     2002     2003     2004     2005    2006     2007    2008     2009     2010    2011**

- 100%

- 10%

- 1%

- 4%

**Air Travel ****Planning Kiosk Speech**

- 2%

- Range of Human Error In Transcription

**Meeting - MDM**

**Meeting - IHM**

**Meeting - SDM**

**Meeting Speech**

**NIST ASR Benchmark Test History – May. ’06**

**Notes:**
We have focused the community for many years
as error rates approached human performance we
Increased task complexity moving towards more robust and useful systems
Increased the number of domains and test conditions 

## QUESTIONS?