# Presentation: 936113

## The Image Query (IQ) Project

**Daniel L. Rubin M.D., M.S.**
**Clinical Asst. Professor of Radiology**
**Research Scientist, Center for Biomedical Informatics Research**
**Stanford University**

**David S. Channin, MD**
**Associate Professor of Radiology**
**Northwestern University**

**Joel Saltz, MD, PhD Professor and Chair**
**Ashish Sharma, PhD**
**Tony C. Pan, MS**
**Department of Biomedical Informatics**
**The Ohio State University**

**Brenda Young, BA**
**American College of Radiology Imaging Network (ACRIN)**

## Image Query (IQ) Project

**Researchers need to be able to search caBIG resources in an intuitive way**
**Current caBIG infrastructure does not focus on query**
- Middleware provides access to images on the Grid
- AIM provides the *representation/storage* of image metadata
**Need to be able to explore (browse) and query (search) image metadata by many different criteria**

## Image Query Use Cases

**Retrieve particular types of image data from Cancer Centers using caGrid Services **
**Retrieve data on caGrid from XIP **
**Retrieve data on caGrid from a DICOM Workstation **

## What is an Image Annotation?

**Annotations are explanatory or descriptive information, generated by humans or machines, directly related to the content of a referenced image or images**
**Image annotations let us **capture information about the meaning of pixel information in images** such that similar meaning in other images can be found and used.**

## An Image, an Image Markup and an Annotationin AIM Schema

- The pixel at the tip of the arrow [coordinates (x,y)] inthis image
- [DICOM: 1.2.814.234543.23243]represents the Ascending Thoracic Aorta[SNOMED:A3310657]

**AIM provides the vehicle for accessing image data on the caGRID.**
**But a QUERY layer is needed... **

## Phase I activities to define scope of Phase II

## IP ProjectPhase I Goals

**Define **_**Use Cases**_** for image query**
- Requirements gathering: what groups currently seek image query?
**Survey: Gather **_**common questions/queries**_** for image data to establish information requirements**
- What information do users actually seek from image resources/databases?
- Compile types of questions people ask and kinds of data elements in databases relevant to search
- Review existing approaches to query processing in caBIG and related projects
_**Proof-of-concept demonstration**_** of query formulation/execution at RSNA 2007**

## IQ Project Progress #1:Use Cases

**Definition of Use Cases for Image Query**
- We have collected use cases related to
- Cooperative groups: RTOG, CALGB, and CVRG
- ACRIN
- caBIG/NCI initiatives (NCIA, caTRIP, caB2B)
- Abstracted outlines for imaging related aspects of  RTOG 0522 and CALGB 80302 cancer studies
- Next will define queries that one can formulate in the context of clinical trial activities
- Accruing patients
- Quality control
- Analyzing trials results

## IQ Project Assumptions and Issues

**Assumptions about prerequisites supporting infrastructure**
- All data is _*electronically accessible*_ (on the Grid)
- All data is _*publicly accessible*_
- Data in ACRIN image archive often not freely available to public; NCIA  data is freely available
- Image metadata is _*annotated in AIM*_
- PRE-AIM: Lots of historical image data that has no AIM annotations; cannot query with IQ?
- POST-AIM: Once AIM is available, images will be annotated with AIM.
**IQ  Phase II will need a strategy on how to handle historical and future data**** **

## IQ Project Progress #2:Common Queries and Questions

**Common queries/questions most researchers will want:**
*“**Get me all images in NCIA that are part of the RTOG 0522 study (or other study)”*
*“**Of those, find me all of those that have PET/CT”*
*“**Where are reports and measurements stored?”*
- Must deal with existing data and put that into AIM
- Going forward people will annotate RTOG studies and save in AIM server
*“**Find images that have no annotations; find annotations showing >20% growth in disease”*
*“**Get all images where AIM annotations show progression in disease”*

## Example Queries from RTOG

**The key needs for image query in the RTOG use case is identifying evidence on images for treatment response and to predict survival**
- “Retrieve *DVH* and *CT Images* and *RT Contours where Case*=0905 and *Protocol*=9999”
- “*Find  paired CT and PET/CT images and patient IDs where the number of involved nodes differs between CT and PET/CT image analyses*”
- Example query from CERR Matlab Client:  Query/Retrieve study as a MATLAB object using the following attributes:
- 	*archive*
- 	*tapeStandardNumber*
- 	*intercomparisonStandard*
- 	*institution*
- 	*dateCreated*
- 	*writer*
- 	*sponsorID: 'RTOG'*
- 	*protocolID: 'ASHI1234'*
- 	*subjectID: 222*
- 	*submissionID: 7*
- 	*timeSaved: {'3-Apr-2007 08:07:37'}*

## Example Queries from CVRG

**Quantitative measurements to detect regional wall motion abnormalities and correlate the cardiac morphologies with ECG and genetic data**
**Goal is to relate the cardiac morphology with molecular biomarkers to predict key phenotypes (inappropriate firing of ICD and sudden cardiac death). **
**Example queries: **
- Identify subjects for further analysis (identify inappropriate firing of ICD)
- “Find ECG, SNP, and Images for patients where ICD firing meets criteria X”
- “Find ECG and Images for patient who has SNP A”
- Identify high risk patients (identify risk factors for sudden cardiac death)
- “Find patients where ECG shows QT variability index greater than X, possess SNP A and C, has CT ejection fraction is below Y, and MR infarct size greater than Z”

## From Queries to QUESTIONS

**Ultimately, Image Query needs to satisfy higher-level **_**questions**_** that people have about clinical trials (a series of queries would need to be able to answer):**
*“**What is predictive value of PET/CT?”*
*“**Does pre-treatment PET correlate with (and predict) disease-free survival and lack of recurrence or particular surgical findings of the disease?”*
- To answer this, we need to query AIM annotations on PET images as well as _non-image clinical data_
*“**Do post-treatment PET findings correlate with (and predict) the nodal response and nodal relapse in patients?”*
- To answer that, we could simply query AIM annotations on PETs and CTs

## IQ Project Progress #3:Current Approaches to Query

**Survey of Current Approaches to Image Query**
- We surveyed existing **caBIG** efforts outside IVI-WS:
- No active work currently ongoing within caBIG Architecture WS
- We reviewed  query approach in ACRIN data warehouse project
- (See next slides)
- Provides GUI-based query mechanism for query formulation
- Could be useful idea to extend as interface to query images from federated Grid resources
- We review query approaches in Cooperative Groups (RTOG, CALBG, CVRG)
- We reviewed the query approach in NCIA and caB2B

## Type in question or key word and response desired

**Type in question or key word and response desired**

**Or by filtering through **
**predefined layers based**
**upon the structure of the **
**ACRIN data dictionary**
**or DICOM hierarchy**

**This is a GUI for composing queries in the ACRIN data warehouse to give us ideas how query interfaces should look in the IQ project**

## Production release will contain 

**Production release will contain **
**230 searchable DICOM fields**

**Based upon feedback from**
**researchers using the data **
**warehouse, more DICOM fields**
**can be added.**

**In ACRIN, queries are constrained using controlled terms and pick-lists**

## Responses for the chosen question then become available 

**Responses for the chosen question then become available**

**Query is built**

**In ACRIN, queries are composed with controlled terms and boolean operators**

## Query in NCIA

## Query in caB2B

## IQ Project Progress #3Working Prototype

**Demonstration of Working Prototype**
- Working prototype of query federation and execution components of the IQ Tool
- Simple GUI for user to construct a cross-domain query
- Targets an AIM data service and the grid-enabled ACRIN image warehouse (gACRIN)
- Retrieves the relevant image metadata and associated annotations
- Will use high-performance data transport, and leverage role-based authorization
- Will leverage the In Vivo Imaging Middleware and the caGrid federated query processing infrastructure

## Query Execution Engine

- Once a query is formulated as an ontology-based query graph, this query must be translated in such a way so that it can be executed on the diverse caBIG data sources.  This is done in the **Query Execution Engine**

## Query Language — DCQL

**DCQL — Distributed caGrid Query Language**
- Language used to express federated queries
- An extension to CQL (a uniform query language used by all caGrid data services)
**Specifies a target object (result) type and selects the instances which satisfy the specified properties and nested object properties**
- Allows path navigation
- Provides logical grouping
- Provides name/predicate/value filtering on properties of objects
**Recursively defined**
**Ability to return full Objects, Set of attributes, count of results, or distinct attribute values**
***For more details see: http://www.cagrid.org/mwiki/index.php?title=Federated_Query:DCQL***

## Federated Query Processor

**Provides a mechanism to perform basic distributed aggregations and joins of queries over multiple data services**
**Federated queries are expressed in DCQL**
**Since caGrid data services all use a uniform query language, CQL, the Federated Query Infrastructure can be used to express queries over any combination of caGrid data services**
**Implemented as a stateful grid service, queries may be executed asynchronously and results retrieved at a later time**
- Supports secure deployments wherein result ownership is enforced
**Coupled with semantic discovery capabilities of caGrid, provides a powerful framework for data discovery, mining, and integration**

## Data Service Overview

**caGrid Data Services provide capability to expose silver data resources to the Grid**
**Specialization of caGrid grid services to expose data through a common query interface**
- Meet all base service requirements of caGrid services
**Present an object view of data sources**
- Exposed objects are registered in caDSR and their XML representation in GME
- Data Service Metadata describes information model
- Queries made with CQL Query objects
- Results returned as objects nested in a CQL Query Result Set

## Federated Query Processing Example

**Sample queries**
***“******Find all DICOM images where the RECIST length criterion is greater than a certain value”***
***“******Find all AIM annotations related to a specified set of patients”***
**Leverage caGrid FQP infrastructure**
- Mechanisms to perform basic distributed aggregations and joins of queries over multiple data services
- Distributed caGrid Query Language (DCQL)
**Components**
- Federated query processing service, and the query engine
- Federated result service
- caGrid data services which provide the data
- DICOM data services using the NCIA DICOM data model
- AIM data services using the AIM data model

## FQP Sequence Diagram

**FQP Sequence Diagram**

## FQP Client

## AIM Data Service

**caGrid data service for storing AIM annotations**
**Annotations are stored as XML documents**
**Components**
- XML Backend (Berkeley DB XML database)
- Grid data service Wrapper
- CQL to XPath translation and query
**Introduce extension to build data services using XML schema**
- Can be generated from data model in XMI
- Can be generated from data model in caDSR and GME
- Can be linked to caDSR

## DICOM Data Service

- caGrid data service for storing DICOM image objects stored in a PACS
- Supports DICOM C_FIND, C_GET, C_MOVE, and C_STORE commands
- Leverages caGrid for service discovery, remote access, and security (authentication and authorization)
- Provides role based data-access control

- Use GridFTP for high performance bulk data transfer
- Provide bulk download or progressive download
- GridFTP security mechanism has been harmonized with the caGrid security mechanism to enforce requester-only download of data
- Only the requester can access data retrieved as a result of a query
- Requester can only access data marked as available to the user, not all data on the server

- Without BDT
-      34MB in ~300 sec
- With BDT
-       4GB in ~350 sec

## Requirements for Phase II

## IQ Project Phase II

**The actual work of creating the Image Query tools and infrastructure**
**Application ontology**
- Represent kinds of information that users seek and granular data fields actually contained in various image-related databases
**User interface**
- Allow users to select data elements and data element values and combine them with Boolean operators
**Query engine**
- Execute the query that is formulated by the UI and application ontology

## IQ Project: System Architecture

## Application Ontology

**Encode terminological knowledge**
- _Synonymy and polysemy_
- E.g., if a user enters a search term “hepatic carcinoma,” then synonymous search terms such as “heptocellular carcinoma” and “hepatoma” would be included in the search to increase coverage
- _Hierarchical relations _to broaden the search or to narrow it by searching on parents or children of search terms
- E.g., if a user searched for “pheochromocytoma,” the search could be expanded to include terms such as “adrenal medullary neoplasms” if a broader search is desired
**Use ontologies for creating user queries**
- Match user information needs to the ontology terms used to annotate caBIGTM image resources

## User Interface

**Most users know little about ontologies (nor should they **_***need***_** know anything about them!) **
**The UI must be intuitive and easy to use.  **
**UI core functional requirement is to enables users to:**
- Browse image resources to get a sense for the types of data they contain
- Search image resources for specific types of images or image results
**UI will provide a seamless mechanism to map from a user’s intuitive queries and ontology terms that are used to annotate images (AIM project)**
**UI may incorporate components from existing tools that has cross-domain query capability, such as caTRIP**

## Query Engine

**Query Formulation Engine will translate user queries that are formulated using the IQ Tool UI into an ontology-based query graph**

## Types of Queries to be Supported

**Exact term query**:  Find images and data exactly matching the query
**Concept-based query**:  Map the query ontology terms and search for those concepts
**Query expansion by synonym**:  Expand query with synonyms
**Query expansion by abstraction**:  Expand user query with *parents* of the search terms (i.e., hypernym searching)
**Query expansion by progressive refinement**: Scope the query to *children* of the search terms to refine it (i.e., hyponym searching)
**Boolean query**:  Combine multiple search terms with Boolean operators AND, OR and NOT
**Query by example**:  Search  based on an *example*
**Correlative query**:  Build and execute queries using terms from multiple ontologies and on services that conform to different data domains and models

## Semantic Query Execution

**Ontology-based query graph captures semantics of user query**
**caGRID services require low-level syntax-based query (e.g., CQL query language)**
**A semantic translation ****layer will be added to ****caGrid middleware to ****translate query graph ****to CQL**

## Query Execution Engine

**Query Validation**: Query graph is analyzed for validity in the caBIGTM environment
**Query Translation**:  Query graph is converted into a query statement in DCQL form
**Query Planning and Optimization**:  The DCQL query statement is simplified and optimized based on the data models that have been identified to be involved in the query
**Query Processing**:  The optimized DCQL query statement is executed either using a local caGrid Federated Query engine or a remote caGrid Federated Query Service
**Result Aggregation**:  The result of the query from multiple data services is aggregated into an efficient storage structure and returned to the user

## Acknowledgements and Support

**caBIG Imaging Workspace, Subcontract from Booz-Allen & Hamilton, Inc. **

## Stay tuned for Phase II !

**Contact info:**
**rubin@med.stanford.edu**