# Presentation: 936112

## Federated Query Break-Out Session

***Ian Fore, NCICBIIT***
***David Ervin, Ohio State University***
***Michael Keller, Booz Allen Hamilton***
***TBPT Face-to-Face Meeting***
***San Francisco, CA***

## Data Services Overview

**Part 1:  Introduction (30 min)**** **
- CQL
- Current query capabilities provided by caGrid
**Part 2:  Team Assignments  (2 hours)**
- Develop use cases for federated queries on caGrid
- Detail specific queries to be performed over specific services
- Categorize these use cases by:
- Queries that can be currently performed on caGrid
- Queries that are difficult to currently perform on caGrid
- Queries that can not be currently performed on caGrid
**Part 3:  Team Briefings and Summary  (30 min)**

## Data Services Overview

**Specialization of caGrid grid services to expose data through a common query interface**
**Present an object view of data sources**
**Exposed objects are registered in caDSR and their XML representation in GME**
**Queries made with CQL Query objects**
**Results returned as objects (or identifiers) nested in a CQL Query Result Set**

## Data Services Overview (metadata)

**Standard metadata exposed by all Data Services describes the UML model being exposed, and contains links to semantic underpinnings**
- Expresses available targets, attributes, and associations between data types
**Provides information needed to formulate the Object-Oriented Query (CQL)**

## Data Services Overview (CQL)

**Specifies a target object (result) type and selects the instances which satisfy the specified properties and nested object properties**
**Allows path navigation**
**Provides logical grouping**
**Provides name/predicate/value filtering on properties of objects**
**Recursively defined**

## Data Services CQL Example

| Return all Genes with a symbol beginning with BRCA and have an associated Taxon with an id equal to 6: |
| --- |
|  |

## Data Services CQL Example

| Return all Genes with a symbol beginning with BRCA and have an associated Taxon with an id equal to 6: |
| --- |
| <CQLQuery xmlns="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLQuery">
  <Target name="gov.nih.nci.cabio.domain.Gene">

  </Target>
</CQLQuery> |

## Data Services CQL Example

| Return all Genes with a symbol beginning with BRCA and have an associated Taxon with an id equal to 6: |
| --- |
| <CQLQuery xmlns="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLQuery">
  <Target name="gov.nih.nci.cabio.domain.Gene">

      <Attribute name="symbol" predicate="LIKE“ value="BRCA%"/>

  </Target>
</CQLQuery> |

- LIKE “BRCA%”

## Data Services CQL Example

| Return all Genes with a symbol beginning with BRCA and have an associated Taxon with an id equal to 6: |
| --- |
| <CQLQuery xmlns="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLQuery">
  <Target name="gov.nih.nci.cabio.domain.Gene">
    <Group logicRelation="AND">
      <Attribute name="symbol" predicate="LIKE“ value="BRCA%"/>
      <Association roleName="taxon“  name="gov.nih.nci.cabio.domain.Taxon">

      </Association>
    </Group>
  </Target>
</CQLQuery> |

- LIKE “BRCA%”

## Data Services CQL Example

| Return all Genes with a symbol beginning with BRCA and have an associated Taxon with an id equal to 6: |
| --- |
| <CQLQuery xmlns="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLQuery">
  <Target name="gov.nih.nci.cabio.domain.Gene">
    <Group logicRelation="AND">
      <Attribute name="symbol" predicate="LIKE“ value="BRCA%"/>
      <Association roleName="taxon“  name="gov.nih.nci.cabio.domain.Taxon">
        <Attribute name="id" predicate=“EQUAL“ value="6"/>
      </Association>
    </Group>
  </Target>
</CQLQuery> |

- LIKE “BRCA%”

- = 6

## Data Service Creation

**Data services can be built via an Introduce Toolkit extension**
**Extension adds functionality to both service creation and modification**
- Creation adds query operation
- Modification adds CQL implementation, selected data model, and appropriate metadata
- Specialized client configured to support necessary custom serialization and security settings
**Generic Data Service Client can be used to communicate with ANY data service**, but a service-specific client knows what security the service is configured with, and any non-default serialization it may make use of

## Data Service Creation (styles)

**Data Service creation can be further specialized with styles.**
- Eg. caCORE SDK 3.2 style
**Allows developers to add a creation wizard.**
**Further modification to Introduce service model**
- Add methods and types
- Modify code as required
- Configure data service internals

## Data Service Querying

**Primary query method**
- Required for all data services
- Takes single CQL query parameter
- Returns CQL Query Results instance
**Alternate results delivery**
- WS-Enumeration
- Returns enumeration context
- Allows client to consume results as it is able
- caGrid Bulk Data Transport (BDT)
- Returns BDT Reference
- Supports WS-Enumeration, WS-Transfer, and GridFTP

## Federated Query

**Distributed aggregations**
- Broadcast queries to multiple services
- Identical data types on each service
**Distributed joins**
- Disparate data types on each service

## Federated Query Language (DCQL)

**DCQL derives from CQL**
- Hierarchal approach
- Recursively defined
**Data Model drives the query**
- Underlying data service domain model used
**CQL is context dependent**
- Depend on data model of target service
**DCQL identifies context**
- Identifies service each part of the query targets

## Federated Query Language (DCQL)

**Foreign Association**
- Describes a relationship between the containing object and an object on another data service
- Processing results in a new CQL subquery directed to the target data service
**Join Condition**
- Specifies attribute relationship
*Local *and *remote* attribute names
- Predicate (EQUAL, LESS_THAN, etc)

## Federated Query Engine

**Requires no direct support from Data Services**
**Joining offloaded to FQP service**
**Engine may be reused in a client application**
- Simplicity over efficiency
- Portability over optimization

## Federated Query Engine

**Three Core components**
***Federated Query Engine***
- Client facing API accepts DCQL
- Orchestrates use of other components
***Federated Query Processor***
- Traverses the query, executes engine logic, handles intermediate aggregations
***Data Service Executor***
- Invokes standard CQL against data services

## Federated Query Processor (Functionality)

**Converts DCQL to standard CQL**
- Traverses query recursively
**Foreign Associations are resolved**
- Processed against target data service
- Utilizes CQL *Query Modifier* to obtain attribute values from target
- Produces CQL *Group* based on result values

## Federated Query Client

**Three ways to invoke**
***Execute***
- Blocks until processing is complete
- Results maintain information about source
***executeAndAggregateResults***
- Blocks until processing is complete
- Standard CQL Query Results
***executeAsynchronously***
- Creates resource to handle processing

## Federated Query Security

**May be deployed securely (TLS)**
- Asynchronous processing uses resource security – Creator only, single gridmap entry
**Security configured with Introduce**
- Standard Introduce service
- Methods / service may be secured via Introduce
**No delegated credentials**
- Remote data services must trust FQP service

## Part 2:  Develop use cases for federated queries on caGrid

**These use cases should include:**
- The primary stakeholders (i.e. who is performing the queries)
- Specific queries to be performed
- Specific services these queries will be performed across
**The use cases should be categorized by:**
- Queries that can be currently performed on caGrid
- Queries that are difficult to currently perform on caGrid
- Queries that can not be currently performed on caGrid
**Highlight potential issues impeding the execution of the proposed federated query use cases.**
- Bandwidth issues?
- Response times?
- Others?

## Part 3:  Team Briefings and Summary

**Summary of use cases developed**
- Queries that can be currently performed on caGrid
- Queries that are difficult to currently perform on caGrid
- Queries that can not be currently performed on caGrid
**Summary of issues identified **
- Description of issue
- Next steps/action items to begin addressing issues