# Presentation: 866180

## caBIG Architecture Workspace: Common Query Language SIG

- Georgetown, DC
- August 16th - 18th, 2005

**caBIG Architecture Workspace: Common Query Language SIG**
- Review of Recommendations and Steps towards Agreement

- Scott Oster
**SIG Lead**

## Agenda

- Overview
- Current SIG Recommendations Review
- caGrid 0.5 Query Language Review
- Example Query
- Other SIG Interactions
- Next Steps

## Query SIG Mission

*	**“**To come to a consensus on the requirements, properties, and details of the language which will be used to query caBIG grid resources, and also to define the requirements of a query engine capable of performing the caBIG query use cases, and create/identify a query language which meets these criteria.”*

## State of the Grid

- Common Query Language SIG Initial Recommendations Presented March 2005
- caGrid 0.5 Released August 2005
- Several reference implementations in place, using a common query language
- caBIO
- caArray
- Any caCORE SDK generated data model
- Post August 2005, many more reference implementations will join the grid
- Using the wealth of data providers that will soon be available an objective comparison of the two approaches can soon be made using real caBIG data
- Best practices recommendation will follow

## Current Approach Recommendations

- The common query language will be the “minimum entry point” for caBIG data services.
- Distributed Query Services and other higher-level services which interact with data services should do so through this language
- This does not preclude data services from implementing other, more specialized, languages
- They just must also provide an implementation of the common query language
- Data Services should be classified in terms of their support of the common query language
- e.g. Tier 1, Tier 2, Tier 3

## Current Language Recommendations

- W3C XML Query (now a W3C Working Draft! April 05)
- “The mission of the XML Query project is to provide flexible query facilities to extract data from **real and virtual documents** on the World Wide Web, thereby **finally** **providing the needed interaction between the Web world and the database world**. Ultimately, collections of XML files will be accessed like databases. The ambitious task of the XML Query (XQuery) Working Group is to develop the first world standard for querying Web documents...”
- XQuery _[http://www.w3.org/XML/Query](http://www.w3.org/XML/Query)_
- Use Cases: _[http://www.w3.org/TR/xquery-use-cases/](http://www.w3.org/TR/xquery-use-cases/)_
- Requirements: _[http://www.w3.org/TR/xquery-requirements/](http://www.w3.org/TR/xquery-requirements/)_
- XPath _[http://www.w3.org/TR/xpath20/](http://www.w3.org/TR/xpath20/)_
- Full Text _[http://www.w3.org/TR/xquery-full-text/](http://www.w3.org/TR/xquery-full-text/)_
- XUpdate (*not W3C*) _[http://xmldb-org.sourceforge.net/xupdate](http://xmldb-org.sourceforge.net/xupdate/)__[/](http://xmldb-org.sourceforge.net/xupdate/)_

## XQuery – XML Query Language

- XQuery is to XML what SQL is to databases; defined by the W3C
- Built on XPath expressions
- XPath is a syntax for defining parts of an XML document: Path expressions, Axes and Node Tests, and Predicates
- XPath defines a library of standard functions: Node Set Functions, String Functions, Number Functions, Boolean Functions
- XPath supports numerical, equality, relational, and Boolean expressions.
- FLWR (pronounced "flower") expression is the analogue of the SELECT-FROM-WHERE construction in SQL:
**FOR**-clause: binds one or more variables to a sequence of values returned by another expression (usually a path expression) and iterates over the values.
**LET**-clause: also binds one or more variables but without iterating.
**WHERE**-clause: contains one or more predicates that filters or limits the set of nodes as generated by the FOR/LET-clauses.
**RETURN**-clause: generates the output of the FLWR expression. The RETURN-clause usually contains one or more element constructors and/or references to variables and is executed once for each node-reference that is returned by the FOR/LET/WHERE-clauses.
- Example Basic Query:
- FOR $b IN collection(“bib”)/bib/book
- 	WHERE $b/publisher/text() = "Addison-Wesley" AND $b/@year = "1994“
- RETURN $b/title

## Preliminary Language Levels Recommendations

**Tier 3 Support**
- All of Tier 2
- Full XQuery
**Tier 2 Support**
- All of Tier 1
- Full XPath, Limited XQuery (basic FLWR support)
**Tier 1 Level Support**
- Limited XPath support with simple predicates
**Optional Add-ons**
- Full-Text
- XUpdate

## caGrid 0.5 Query Language

- Specifies a target object (result) type and selects the instances which satisfy the specified properties and nested object properties
- Allows path navigation
- Provides logical grouping
- Provides name/predicate/value filtering on properties of objects
- Recursively defined

## State of Query in the Grid

- Current Data Services can fairly easily support Object Traversal with filtering
- Think of the data model as a graph to be walked, testing properties of the objects along the way
- Query SIG Tier 1 (XPath) supports this construct
- caGrid Query Language embodies this construct

## Example Query over a Subset of the caBIO Model

- “Give me the *Proteins* that have *Genes* which have *NucleicAcidSequences* with *id* equal to *1507*”

## Example Query as Object Graph Traversal

**Example Query as Object Graph Traversal**

- “Give me the *Proteins* that have *Genes* which have *NucleicAcidSequences* with *id* equal to *1507*”

- For every *Protein*, walk to all of its *Genes* contained in *geneCollection*

## Example Query as Object Graph Traversal

**Example Query as Object Graph Traversal**

- “Give me the *Proteins* that have *Genes* which have *NucleicAcidSequences* with *id* equal to *1507*”

- For each of those *Genes* walk to its *NucleicAcidSequences* contained in *NucleicAcidSequenceCollection*

## Example Query as Object Graph Traversal

**Example Query as Object Graph Traversal**

- “Give me the *Proteins* that have *Genes* which have *NucleicAcidSequences* with *id* equal to *1507*”

- For each of those *NucleicAcidSequences*  keep those which have an *id* value equal to *1507*

## Example Query as Object Graph Traversal

**Example Query as Object Graph Traversal**

- “Give me the *Proteins* that have *Genes* which have *NucleicAcidSequences* with *id* equal to *1507*”

- Return the *Proteins* which satisfied this graph traversal

## Example Query expressed in the caGrid Language

## Example Query Expressed in Tier 1 Language

- Using **class** based traversal (as in caGrid):
- /Protein[Gene/NucleicAcidSequence/@id="1507"]
- Using **identifier** based traversal:
- /Protein[geneCollection/nucleicAcidSequenceCollection/@id="1507"]
- Open Questions (applies to caGrid language as well):
- Which of the above is appropriate?
- Is there a hybrid?

## Discussion Items with Identifiers SIG

- How will caching of data locations, by the resolution engine, effect query engines that create and want to use data object caching?
- Should there be an identifier data type (CDE)?
- Should there be a resolve(id) operator in the query language, and what are the ramifications?
- An LSID is sort of a *void ** pointer, how can we determine the data type it points to?
- The existence of multiple ids for a given data object creates a problem for the query engine doing join optimization by using just ids.

**Notes:**
Identifier SIG:
How will caching of data locations, by the resolution engine, effect query engines that create and want to use data object caching?
If a data object is cached by the query engine, and has resolved an id pointing to the source, will it be able to use this cache when it resolves the id?
If a query engine caches a data object, does it need notify a part of the resolution framework so it can be "discovered" as a source for that id
Should there be an identifier data type (CDE)?  If so, is the SIG providing a recommendation for it?
Should there be a resolve(id) operator in the query language, and what are the ramifications?
An LSID is sort of a void * pointer, how can we determine the data type it points to?  This may be important for the query engine, and is required if we support resolution in the language (as above).
Should we recommend a namespace or CDE id metadata attribute?
Should we recommend the data service can be returned for an id and require data services support this ID to type information?
The existence of multiple ids for a given data object creates a problem for the query engine doing join optimization by using just ids.
maybe need to have a flag in the query language to specify "value only" joins
or require a "cannonical" id to be used by all data services for these objects (difficult to enforce)
or require caBIG only allows one id for a given data (impossible to enforce)
The query engine will probably need batch id resolution
What parameters could be supplied to give hints in the presence of redundant copies?
Sometimes it is appropriate to utilize as many sources as possible; other times it is appropriate to minimize the number of sources as possible.  
How to identify materialized or partially materialized query results (such that the query may be re-executed to be resolved and thus would not be byte identical)?
We could possibly use the versionless retrieval (gets the latest version) and update the version each time the query is executed.
Versionless retrieval may be dangerous (same (versionless) id won’t always returns the same data.  The identifier SIG may want to explicitly provide a "give me the latest" operator instead.

## Discussion Items with Workflow SIG

- How could (or should) the query language support invoking an analytical service (or even a whole workflow)?
- How could workflow invoke queries?
- Embed the query in the workflow specification
- Reference a "stored query" or a service which executes a query given some parameters.

## Discussion Items with Security SIG

- How to propagate permissions with data as it moves around the grid?
- query results
- cached data
- user’s local copy of results

## Next Steps

- Iron out the details of both languages (such as the class/identifier issue)
- Create a plan to evaluate the languages against real query scenarios against real data provided by real data services
- Revisit SIG’s requirements in this context
- Create a best practices document describing the strengths and weakness of both approaches
- Make a recommendation for use of these languages
- Full adoption of one versus the other?
- Coexistence?
- If so, when should one be used over the other?
- Is one a base line requirement and the other an option?

## Acknowledgements

- Architecture Workspace Common Query Language SIG Participants
- caGrid Team

## caBIG Architecture Workspace: Common Query Language SIG

- Georgetown, DC
- August 16th - 18th, 2005

**caBIG Architecture Workspace: Common Query Language SIG**
- Review of Recommendations and Steps towards Agreement

- Scott Oster
**SIG Lead**

## 

Additional Information
(not presented)

- Additional Information
- (not presented)

## Requirements Matrix

| Requirement Abbreviation | Functional / 
Non-functional | Must / Should 
/ May |
| --- | --- | --- |
| expression | Functional | Must |
| structured data query | Functional | Must |
| full text search | Functional | Should |
| semantic query | Functional | Must |
| semantic reasoning | Functional | Should |
| data creation/update | Functional | Should |
| core operations | Functional | Must |
| operation extension | Functional | Should |
| persistence | Functional | Should |
| XML expression | Non-Functional | Must |
| XML results | Non-Functional | Must |
| result set expression | Non-Functional | Should |
| metadata consistency | Non-Functional | Must |
| implementation independent | Non-Functional | Must |
| security | Non-Functional | May |
| security of results | Non-Functional | May |
| standards-based | Non-Functional | Should |
| open implementation | Non-Functional | Should |

## Functional Requirements

- Expression
- must allow the client to express complex queries easily and consistently
- Structured data query
- must minimally provide users and services the ability to query structured data correctly and completely.
- Full text search
- should provide a full text query capability correctly and completely.
- Semantic query
- must provide a capability to query semantic information correctly and completely.
- Semantic reasoning
- should provide a capability to perform semantic reasoning, and express queries using semantic inferences.
- Data creation / update
- should support the creation and modification of datasets.
- Core operations
- must provide a moderate set of core operations.
- Operation extension
- should support extensions to the core set of operations and transformations.
- Persistence
- should support the concept of persistent queries.

## Non-Functional Requirements

- XML expression
- must be expressible in XML
- XML results
- must be consistent with results sets being in XML.
- Result set expression
- should describe how result sets will be structured.
- Metadata consistency
- must be straightforward for users to construct queries over data sets from examining an information model representation of the dataset.
- Implementation independent
- must be data resource agnostic, and not have any dependencies on a particular underlying storage technology.
- Security
- may need to have security-related constructs in it.
- Security of results
- the language may need to formalize the security requirements of result sets.
- Standards-based
- the language should follow a community accepted standard.
- Open implementation
- the language should have a freely available query engine implementation.

## Languages vs. Requirements 

**Languages vs. Requirements **

|  |  | Technologies |  |  |  |
| --- | --- | --- | --- | --- | --- |
|   | Priorities | SQL* | OQL** | XQuery*** | caGRID Phase 1 |
| Functional Requirements |   |   |   |   |   |
| expression | MUST | A | A | A | A |
| structured data query | MUST | A | A | A | C |
| semantic query | MUST | C | C | C | B |
| core operations | MUST | A | A | A | A |
| full text search | SHOULD | B | B | B | C |
| semantic reasoning | SHOULD | C | C | C | B |
| data creation/update | SHOULD | A | A | A | C |
| operation extension | SHOULD | B | B | A | B |
| persistence | SHOULD | A | A | A | A |
| Non-functional Requirements |   |   |   |   |   |
| XML expression | MUST | C | C | A | A |
| XML results | MUST | B | A | A | A |
| metadata consistency | MUST | C | A | A | A |
| implementation independent | MUST | A | A | A | B |
| result set expression | SHOULD | B | A | A | A |
| standards-based | SHOULD | A | A | A | D |
| open implementation | SHOULD | A | B | A | A |
| security of results | MAY | C | C | C | C |
| security | MAY | C | C | C | C |
| Capability Score Totals (A / B / C / D) |  | 8 /4 /6 /0 | 10 /3 /5 /0 | 13 /1 /4 /0 | 8 /4 /5 /1 |

| Technology Capability Legend |  |
| --- | --- |
| A | Fully Supported |
| B | Limited Support |
| C | No Support, but could be added |
| D | Not Supported |

| Requirement Priority Legend |  |
| --- | --- |
| MUST | Any suitable implementation
should satisfy it. |
| SHOULD | Might be valid reasons not
to support, but should be
justified. |
| MAY | Deserves attention, but is
currently unclear if its
required. |

**NOTE:** In this phase, distinctions between versions of languages are not considered.

**Indicates non-consensus or indecision**

- * = Family of SQL-based languages
- **= OQL-like languages (OQL,HQL,etc)
- ***= Family of XML Query languages
-         (XQuery,XPath,XUpdate)

## XQuery Example Description

- Give me all the expression data where there are at least 50 conditions for genes found in the vacuole.

| Gene |
| --- |
| goAcc |
| gbAcc |

| Microarray |
| --- |
| data/geneId |
| data/condition |
| data |
| subject/lsid |
| subject/species |

| GeneOntology |
| --- |
| term |
| acc |

- “vacuole”

- count > 50

## XQuery Example

- FOR $gene IN service(“http://cabio.osu.edu/GeneService.wsdl”)/Gene,$go IN service(“http://cabio.osu.edu/GeneOntologyService.wsdl”)/GeneOntology,$microarray IN service(“http://caarray.duke.edu/caArrayService.wsdl”)/Microarray
- LET $subject := $microarray/experiment/subject
- WHERE$go/term=‘vacuole’AND $gene/goAcc=$go/accAND $gene/gbAcc=$microarray/data/geneIdAND count($microarray/data[geneId=$gene/$gbAcc]/condition)>50
- RETURN<subject>  <subjectId>{ $subject/lsid }</subjectId>  <species>{ $subject/species }</species>  <microarrayData>    { $microarray/data }  </microarrayData></subject>

**Join across objects**

**Simple XPath**

**Complex XPath**

**Result composition**

## System Overview

## Towards a Semantic Grid

- XML provides syntax transport layer
- RDF(S) provides basic relational language and simple ontological primitives
- the meaning of Grid services, resources and entities by assertions in a common data model
- OWL DL provides powerful but still decidable ontology language
- publish and share consensually agreed upon ontologies
- Further layers may (will) extend OWL
- Query, filter, integrate and aggregate the metadata
- Reason over metadata

## Query Processing Models

## OGSA-DAI Distributed Query Processor (DQP) Architecture

- Currently works with OQL and targets only relational grid services
- May ultimately support XML data services
- May (probably will) still use OQL as client language, but will use XPath to query data services
- Timeline for XML may not work for us, but it is a useful reference implementation.
- Code may not work for XML, but architecture should