# Presentation: 401178

## DataFoundry: An Approach to Scientific Data Integration

**Terence Critchlow**
**Ron Musick  ****	****	****      Ida Lozares**
***Center for Applied Scientific Computing***
**Tom Slezak****	****	****Krzystof Fidelis**
***Biology and Biotechnology Research Program***
***Lawrence Livermore National Laboratory***
**IBC Bioinformatics**
**October 1, 1999**

**Notes:**
Good morning.

It is my great pleasure to be here today to tell you about our work on the DataFoundry project at Lawrence Livermore.  DataFoundry is an ongoing CS research project aimed at improving the scientist’s access to data.  In particular, we have been working closely with local genomics researchers to provide better access to data of interest to them.

## Outline

**Motivation**
** ****DataFoundry’s integration strategy**
**Improving user interfaces**
**Beyond fully integrated data**
**Conclusions  **

**Notes:**
During this talk, I will present a brief overview of the DataFoundry project - where we started, what we have accomplished, where we are now, and where we are going.  

To start with, I will describe our initial motivating problem - integrating  heterogeneous genomics data sources - and our approach to solving it.  Then I will focus on our current efforts to improve scientists interaction with data through better user interfaces.  Finally, I will talk about a hybrid approach to data access we are proposing to develop over the next year or so.

The underlying theme through all of this will be our utilization of meta-data to declaratively represent the knowledge required to perform these tasks.

## Current environment

**Current environment**

**Data is**
**Hard to find**
**Hard to understand**
**Hard to reconcile**
**Hard to analyze**

**Scientists waste time and energy doing data management.**

**SCoP**

**SWISS-PROT**

**PDB**

**Notes:**
The problems in the current environment are plentiful.  If a scientist wants to answer a reasonably simple query, for example the one shown here, they must access several data sources individually.  For each one, they need to explicitly go to the source,  ask the pertinent subset of the query to the source, and obtain the information they require. Once they have the data from each site, they must identify the related instances and  convert the results to a consistent data format.  Then, and only then, can they start to answer their question.

In other words, they are wasting a lot of time and energy doing data management tasks instead of scientific research.  Obviously this is not productive.

## What is our ideal environment?

**Businesses use data warehouses****to accomplish this.**

**A *****single***** location that provides *****effective***** access to a *****consistent***** view of data from *****many***** sources through an *****intuitive***** *****and***** *****useful***** interface.**

**Notes:**
Ideally what we would like to have is a single location that provides intuitive and reliable access to a consistent view of data from multiple data sources.  

Now, by consistent view, what I mean is that the data has been both syntactically and semantically integrated. For example, syntactically, all protein sequences are represented in the same format no matter what their original representation was, and semantically, what is  meant by a chain is well defined and consistent across data from multiple sources.

What this would do is to hide  the data management tasks behind that single point of access, and allow users to concentrate on answering their questions.

Now, this challenge is not unique to genomics, or even to scientific data in general.  Businesses encountered this problem several years ago, and came up with data warehouses as a solution to the problem in their domain.

So,

## Data warehouses

**Data warehouses**

**Wrapper**

**Mediator**

**Wrapper**

**dbEST**

**A data warehouse is a repository that provides a single access point to a collection of data obtained from a set of distributed, heterogeneous sources.**

**Notes:**
What is a data warehouse?

Quite simply, a data warehouse is a repository providing a single point of access to a collection of data obtained from a set of distributed, heterogeneous data sources.

## Data warehouses

**provide intuitive access to the data**
**possibly change data format to meet user expectations**
**Warehouse**
**stores a consistent view of data in a local repository**
**Mediator**
**transform data from source format to warehouse format**
**Wrappers**
**read data from source into****internal representation**

**Data warehouses**

**Wrapper**

**Mediator**

**Wrapper**

**dbEST**

**Notes:**
A mediated data warehouse, the most common type of warehouse, has 4 layers.

The wrappers interact directly with the data sources, to obtain data from them and read it into an internal representation.    The mediators are responsible for transforming the data from the source representation to the warehouse representation and entering it into the warehouse.  The warehouse itself is simply a large data store that holds the information in a consistent format.  Finally, interfaces are provided to query the warehouse and return results to the users.  

## Warehouses don’t work in dynamic domains

**When schemata are modified, ****or new sources are added, **** wrappers and mediators break.**

**Wrapper**

**Mediator**

**Wrapper**

**Swiss****Prot**

**dbEST**

**SCoP**

**PDB**

**Notes:**
It sounds like this is perfect, so why aren’t we using data warehouses in genomics?

The problem is that they break when things change - and in dynamic domains like genomics, things change a lot.  This makes maintaining a warehouse in this type of environment prohibitively expensive.  Consider what happens if a previously integrated source like PDB changes its data representation. First, the wrapper needs to be updated to read the new format, then the mediator has to be able to convert it into the warehouse representation.  Furthermore, if there is new information being represented,  as would be the case if we were integrating a new source, the warehouse must be updated to include it, and the interfaces changed to pass it on.

Overall, this is a lot of work when sources are changing their representation every year (or less) and and users constantly want access to data from more sources.

## Key insight

**Key insight**

**API**

**Extensive use of meta-data can dramatically reduce maintenance costs.**

**Wrapper**

**Mediator**

**Wrapper**

**Swiss****Prot**

**dbEST**

**SCoP**

**PDB**

**Notes:**
Since warehousing technology appears to meet our needs, and would be useful if the maintenance cost were lower, we looked into ways of reducing that cost. We found that much of the maintenance costs involve creating and updating the mediators and wrappers. And we discovered that extensive use of meta-data can reduce these costs, making warehousing feasible in scientific domains.

## The DataFoundry approach:

**A**
**P**
**I**

**Sources**

**Warehouse**

**Wrapper**

**Wrapper**

**Wrapper**

**Meta-Data**

**Notes:**
In particular, we use meta-data to automatically generate the mediators and an API. This reduces costs in two ways.  First, the cost of maintaining the mediators is reduced to the cost of maintaining the meta-data - a significant improvement.  Second, the wrappers become easier to maintain because they are clearly separated from mediators by the API and can make use of the data representations developed for the mediators.

To see the meta-data needed to derive the mediators, we take a closer look at what the mediator needs to do. 

## Four types of meta-data are required

**A****P****I**

** ****Mediator**

**Wrapper**

**Notes:**
The mediator takes data in the data source representation, convert it into the representation required by the data warehouse then enters it into the warehouse.  In other words, it takes one representation of an abstract concept, transforms it into an different, but equivalent representation, then maps it onto the warehouse.  To do this automatically, we need meta-data to represent the abstractions, transformations, mappings and schema.

## Generating the mediators

**Transformation**
**Calls**

**Population**
**Code**

**SQL**
**Interface**

**Mediator Class**

**Mediator**
**Interface**

**Notes:**
Once we have this meta-data, we need to translate it into the actual mediator class in order to use it.  This is accomplished through our mediator generation program which reads the meta-data and creates the appropriate C++ classes and methods.  

As a side effect of this process, a library containing a class hierarchy  is defined which describes the internal data format used by the mediators.  Each abstraction in the meta-data has a corresponding class in the library that defines all of the attributes of that abstraction, as well as appropriate get and put methods for each attribute, and appropriate constructors and destructors. 

## The translation library and mediator class are used by the wrapper

**The translation library and mediator class are used by the wrapper**

**A****P****I**

** ****Mediator**

**Wrapper**

**parser**

**source rep**

**mediator**

**semantic ****mapping **

**high-level****object **

**A****P****I**

**Notes:**
These classes can be used as an internal data representation for the wrapper -  simplifying its creation.  Instead of having to develop their own data representation for each wrapper, implementers simply write the appropriate parser to read the data into the class hierarchy defined by the translation library, then pass the resulting high-level object to the mediator class where it is entered into the warehouse.

Updating the wrapper only requires updating the parser to read the new data representation - the class hierarchy is automatically updated when the meta-data is updated.

## Results:

**understanding SCOP****	****     2.0****	****           2.0****	****   0.0         0****writing wrapper****	****	****     4.5****	****           2.5****	****   2.0       44%****modifying schema****	****	****     0.5****	****           0.5****	****   0.0         0****writing mediator****	****	****     4.0****	****           0.0****	****   4.0        ---****modifying meta-data****	****	****     0.0****	****           1.0****	****  (1.0)      --- **
**total time in days****	****	****    11.0****	****           6.0****	****   5.0       ****45%**

**Results:**

**Integrating SCoP into warehouse that already contains PDB and SWISS-PROT.**

**Notes:**
We have used this meta-data based architecture to integrate two data sources into our warehouse - SCoP and dbEST.  This table shows the time required to integrate SCoP into the warehouse using our meta-data based approach as compared to the time required to do the integration by hand.  
As you can see, there was a 45% time savings using the meta-data based approach. 

## Results:

**understanding SCOP****	****     2.0****	****           2.0****	****   0.0         0****writing wrapper****	****	****     4.5****	****           2.5****	****   2.0       44%****modifying schema****	****	****     0.5****	****           0.5****	****   0.0         0****writing mediator****	****	****     4.0****	****           0.0****	****   4.0        ---****modifying meta-data****	****	****     0.0****	****           1.0****	****  (1.0)      --- **
**total time in days****	****	****    11.0****	****           6.0****	****   5.0       45%**

**Results:**

**Integrating SCoP into warehouse that already contains PDB and SWISS-PROT.**

**Notes:**
And, as expected, most of the time savings comes from the fact that we only need to update the meta-data to create the mediator - we don’t need to actually write the code to perform the warehouse update.

## Results:

**understanding SCOP****	****     2.0****	****           2.0****	****   0.0         0****writing wrapper****	****	****     4.5****	****           2.5****	****   2.0       44%****modifying schema****	****	****     0.5****	****           0.5****	****   0.0         0****writing mediator****	****	****     4.0****	****           0.0****	****   4.0        ---****modifying meta-data****	****	****     0.0****	****           1.0****	****  (1.0)      --- **
**total time in days****	****	****    11.0****	****           6.0****	****   5.0       45%**

**Integrating SCoP into warehouse that already contains PDB and SWISS-PROT.**

**Results:**

**Notes:**
However, an interesting, and initially surprising,  side note is that we also found significant savings in the time required to write the source wrapper.  This savings comes from the fact that we can use the automatically generated class library as the internal data representation - we don’t need to spend the time generating our own internal representation.

## Improving Data Access 

**Scientists need **
**Better access to the data**
**combine data from ****	****  multiple sources**
**annotate data **
**perform complex queries**
**Better functionality**
**customized notification ****	****   messages**
**integrated interface to tools**
**personalized responses to queries**

**By extending our meta-data representation, we can provide powerful, customizable access to data.**

**Notes:**
Now, being able to integrate new data source easily and keep up with the changes in more previously integrated sources is great - but it isn’t the end of the story.  It is only the beginning.

Our goal is to provide scientists the data management infrastructure they need to perform their research.  In order to fully utilize the data in a warehouse, they need to have better access to the data and better ways of interacting with it than they currently do.  

Better access implies not only semantically integrated data from multiple sources, but also the ability for scientists to add their own, personal annotations to the data and to perform complex queries on the data.

Better functionality implies tools that work on the integrated data, and personalized interaction with the data - from customized notification messages when data of interest is added, to customized query results based on user preferences.

Currently, we are working on extending our meta-data infrastructure to provide this type of powerful, customizable access to data.

## DataFoundry’s meta-data driven interface

**Notes:**
The starting point for the web-based interface we are currently developing is this rather simple window.  The user selects one of a set of high-level objects to initiate the query, and the type of query they wish to perform.  Then they enter the information they are using as the basis of the search and press go.

In this example, the user is looking for all sequences that are similar to the sequence they have entered using a basic blast search as a comparison

The set of high level objects and the list of allowable comparators are dynamically created based on user-specific meta-data.  As a result, different users may have different options.

## DataFoundry’s meta-data driven interface

- Src         Name       score    e-val    organism
- SP        P29728        1514  0.0       Human
- SP        P29081        362    1e-99   Mouse
- SP        P04820        358    2e-98   Human
**	****.**
**	****.**
- dbEST  zl85d08.r1  294    3e-78   Homo sapiens colon
**	****.**
**	****.**
- PDB     4AAH	       28        2.5       Methylophilus meth
- PDB     1ZAP         28        3.3       Candida albicans
**	****.**
**	****.**

- PDB     4AAH	       28        2.5       Methylophilus meth

**Notes:**
The results displayed when the query returns are presented as part of a second screen - which is shown here as a cartoon.  We are still working on developing this interface completely, and have only a subset of the final capabilities in place at this time.
In this case, for blast results, the user wants to see the source database and sequence name, as well as the score and e-value from the blast run, and the name of the organism the protein comes from.  One of the important things to note is that we are returning both protein and DNA information from several data-sources  SP, dbEST and PDB, as the result of a single query. 
To blast against sources that have not been integrated into our warehouse, or to use these results as the starting point of another blast query, the user would select the desired proteins then press the expand button and make the appropriate selection(s) from the available options.  To filter the proteins displayed, for example to remove all mouse proteins, the user would select the reduce option.  
To find out more about a particular protein, the user would click the describe button and be presented with options, based on the data they have selected in the first column.  In this case, since we have selected a PDB protein that came from a blast search, options include viewing the protein’s structure, or its alignment with the original query sequence, among others.  By selecting the alignment option, we get the textual display of the blast alignment for all selected proteins appearing in a separate window.
The set of data types, the legal options for each data type, and how to execute the options, are all stored as meta-data - allowing us to easily add new methods and types.
We believe that this type of interface is intuitive to the scientist, yet powerful enough to allow complex queries against the data warehouse.

## Beyond fully integrated data

**There are over 500 genomics data sources available on the web.**
**Scientists want as much *****relevant***** information as possible.**
**Integrating data from all of these sources is impossible.**

**Semantically integrating critical data sources, and providing basic access to others, offers the best possible solution. **

**Notes:**
However, this still isn’t enough to fulfill the scientists requirements.  Right now, there are over 500 genomics data sources publicly available over the web - and that number is growing at an alarming rate.

For any given scientist, only a subset of these sources are relevant at any one time.  However, that set of relevant sites varies greatly between scientists and even between particular queries by the same scientist.  While the scientist wants, and even needs, access to as many of the relevant data sources as possible, it is impossible to integrate all of these source - much less keep up with the maintenance of the resulting warehouse.  There is just too much information to be manipulated and not enough cycles to do it.

To address this problem, we intend to pursue a hybrid approach to data integration in the future.  We will continue to fully integrate critical data sources - those sources used most by our scientists - while attempting to provide non-integrated access to as many other data sources as possible.

We believe that by extending our meta-data-based interface, we will be able to dynamically query a large number of external sites and return the unparsed data to the user for their analysis.  While this hybrid approach is not an ideal situation, and there are still many issues to be resolved, realistically, it is the best option we can see.

## Summary

**integrated data**
**allows complex queries **
**is easier to understand**
**limits the number of sites**

**non-integrated data**
**allows more sites**
**is more flexible**
**is harder to query**

**Scientists need *****intuitive***** access to data from both internal and external sites.**

**Notes:**
To wrap up.  The current state of scientific data access in genomics is at best poor.  The burden of data management is placed squarely on the shoulders of the biologist - where it does not belong.  We need to go far beyond where we are now  in order to provide scientists the type of intuitive and powerful access to data that they require.  

Furthermore, because of the tremendous growth in the amount of data and data sources available, we believe that any successful interface must provide access to both integrated  and non-integrated data. Integrated data is needed to ask complex queries, while non-integrated data is realistically necessary to keep up  with all of the sources being made available.

## Conclusions

**DataFoundry is building on its meta-data based infrastructure to develop a scalable, flexible, and useable system.  **

**Meta-data provides a way to **
**reduce the cost of integrating new sources**
**reduce the cost of accessing non-integrated sources**
**provide a powerful, and *****intuitive*****, query mechanism**
**customize the user interface **

**Notes:**
Meta-data provides a way to reduce both the cost of integrating new data sources into the warehouse and the cost of accessing non-integrated data sources, while providing a powerful, and intuitive query mechanism and a customizable user interface.

We believe that making extensive use of meta-data is the only way to develop a powerful and flexible scientific data management system

## DataFoundry: An Approach to Scientific Data Integration

**Terence Critchlow**
**Center for Applied Scientific Computing**
**Lawrence Livermore National Laboratory**
***critchlow@llnl.gov***
***www.llnl.gov/casc/datafoundry/***
**Work performed under the auspices of the U.S. DOE by LLNL under contract No. W-7405-ENG-48. UCRL-JC-134854**

**Notes:**
Any questions?