# Presentation: 125865

## caBIG ArchitecturalOverview  

**Rakesh Nagarajan**
**Alvin J. Siteman Cancer Center**
**Washington University**

**Notes:**
all TODO and similar comment line's are prefixed by #

#Check the background, title of this page

## Complexities of Data Integration

- Data from various incompatible sources
- Text, XML, Proprietary databases, Public databases, Web (URLs)
- No standard naming convention
- Example: ‘Translation’ or ‘Protein Synthesis’
- No standard interface
- Each source has a custom browsing and querying mechanism
- Poor interaction with other data sources

**Notes:**
Aditya# I have added this slide just to start with the problem before going to data integration. Do we need this?
# naming convention point – Use of ontologies to address the need for consistent descriptions of gene products

## Approaches to Data Integration

**Link integration**
- Query with one data source and follow hypertext links to related information in other data sources
- Example: Sequence Retrieval System (SRS),  Entrez (NCBI)
**View integration**
- The information resides in the respective source databases
-  An environment is built around the databases that make them all seem to be part of one large system
- Examples: DiscoveryLink (IBM), K2 (Univ. of Pennsylvania)
**Data Warehousing**
- All data under one roof
- A unified data model is developed that can accommodate all the information from various source databases

**Notes:**
#Aditya: Just quickly explain the existing solutions and what they mean. Pros/Cons are just FYI.

Link integration –
Pros: Quick to set up, Achieves a basic level of integration with minimal effort
Cons: Vulnerable to naming clashes and ambiguities, Doesn’t scale well, Joins are manually done, Puts the onus of integration and interpretation on the researcher

View integration –
Pros: The query is on virtual schema, so no knowledge of source is needed, Query optimization possible, Low maintenance as views are not materialized, Querying can be done with standardized interfaces like SQL, OQL, XQuery etc
Cons: Creating and maintaining virtual schemas is a difficult task, Difficult to map all data-sources into a relational model, Query doesn’t execute when one or more data sources are down, Processing a query is limited by the slowest data source

Data warehousing –
Pros:Query is data-source independent, High performance, Suited for data-mining
Cons: Time factor is huge during implementation, Maintenance is difficult Near-real-time querying not possible

## Data Requirements

- Annotation Data- ~6 MB/array
- 10,000 genes
- 600 bytes per gene from 3 to 4 data sources
- Expression Profiling Data- ~20 MB/experiment
- 10,000 probes
- 10 samples
- 200 bytes per probe-sample
- Mutational Profiling Data- ~2.5 MB/gene
- 100 samples
- 24,000 bytes per sample for trace files and consensus sequences

**Notes:**
# Rakesh / Aditya Is there some number that we can put here for Microarray.

## Architectural Design Considerations

- Efficient handling of large amounts of data
- Campus-wide (Intranet) access
- Fast querying of data from various sources
- Quick setup with minimal overhead
- Allow users to upload and categorize data
- Less than 50 concurrent users

**Notes:**
# Should this slide be placed at the beginning.
# Less than 50 concurrent users – stress for No need for a middle tier
# Efficient handling of large data 

- All data is stored in a single database
- Querying information from multiple sources is PRACTICAL
- High performance
- Suitable for data mining
- Integration of data occurs at the lowest level
- No need for query or result integration
- Needs schema integration
- A standard schema or an ETL tool
- Source data updates
- Link integration of data that does not need to be warehoused
- Easy to set up
- Achieves a basic level of integration with minimal effort

**Our Approach: **
**Data Warehousing + Link Integration**

**Notes:**
Aditya#  Include sources that we are warehousing  ( unigene, locuslink, homologene, go, dbSNP) and linking (omim, etc.)
Aditya# a. As microarray data has common schema standard, integration and hence querying is easy/fast. B. Need to push for common schema in other areas as well.

## Siteman Cancer CenterArchitectural Schematic

- Annotation Server
- (ETL, Java)

- Web-based
- Tools
- (Browser)
- JSP,Servlets

**Data Sources**

**Presentation**

**Client**

**Data**

- ...ATTGGTCCGTCGTTACTGGATC...
- Genome Sequencing center

- GeneChip/Proteomics Core

- Integrated
-  DB
- (Oracle 9i)

**Siteman Cancer Center****Architectural Schematic**

- Function Express/
- Mutation Viewer
- 2-tier client server analysis tool
- (C++)

- LocalDB
- (MySQL)

## Two-tier Client Server Applications

- Integrate analysis data with gene annotation
- Powerful querying and visualization capabilities
- _Function Express:_ Expression Profiling Analysis
- Visualize the expression of all genes that are transcription factors located on chromosome 1p and that are down-regulated in tumor specimens relative to non-malignant tissue
- _Mutation Viewer:_  Mutational Profiling Analysis
- Visualize those mutations in tumor specimens where the mutation is non-synonymous, is not found in existing SNP databases, and falls in the kinase domain

**Notes:**
#For FE, tell the other two queries in the passing
# Rakesh – Can  you add a bullet on MV?

Leverages data integration power of warehousing

## Web Based Tools

- Allow web based simple queries on our data warehouse (_[http://](http://bioinformatics.wustl.edu/)__[bioinformatics.wustl.edu](http://bioinformatics.wustl.edu/)_)
- Gene Annotation Search
- Gene Expression Search
- Download experiment or annotation data
- Share data with other investigators
- Publish experimental findings

## Experiences

- Fetching large amounts of data over the Internet for real time analysis is not practical
- Our warehouse approach demonstrates the practicality of a central repository
- Warehouse approach facilitates one-time pre-calculating and pre-annotating
- Web based interface is a must
- Link integration: Not possible to warehouse all data
- Web based tools: View chip data for one gene
- Data sources must provide a standard interface
- Data update notification
- Retrieving updates

## caBIG Architectural Requirements

- caBIG compliant nodes across the country
- caBIG provides nodes with software to set up the complete node
- caBIG mandates the APIs, Interfaces and Publish Mechanism across nodes
- caBIG provides robust privilege mechanism for node collaboration
- Each center is free to choose the information it warehouses
- Open access, open source application sharing across nodes

**Notes:**
Also discuss what caBIG will mandate and what centers will need to do, what caBIG will provide that centers can choose to use/practice.
Mention senior architect here

## Architectural Schematic of a caBIG Compliant Node

- ...ATTGGTCCGTCGTTACTGGATC...
- Genome Sequencing center

- Integrated
-  DB

- Annotation Server
- (ETL, Java)

**Data Sources**

**Presentation**

**Data**

- LocalDB

- External
- caBIG
- Node

**Architectural Schematic ****of a caBIG Compliant Node**

- Client outside node

- Web-based
- Tools
- (Browser)

- Analysis tools
- (C++ or Java)

- Analysis tools
- (Java)

- O/R M
- Objects

- Servlet/JSP
- container`

- SOAP
- listener`

**Client**

- Analysis tool
- (C++ or Java)

- Client inside node

**D**
**A**
**O**

**Notes:**
Aditya# This architecture talk should mention the following
Stress on warehousing approach and separate the clients into “inside the node” and “outside to the node”.
Users within the nodes should have fast access to all the data they need to analyze their experiment. The data warehouse needs to bring frequently accessed data from other nodes.
 Inside users should have ability to connect to the warehouse using 2 tier applications.
d. Each node should provides a standard framework for outside users to connect via browser or applications (C++/Java).

The middle tier consists of 
a. Core objects (e.g. Domain, Role, Privileges, Organization, Investigator, etc.)
b. Data Access Objects (Object relational mapping layer)
d. Common objects (e.g. Data Storage, Logging, SMTP, etc.)
e. Application objects (e.g. Probe, ProbeSet, Annotation Framework, Annotation Category, Annotation, User Category, Array Design, etc.)
d. SOAP listener

------Some Useful Comments to add/discuss ------
Initially caCORE had implemented DAO pattern with their own Java classes. Then then switched to OJB (O/R mapping). We should recommend using Hibernate/JDO for O/R mapping as they are more mature and have larger group following.
We should recommend not to use EJB for its overhead. Instead we should suggest using a lightweight framework. Should investigate Picocontainer/Spring framework.
c. JavaBeans/POJO is a better way than EJB
d. It's best to program to interfaces, rather than classes. 
e. OO design is more important than any implementation technology, such as J2EE. 

## caBIG Interconnectivity

**caBIG Interconnectivity**

**Notes:**
Aditya# We need to group the nodes with same color in the animation
Each node collaborates with each other via privilege mechanism.
Each Node can provide Web services to others, Bulk data transfer interface or notify others / publish in Central registry what data they are sharing.

Each Node can choose to have its own set of applications or pick and choose from what caBIG / caBIG collaborators will provide.
Each Node can choose what data they want to warehouse, all they have to ensure it should be published via caBIG compliant mechanisms.

Central registry stores what data / web services / interfaces each node is providing. So if we want to let the world know that some data is available with us, we just need to publish it in central registry. This registry is just a facilitator and not a bottle neck. The information that it provides should be queriable from individual nodes also.