# Presentation: 090113

## GADU/Gnare:Grid-based Server for High-throughput Genome Analysis

**Mathematics and Computer Science Division**
**Argonne National Laboratory**
- Natalia Maltsev, Alex Rodrigez, Dina Sulakhe,
- Mike Wilde, Nika Nefedova, Jens Voeckler
- Ian Foster, Rick Stevens

## Thousands of Complete genomes will be available by 2010 and a lot of experimental, biochemical, phenotypic etc data

**Analysis of Large amounts of data (genomic, biochemical, functional, phenotypic, etc) requires:**
- Mature data integration and distributed technologies
- Scalable computational resources (Grids, supercomputing)
- New tools and algorithms for pattern recognition and comparisons of biosystems on various levels of organization

**In Bioinformatics**

**Number of Sequencing Projects**

**119**

**538**

**965**

**169**

## MCS bioinformatics group: too many projects... 

- _[Metabolic Knowledge Base](slide4.xml)_ – a system for **high-throughput comparative and evolutionary analysis of metabolism**
- _[PhyloBlocks](file:///../../../program%20files/qualcomm/eudora/Demo/Demo/BISTI.ppt)__[, SVMMER](file:///../../../program%20files/qualcomm/eudora/Demo/Demo/BISTI.ppt)__[ ](file:///../../../program%20files/qualcomm/eudora/Demo/Demo/BISTI.ppt)_pipelines to support **evolutionary analysis of protein families**
**TarGet system** (to support NIH Midwest Structural biology Center) – **identification of the proteins with unique folds of interest to drug discovery**
**Pathos** (to support NIH Great Lakes Center of Excellence in Biomedical research) – **identification of proteins of interest to biodefence research**
**Sentra **– a database of prokaryotic **Signal Transduction Proteins**...
**Etc...**

## Hmm... there should be a solution....

**Hmm... there should be a solution....**

## All these projects need:

**Integrated  Database:**
- -Sequence data
- Annotations
- Results of Analyses

**Multistep Data Analysis **
**by variety of bioinformatics tools:**** **
- Blast Blocks, Pfam, Interpro TMHMM, etc
**Aka: workflow pipelines, **
**automated update cycle**

**Scalable (on-demand) **
**Computational Backend**

**Secure Collaborative Computational Framework **

## GADU/Gnare – Genome Analysis Research Environment

**Grid**

- Data Flow and Storage at various levels

**Chimera, Condor, Globus**

**GADU/Gnare – Genome Analysis Research Environment**

**Argonne GADU/GNARE team: **
- 	*Bioinformatics*: Alex Rodrigez, Dina Sulakhe, Natalia Maltsev
- 	*Globus:* Ian Foster, Mike Wilde, Nika Nefedova, Jens Voekler
- 	with the help of *Condor:* Miron Livny, Zach Miller

**Integrated**
**Database**
***Sequence data***
***Annotations***
- (NCBI, PIR, PDB,
- WIT3, Sentra,
- Pathos, etc)
***Results of analysis***
- (114 genomes in WIT3
- Blast, Blocks, Pfam,
- TMHMM for 124 genomes)
- Sentra, SubUnits, Pathos,
- TarGet
***Bookkeeping***

## GADU/Gnare:more details

**Gnare portal**

**GADU**

**Chimera**

**Grid **
**Resources**

## GADU/Gnare:more details

**Gnare portal**

**GADU**

**Chimera**

**Grid **
**Resources**

## DOESG Resource

**DOESG Resource**

## GADU/Gnare:more details

**Gnare portal**

**GADU**

**Chimera**

**Grid **
**Resources**

## GADU Grid Resources

## Grid’y GADU is Fast!

**Data processing for FIG project (R. Overbeek et al.)**
- Blast of a DB containing **2,1 M protein sequences** against itself (~450 average prokaryotic genomes 4000 each)
**84 hours run time** using 220 CPUs (out of 350)
- On 1 CPU ~**3 years 8 months**
**Data processing for PIR database (C. Wu et al.)**
- Blast of a DB containing **1,3 M protein sequences** against itself (~325 average prokaryotic genomes)
- On multiple resources:
- _[Grid3](http://www.ivdgl.org/grid2003/catalog/)_** **_**[(University of  Buffalo, J. Hopkins university, UWM, etc.)](file:///../../../program%20files/qualcomm/eudora/Demo/BISTI/GRID2003.htm)**_
- ANL jazz cluster
- UWM Condor pool
**4 average prokaryotic genomes in 31 min**
- Plans: TeraGrid, PNNL, DOE SG, Robust runs on Grid3

## Globus/Chimera manage Grid Jobs

- The current applications being used are protein and nucleotide sequence comparison tools such as BLAST, PFAM, BLOCKS, TMHMM and more.  These tools can be used to define a user workflow. These workflows can include such tasks as formatting the input data, submitting the data to several bioinformatics tools, gathering the results, parsing the results and delivering the results to the user.  This part is called the workflow definition, which is composed of ***transformations ***(executables) and ***derivations*** (what to do with the executables and input data from a previous step).
***The transformations and derivations are a major part of the Chimera language and are used to create the DAG, which is composed of a list of serial and parallel events.***
***The DAG is then submitted from the master location to the remote grid sites through the use of Globus and Condor-G*** in order to manage a job and transfer the input and output data from master site to remote sites and conversely.

## Chimera tracks Data Provenance

**Chimera provides an easier access to large-scale grid computation and storage resources**
**Accurate tracking of the provenance of how results were obtained**
**Easier ability to discover data and processing tools (easy search for transformations of data).  Metadata cataloguing.**
** ****Easier ability to produce new analysis “work” based on previously executed work (comparisons of patterns of transformations executed in different times)**
** ****Audit ability and dissemination of results (give users Chimera tracks : DAGs etc)**

## Grid As a Workstation	

**Chimera** – describes workflows that automate the integration of data transport and cataloging with computation, and keeps accurate records of what was computed how and where.
**GridFTP **– a ubiquitous high performance way to move data.
**Globus/Gram** – make our jobs independent of local cluster schedulers (PBS, condor, LSF).
**CondorG **– helps manage a single work queue to the grid, credentials, etc.
**MDS **– enables us to find out what’s where in a large grid.
**RLS** – tells us where our data is at any given time.

**Behind the Scene**

## GADU/Gnare:more details

**Gnare portal**

**GADU**

**Chimera**

**Grid **
**Resources**

## Gnare: Secure Collaboration... 

- Portals technology supports various levels of Security:
- Hit-and-run – web access to pre-computed data from the Web
- Registered users
- Groups
- Collaborators

**Security**

**Systems Biology requires collaboration:**
**Distributed data**
- (sequence data, biochemical networks,
- regulatory, gene expression, etc)
**Distributed Expertise**
**Distributed tools**
**Distributed users**
**Distributed students**
**And one elephant model!**

**Collaborative Tools:**
- Authentication
- Customized Project space
- Chat, Whiteboard, News, Mailing Lists
- Shared Views, Secure Data
- AG Tools

## GADU/Gnare:more details

**Gnare portal**

**GADU**

**Chimera**

**Grid **
**Resources**

## Examples: what GADU/Gnare can do for the projects...

**GADU Integrated  Database:**
- -Sequence data
- Annotations
- Results of Analyses

**GADU Multistep Data Analysis:**
**Aka: workflow pipelines**
**Blast,**** Blocks, Pfam, TMHMM, ****parsers**

**GADU Scalable Computational Backend**

**Gnare Secure Collaborative Computational Framework**

- ?

## GADU/Gnare: past, present, future

**Currently GADU/Gnare supports:**
- The SEED project
- Test runs for PIR
- Our Own projects:
- Metabolic Knowledge base
- TarGet
- Pathos
- SubUnits
- Sentra
- PhyloBlocks
- SVMMER
**Next Stage: A group of close collaborators (under development)**
- PIR (Protein Information Resource)
- The SEED
- University of Oklahoma
- Great Lakes NIH RCE in Biomedical research bioinformatics core
**Future**
- Public access to the server from the Web

## Acknowledgements

- MCS ANL Bioinformatics group: Alex Rodriguez, Dina Sulakhe
- Globus: Mike Wilde, Nika Nefedova, Jens Voeckler, Gregor von Laszewski, Ian Foster
- Condor: Zach Miller, Miron Livny
- Grid3 people
- MCS: Rick Stevens, systems, Susan Coghlan, Von Welch and a lot of others....