# Presentation: 454798

## High End Computing Context forGenomes to Life

***William J Camp, PhD***
*Director*
- Computers, Computation, and Math Center
- January 22, 2002

## What’s Happening with High End Computing ?

- High-volume building blocks available
- Commodity trends reduce cost
- Assembling large clusters is easier than ever
- Incremental growth possible

- HPC market is small and shrinking (relatively)
- Performance of high-volume systems increased dramatically
- Web driving market for scalable clusters
- Hot market for high-performance interconnects

- (Commodity, Commodity, Commodity)

## Commodity Value Propositions

- Low cost to drive down the cost of simulation
- Flexible and adaptable to changing needs
- Manage and operate as a single distributed system

**Price**

**Performance**

**PCs and **
**Workstations**
**  ****10,000,000s units**

**Enterprise **
**servers**
**  ****100,000s units**

**High-Performance **
**Computing**
**  ****1,000s units**

**Best Price **
**Performance**

**High Volume Technologies Give Favorable Price-Performance**

## Cray-1

## nCUBE-2 Massively Parallel Processor (1024)

- Figure 1: Two hypercubes of the same dimension, joined together,
- form a hypercube of the next dimension. N is the dimension of the hypercube.

## Intel Paragon

- 1,890 compute nodes
- 3,680 i860 processors
- 143/184 GFLOPS
- 175 MB/sec network
- SUNMOS lightweight kernel

## High Performance Computing at Sandia:Hardware & System Software

**ASCI Red: The World’s First Teraflop Supercomputer**
- 9,472 Pentium II processors
- 2.38/3.21 TFLOPS
- 400 MB/sec network
- Puma/Cougar lightweight kernel OS

**Cplant****TM****: The World’s Largest Commodity Cluster**
- ~2,500 Compaq Alpha Processors
- ~2.7 Teraflops
- Myrinet Interconnect Switches
- Linux-based OS (Portals)

## Cplant Performance Relative to ASCI Red

## Molecular Dynamics Benchmark(LJ Liquid)

***Fixed problem size*** (32000 atoms)

## Distributed & Parallel Systems

- Distributed
- systems
- hetero-
- geneous

- Massively
- parallel
- systems
- homo-
- geneous

- Legion\Globus

- Beowulf

- Berkley NOW

- Cplant

- Internet

- ASCI Red
- Tflops

- Gather (unused) resources
- Steal cycles
- System SW manages resources
- System SW adds value
- 10% - 20% overhead is OK
- Resources drive applications
- Time to completion is not critical
- Time-shared

- Bounded set of resources
- Apps grow to consume all cycles
- Application manages resources
- System SW gets in the way
- 5% overhead is maximum
- Apps drive purchase of equipment
- Real-time constraints
- Space-shared

## Communication-Computation Balance forPast and Present Massively Parallel Supercomputers

***Machine******	******	******	******	******	******	******Balance Factor ***
***	******	******	******	******	******	******	******	******	******(*****bytes/s****/****flops*****)***
- Intel Paragon						1.8
- Ncube 2								1
- Cray T3E							.8
- ASCI Red							.6
- Cplant								.1

*But Does Balance Matter for Biology?*

## BiologyA Field With Increasing Impact on High End Computing

- Why is it important to high-end computing ?
- What effect is it having ?

## High-Throughput Experimental Techniques Are Revolutionizing Biological and Health Sciences Research

- DNA Sequencing
- Gene Expression Analysis With Microarrays
- Protein Profiling via High Throughput Mass Spectroscopy
- Protein-Protein Interactions
- Whole-Cell Response

## The Ultimate Goal of “Systems Biology”Is Driving A Broad Range of Computing Requirements

- Bioinformatics: Accumulating data from high-throughput experiments followed by pattern discovery & matching.
- Molecular Biophysics & Chemistry
- Modeling Complex Systems (e.g. Cells)

## Computing-for-the-Life Sciences: The Lay of the LandThe Implications of the New Biology for High-End Computing are Growing

- IT market for life sciences forecast to reach $40B by 2004, e.g.
**Vertex Pharmaceuticals: ****100+ Processor cluster, company featured in Economist article**
**Celera** **builds 1****st**** tera-cluster for biotechnology– speeds up genomics by 10x**
**IBM, Compaq****: $100 million investments in the life sciences market**
**NuTec**** 7.5 Tflops IBM cluster (US, & Europe-planned)**
**GeneProt**** Large-Scale Proteomic Discovery And Production Facility:1,420 Alpha processors.**
**Blackstone**** Linux/Intel Clusters (Pfizer, Biogen, AstraZeneca, & 10-15 more on the way)**

## Computing For Life Sciences at the TerascaleConsider One Example: In Silico Pharmaceutical Development ?

**Sequence Genome**

**Assemble**

**Gene Finding****“Identification”**

**Annotate**

**Gene to Protein****“Map”**

**Protein Protein****Interaction**

**Pathways Normal****& Aberrant**

**Function in pathway**

**Structure **

**Drug Targets**

**Cellular Response**

**1**

**10**

**100**

**1000**

*A Guess at the Computing Power Needed*** (****in TeraOps)**

**Bioinformatics**			**Molecular Biophysics**	**Complex Systems**

## Definitions

- Molecular Biophysics
- Biological molecular-scale physical and chemical challenges and phenomena (e.g. structural biology, docking, ion channel problem.)
- Bioinformatics
- Informatics challenges (I/O, data mining, pattern matching, parallel algorithms etc.) presented by high-throughput biology and opportunities of application of terascale computing
- Complex Systems
- Developing models (most likely computationally intensive) of complex biological systems (e.g. cells):
- Volume methods, Circuit Models (Biospice, Bio-Xyce)), stochastic dynamics (e.g. Mcell), informatics (e.g. AFCS) 5) unit op’s/rule-based interactions, reductionism (as last resort)

## Definitions

- Distributed Resource Management & Problem Solving Environments
- Developing understanding & collaborations to establish a role in developing the tools & environment to enable the use of terascale computing to produce rapid understanding of complex biological systems through the combined approaches of
- high-throughput experimental methods
- data
- parallel I/O
- system software
- meta OS
- simulation & complex system modeling
- integrated understanding
- People, data, software, hardware, algorithms distributed geographically, organizationally & institutionally– The **Bio-Grid**

## Some Examples

- (from Sandia’s experience)

## VXInsightTM Analysis of Microarray Data

- sperm

- sperm

- oocyte

- cuticle

- cuticle

- heat shock

- actin, tubulin

- ribosomal proteins

- G protein-
- coupled
- receptors

- odor receptors

## GeneHunter

- Popular linkage analysis code from Whitehead Institute:
- does multipoint analysis (coupled marker effects)
- Computation/memory is exponential in size of pedigree,
- linear in number of markers.
- 15 person pedigree -> 24 bits -> vectors of length 224
- CPU days on a workstation
- Ideal for explosion of genetic marker data (e.g. SNPs).
- Our project:  a distributed-memory parallel GeneHunter.
- run on Cplant
- extend scale of run-able problems, both in memory and CPU

## Gene Hunter: Parallel Performance Results

## New Database Methods forStructure-Property RelationshipsQSAR Equation with Signature

**Signature descriptors **
**(extended connectivity index)**

**Glycine**

**C**

**C=**

**O=**

**O**

**H**

**H**

**H**

**N**

**H**

**H**

**H**

**New Database Methods for****Structure-Property Relationships***QSAR Equation with Signature*

**Typical QSAR (Molconn-Z descriptors)**

**HIV-1 protease inhibitors binding affinities (pIC50)**

- (H3N-CH2-COOHH)

## Computational Molecular Biophysics

- _Molecular Simulation_
**Molecular Dynamics (MD)**
- NVT, NVE
- Grand Canonical MD
- Reaction-ensemble MD
**Monte Carlo (MC)**
- Grand Canonical MC
- Configurational Bias MC
- Gibbs-ensemble MC
- Transition state theory

- _Molecular Theory_
**Classical Density Functional Theory**
- _Electronic Structure Methods_
**Local Density Approximation (LDA)**
**Quantum Chemistry (HF etc.)**
- _Mixed Methods_
**Quantum-MD (Car-Parinello)**
**Quantum-CDFT**
**Brownian Dynamics-CDFT**

## Ion Channel Model & Initial Results

**Measured Ion Channel**
**Current-Voltage Curve**

**Ion Channel Model & Initial Results**

**Calculated Ion Channel**
**Current-Voltage Curve**

## Virtual Cell Project

- NCRR-funded center within the UConn Health Center, Center for Biomedical Imaging Technology
- National Resource for Cell Analysis & Modeling (Virtual Cell)
- http://www.nrcam.uchc.edu/
- Sandia’s Contributions
- efficient parallel implementation
- solving systems of stiff linear PDE’s
- converting digitized images in 3d to meshable geometry

## Virtual Cell CollaborationFirst Fully 3-D Simulation of the Ca2+ Wave Transport and Reaction During Fertilization of the Xenopus Laevis Egg

**Meshed with Sandia’s ****CUBIT**** technology and solved with Sandia’s ****MP-SALSA**** massively parallel diffusion, transport, and reaction finite element code.**

**Notes:**
The Xenipus laevis egg problem is a long-standing problem in computational biology. This frog egg is about 1 mm in diameter (and yet a single cell) and thus has been studied as a model system for decades. Until this work, no one had ever been able to produce a successful 3d model of the calcium wave in the fertilization process, needed because w/o the 3d treatment, the concavity of the wavefront (see movie), seen experimentally, cannot be reproduced. The solution of this problem provided a nice test case for Sandia’s capabilities.

## 3d Mitochondria Cristae Geometry from Confocal Microscopy

- Meshing of Mitochondrial Cristae for 3-D ADP/ATP Transport Study Within Mitochondria

- 3d Hex Mesh for Finite Element Analysis

- Computational Geometry

## Another Possible Scenario

**Sequence Genome**

**Assemble**

**Gene Finding****“Identification”**

**Annotate**

**Gene to Protein****“Map”**

**Protein Protein****Interaction**

**Pathways Normal****& Aberrant**

**Function in pathway**

**Structure **

**Drug Targets**

**1**

**10**

**100**

**1000**

*shortcut*

**100**

## Terascale Supercomputers To Date

**Sandia RedStorm**

**ASCI Q**

**ASCI White**

**ASCI Red 3.2 Tflops**

**Cplant**

**PSC & French**

**ASCI Blue Mtn**

**ASCI Blue Pacific**

- Balance Factor = 1

**ASCI Red 1.8 Tflops**

- $200M

- $25M

## Big Pharma (& Biotech) Are Increasingly DrivingThe High-End Computing Market.

**Annual Sales (est.)****	****	****$300B**
**Historic Growth Rate****	****10-14%**
**R&D Expenditures****	****	****$62B ($26.4B in US)**
**	****	****	****$4.6B external R&D to understand human genome**
**	****	****	****11% of sales in 1980**
**	****	****	****20% of sales in 2000**
**70% of patented drugs come off patent in the next 4 years.**
**80% average drop in sales revenue when patent expires.**
**$600M average drug development cost.**
**Diminishing pool of easy targets.**

## Computing-for-the-Life Sciences: The Lay of the LandThe Implications of the New Biology for High-End Computing are Growing

- IT market for life sciences forecast to reach $40B by 2004, e.g.
**Vertex Pharmaceuticals: ****100+ Processor cluster, company featured in Economist article**
**Celera** **builds 1****st**** tera-cluster for biotechnology– speeds up genomics by 10x**
**IBM, Compaq****: $100 million investments in the life sciences market**
**NuTec**** 7.5 Tflops IBM cluster (US, & Europe-planned)**
**GeneProt**** Large-Scale Proteomic Discovery And Production Facility:1,420 Alpha processors.**
**Blackstone**** Linux/Intel Clusters (Pfizer, Biogen, AstraZeneca, & 10-15 more on the way)**

## Speeding up Informatics - Parallel BLAST framework

- Idea:	Create a tool for running large parallel BLAST jobs
- 		(genome vs genome) on distributed memory cluster
- Problems with current approach:
- tens of 1000s of LSF jobs:
- can't control or schedule them, users don’t know progress
- full database on every proc
- I/O is not managed
- Goals for parallel tool:
- scalable parallel performance
- minimize and measure memory/proc  $$
- minimize and measure time in I/O
- feedback & monitoring for users
- design framework for more than BLAST

## Basic Idea

- Large query file & database = many strings.
- One file chunk per column, one per row.
- BLAST -> N2 subsets of work to do => reduced memory per proc.
- Others have looked at this as well

## Our Implementation of the Idea

- Master scheduler:
- runs on 1 proc
- breaks up sequence analysis problem into M x N pieces
- schedules slave jobs intelligently
- Slave codes:
- run on all procs
- BLAST (or other tools)
- pre- and post-processors
- PVM or MPI:
- launch slave job on particular proc
- messages back to master
- detect when slave is finished

## Scheduler Intelligence

**Query and DB pre-processing have to be run before BLAST sub-job.**
**Give a proc a new BLAST sub-job that doesn't require additional I/O.**
**Slaves can write to local disk to minimize I/O.**
**4 procs in a ES40 box can share files & loaded memory (via UBC).**

## User monitoring

- Master writes status file on slave tasks – JAVA app displays it:
- Timeline on job history on compute farm.
- Stats on parallel performance, load-balance, individual jobs & procs.
- Add instrumentation of BLAST for memory/CPU stats.

## Advantages

**Reduced memory use per processor:**
- each BLAST task uses small portion of database and query files
- can be as small as desired
- User feedback – JAVA app.
**Parallel efficiency & scalability:**
- fine granularity of BLAST sub-jobs -> load-balance
- overall throughput should be much greater and not dependent on high degree of user sophistication
**Reduced I/O:**
- local vs NFS disks
- sqrt(P) effect

## There May Be Other Ways to Attack This Problem:Current Timeline?

**Sequence Genome**

**Assemble**

**Gene Finding****“Identification”**

**Annotate**

**Gene to Protein****“Map”**

**Protein Protein****Interaction**

**Pathways Normal****& Aberrant**

**Function in pathway**

**Structure **

**Drug Targets**

**1**

**10**

**100**

**1000**

- Today

- 2020--2030?

## ...High-Throughput Experiments May  Create a Shortcut

***Existing clinical data and tissue banks***
***could be CRITICAL because ...***

- SNPsdisease association from clinical and genetic history

**Sequence Genome**

**Assemble**

**Gene Finding****“Identification”**

**Annotate**

**Gene to Protein****“Map”**

**Protein Protein****Interaction**

**Pathways Normal****& Aberrant**

**Function in pathway**

**Structure**** **

**Drug Targets**

**1**

**10**

**100**

**1000**

*shortcut*

**100**

## Computing Power Needs MayDecrease with Clinical Collaborations

- Today

- 2005--2015?

**Sequence Genome**

**Assemble**

**Gene Finding****“Identification”**

**Annotate**

**Gene to Protein****“Map”**

**Protein Protein****Interaction**

**Pathways Normal****& Aberrant**

**Function in pathway**

**Structure**** **

**Drug Targets**

**1**

**10**

**100**

**1000**

*shortcut*

**100**

## Summary

- There are bigger problems in GtL and similar efforts
-  than any computer can handle now or in the next 10 years
- However, we need to develop the computational tools and
- Frameworks that take us
- 	from genomics to proteomics
- and beyond to
- 	Whole cell/ whole organism response
- This will require a uniquely challenging juxtaposition of
- 	high-throughput experimental methods
- 	grid-based bio-informatics
- 	sophisticated simulation of info and energy flows,
- 	of structure and function, and of environmental reactions