# Presentation: 506529

## HPC in theHuman Genome Project  

**HPC in the****Human Genome Project****  **

**James Cuff**

**james@sanger.ac.uk**

**Notes:**
This is a picture of the Sanger Centre in England. This building has been in existence for four years but the Sanger Centre’s activities have been going  for over seven  years.

Situated in 55 acres of parkland The Wellcome Trust Genome Campus provides a wonderful setting for the exciting and challenging projects undertaken.

There are three institutes on the Genome Campus. 

The Human Genome Mapping Project - Resource Centre (HGMP-RC), The European Bioinformatics Institute (EBI) part of the European MicroBiology Laboratory (EMBL) of Heidelberg 
and the Sanger Centre.

The Sanger Centre is the largest of the three institutes. 

## The Sanger Centre is a research centre funded primarily by the Wellcome Trust

Located in 55 acres of parkland

Also on site are the 
European Bioinformatics Institute (EBI)
Human Genome Mapping Project Resource Centre (HGMP-RC)

**The Sanger Centre is a research centre funded primarily by the Wellcome Trust**
**Located in 55 acres of parkland**
**Also on site are the **
**European Bioinformatics Institute (EBI)**
**Human Genome Mapping Project Resource Centre (HGMP-RC)**

## The Sanger Centre

**Founded in 1993; >570 staff members now.**
**Our purpose is to further the knowledge of the biology of organisms, particularly through large scale sequencing and analysis of their genomes.**
**Our lead project is to sequence a third of the human genome as part of the international Human Genome Project.**

## Sanger Centre research programmes

** ****Pathogen sequencing programme**

**   ****Informatics**
** ****support data collection**
** ****analyse and present results**
** ****develop methodology: algorithms and data resources**

**   ****Cancer genome project**

**   ****Human genetic programme****   - study genetic variation (SNPs) and find ****      disease genes**

## The Structure of DNA

**in the DNA chain codes for the genes**

- BASE-
- PAIRS

**THYMINE**

**GUANINE**

**CYTOSINE**

**ADENINE**

**Four nucleic acids**
**Guanine, Thymine**
**Cytosine, Adenine**
**  ****are represented**
**computationally by**
**the characters**
**      ****A, C, T, G**

## Typical DNA Sequence

- A typist typing at
- 60 w.p.m for 8 hours
- a day would take
- around 50 years to
- type the book of life.
- Human DNA consists
- of 3000,000,000
- ‘letters’

## The era of genome sequencing

**	****	****Size****	****No. of Genes               ****	****                   Completion**
**	****        ****(Mbases)****	****	****	****	****                          date**
***H. influenzae*****	****  2          1,700    1/1kb       Bacterium****	****             1995**
**Yeast****	****	****13          6,000    1/2kb       Eukaryotic cell****	****1996**
**Nematode       100        18,000    1/6kb       Animal****	****	****1998**
**Human****	****           3000      ?40,000   1/60kb    Mammal****	****             2000/3**

- Sequence data production increase of >2000%

## The Sequencing Facility

**Notes:**
This is a view of part of the Sanger  DNA sequencing facility.
We have around 250 sequencers from ABI (3700's) as seen below. They run 24x7x365 and produce the continuous stream of data that determines the need for the amount of storage I.T systems we have.

## Sanger I.T.

**Sanger network****:- more than 1600 devices**

**300**** PC’s – various**
**150+**** X-terms/Network Computers (NCD)**
**250**** NT/Mac ABI Collection devices**
**Various other servers, Linux desktop systems, printers, etc.**
**Paracel, Compugen and Timelogic systems**

**>****350**** Compaq Alpha systems (DS10, DS20, ES40,8400)**
**+****440**** Node Sequence Annotation Farm (PC/DS10/DS10L)**
**>****750 Alpha processors in total**

**Notes:**
The I.T. systems are key to enabling the sequencing projects. Without the very latest technology we could not finish this enormous undertaking.

When the Nematode Worm and Human Genome projects began it was thought that we could complete the work on a few small workstations. We very soon realised this would not be possible. Today we have over 1500 devices on the Sanger Network.

Following the recent purchase of 320 Alpha DS10L systems, each with 1GB of memory and 2x30GB internal disks (>19.2TB in total) the total number of Alpha processors in Sanger is now greater than 700. This huge resource will be a critical part of the ensembl pipeline.
We are predominantly powered by Compaq Alpha systems which provide the main power for data analysis and assembly. The 64-bit computing environment has been the most stable environment for the last 6 years. We are looking forward to utilising the coming generations of Alpha processor which will be needed to handle the large file sizes and huge memory requirements.

Many users access this power-house via X-terminals or Network Computers on their desks. We also have approximately 150 Personal Computers installed.

There are a large number of PC's and Macintosh systems used as collection devices for the  sequencing machines.    

## Raid,8400,DS20,PC Farm

## Systems architecture hierarchy

**Compute**
**Server**
**     ****Farm**

- Raid
- Storage

- Raid
- Storage

***A T M***

**Front-end**
**Compute Servers**

**Front-end**
**Compute Servers**

- Desk top
- workgroup
- systems

- Desk top
- workgroup
- systems

- LSF - Load Sharing Facility by Platform Computing Ltd

**F/C**

**F/C**

**Notes:**
The network topology is arranged into three tiers.

The lowest tier on the slide (front-end) shows the Desktop systems. These will be X-terms. PC’s or Unix workstations used by each user. The mid-tier of systems are used as login servers and are available to run applications such as Netscape and other tools required for general housekeeping by each user. 

The back-end is where the high performance systems provide the compute power to run the high performance analysis and assembly applications.

LSF is setup to distribute jobs onto the compute engines according to the characteristics of the job, for example, high CPU or large memory requirement. 

Lstcsh is used by default and therefore the front-end user does not know exactly where their job executes - LSF makes that decision - the user will always get the best performance available at the time. LSF load-balances the work across the entire cluster.

## Computer Systems Architecture

**    ****Fibre Channel/Memory Channel**** ****Tru64**** ****Clusters**
**    **
**    ****Implementing tightly coupled clustering with **
**    ****Tru64 V5.x**
**    ****We get:**
**    ****Improved disk I/O (fibre channel),**
**    ****scaleability (multi-cpu, multi-terabyte)**
**    ****Improved manageability **
**    ****- single system image, whole clusters are ****      managed as single entities)**

**Notes:**
Our solution was to install fibre channel, memory channel clusters of Alpha processors running Tru64 V5.0. These clustered configurations of eight, four processor SMP machines can be managed as a single entity thus reducing the management overhead. The fibre channel attached disk enables faster disk I/O and the memory channel provides fast processor to processor communication. Applications can be run in high availability mode and will failpover to another
processor in the event of system failure.

## ES40 Clusters, F.C Storage

**Notes:**
This is a view of the latest hardware. Rows of ES40 clusters with fibre channel Storageworks storage. Each storage cabinet currently can take up to 2.5TB of storage. Newer disk technologies (72GB disks) are now available allowing more than 4TB to be installed in a single cabinet. 

## Annotation Farms

**Two network subnets (multicast and backbone)**** **
**640x100Mb Fast Ethernet ports**** **
**1,920 UTP cable crimps, 8 cabinets ~ 100kW of power**

**   ****8 Racks each with 40 x ****Tru64 v5.0 Alpha**** ****DS10L. **
**    ****320x466Mhz Alpha EV6.7****, 1U High****    Total of 320GB mem, spinning 19.2TB internal storage**

**   *****ca.***** Equivalent to 10 x GS320, perf around 355 Gflops**

## Slide 15

**Notes:**
This is an 'artists impression' of our  latest aquisition. 320 Alphaserver DS10L systems. Each has 1GB of memory and 2x 36GB internal disks. In total this adds up to 320GB of memory and over 19TB of disk space. If an Alphaserver GS320 has 32 processors then this system is almost equivalent in power.
Ten GS320 would yeild 355Gflops of power. This farm of 320 Alpha DS10L must therefore provide in the order of 300Gflops of power.

## Network Overview

**Highly available NFS (Tru64 CAA)**
**Fast I/O (ATM > switched full duplex ethernet)**
**Socket data transfer ****(via rdist, rcp, and MySQL DBI sockets)**
**Segmented network architecture via two elans**

- 8 node
- ES40
- M/C F/C
- cluster

- 172.27

- 172.25

- uplinks

- 172.27

- Sanger

- 172.25

- Farm

## Compute – systems architecture

- 400 node
- S.A. farm

- Pathogen

**Firewall DMZ**
**External Services**

**ATM**

- Trace
- server

## Enterprise Clustering

** ****LSF is still key for job scheduling and batch**
**  ****operations**
** ****LSF offers greater granularity of operation ****  and functionality than Tru64 scheduling**
** ****Schedule individual nodes, cluster-wide and**
**  ****cross-cluster scheduling **
** ****With LSF we still have the capability to **
**   ****use many  of the 750+ compute nodes as **
**   ****a **_**single**_**  Sanger Compute Engine **

**MODULAR  SUPERCOMPUTING**

**Notes:**
We use multi-cluster facilities to allow jobs to be scheduled between the various clusters. This allows us to build Sanger wide farms that offer specific services.

Our 'Night Queue'  is another example where we harness the power of many of our systems. Long running jobs are submitted to the Night Queue and left to run soaking up any spare CPU out of peak hours. This is a very powerful way to create an Enterprise Compute Engine.

## Projects

** ****Will involve **_**thousands**_** of CPU’s**** **
**       ****- Large numbers of PC farm nodes**
**       ****- High-end, large memory SMP configurations**

** ****All are computationally expensive**

** ****Will require  > 100 Terabytes of storage**

** ****We need to continue scaling up**
**    ****and deal with the physical limitations**

**Notes:**
All of these projects are extremely computationally expensive and will require a massive increase in our compute power.

We envisage that  hundreds and possibly thousands of processors will be required.

This will include very large clusters of high-performance, large memory systems and very large numbers of commodity PCs.

Data storage will need to increase dramatically and we envisage in the region of 50 - 100 Terabytes being used.

## Immediate Future

**The Sanger Centre**
**Genome Campus**

**ATM**

**LSF**
**Clustering**

**   ****Institute to Institute Clustering**
**   ****Closer collaborations between Sanger, EBI **
**   ****and other organisations brings the need ****   for site wide shared clusters.**

**Implement Storage Area Network**
** ****Install multi-TB to enable disk **
** ****mirroring, controller/controller **
** ****snapshots**

**Notes:**
For the immediate future we will be implementing a Storage Area Network (SAN). This will allow us to maintain the integrity of the growing storage. We also need to install closer links to the EBI to allow ensembl to function seamlessly. Institute to institute LSF clusters will allow cross site job distribution and scheduling. 

## Longer Term Future

** ****Wide Area Clusters  **
**    ****Needed for large scale collaborations.**

** ****GRID Technology - Global Distributed Computing**
**    ****International Cluster collaborations with other **
**    ****scientific institutes**

**GLOBAL COMPUTE ENGINES***** ***

** ****Sanger is keen to keep abreast of this emerging **
**   ****technology**

**Notes:**
On Campus  -  Institute to Institute LSF.
It is highly likely that we will run a single site-wide LSF installation. As the work of the Sanger Centre and the EBI becomes closer  it makes good sense to share resources.

Wide Area LSF?
There are possibilities of international collaborating LSF clusters.  Remote execution of jobs on  related data at remote sites where it may be more practical or  more appropriate to hold the data is a very attractive prospect.

We are keen to keep abreast of GRID technology as it emerges. In time we see Global Compute Engines being available to work on large scale genomic databases.

## Questions ?