# Presentation: 506404

## The Gridfor Engineers

- Ruth Pordes
- Fermilab
- With thanks for slides from Ian Foster, Vicky White, and many others

## The Grid Vision

- Federico.Carminati@cern.ch

## Coordinated Sharing of Heterogenous 

	Computation, Storage, Network... 

	Resources across a set Administrative Domains by

  		 Multiple Dynamic Organizations

- 	Computation, Storage, Network...
- 	Resources across a set Administrative Domains by
-   		 Multiple Dynamic Organizations

## Interview: The Future in Grid Computing

-  By David Worthington, BetaNews
-  February 21, 2005, 11:41 AM
- INTERVIEW Computing grids are software engines that pool together and manage resources from isolated systems to form a new type of low-cost supercomputer. In spite of their usefulness, grids remained the plaything of researchers for many years. But now, in 2005, grids have finally come of age and are becoming increasingly commercialized.
-  Sun Microsystems recently unveiled a new grid computing offering that promises to make purchasing computer time over a network as easy as buying electricity and water. Even Microsoft is said to be investing in grids and Sony has grid-enabled its PlayStation 3 for movie-like graphics.
-  As interest in these distributed technologies grow, so does the probability for disinformation. With that in mind, BetaNews sat down with some of the world's leading grid guru's, Dr. Ian Foster and Steve Tuecke, to set the record straight and divorce grid hype from grid reality.
-  BetaNews: Since we last spoke in 2001, what significant developments have there been in the commercialization of grid technologies?
- Dr. Ian Foster: Back then we were just seeing earlier interest in grid technologies from companies like IBM etc. Since then we have seen tremendous growth and enthusiasm. And a lot of things are being labeled as grid that perhaps one could argue they are not. Perhaps they are more, in some cases, computing cluster management solutions, but also some substantial early deployments in the industry from companies like IBM and Sun, and others like HP and so forth.
-  Then more recently we have seen Univa being created, which I am involved as founder and advisor.

## Reality can Still be Pretty Simplistic - even in the 

**Reality can Still be Pretty Simplistic - even in the **
**Commercial World**

## Slide 6

## Grid, grid, grids

## The (Power) Grid:On-Demand Access to Electricity

- The (Power) Grid:On-Demand Access to Electricity

- Time

- Quality, economies of scale

## By Analogy a Computing grid

- Decouple production and consumption
- Enable on-demand access
- Achieve economies of scale
- Enhance consumer flexibility
- Enable new devices
- Standardization of interfaces
- Voltage, current, frequency, plugs
- On a variety of scales
- Department, Campus, Enterprise, Internet

## State of the art today for “The Grid” is

- Moving data files between institutions, across continents, (almost) seamlessly and automatically
- Submitting jobs from your desktop and having them run somewhere at one of the “centers” of your “virtual organization”

## There are Many Grids

- 	National
- 		Cooperating
- 				Federated

## Slide 12

## PRAGMA 7: Practicing Innovative, International, and Interdisciplinary Science
La Jolla, Calif., U.S. —The seventh annual Pacific Rim and Grid Middleware Assembly (PRAGMA) conference celebrated the achievements of its middleware projects and the integration of new testbed resources at 10 institutions, two important steps towards PRAGMA’s chief goal of constructing a viable grid.
	Held on the UCSD campus from Sept. 15 through 17, PRAGMA 7 unveiled data gathered from continuously running a sample computational chemistry application over three months on 10 compute platforms in the PRAGMA testbed. “By allowing these applications to drive the underlying middleware deployment and configuration, we are learning how to share resources across international boundaries.” said Mason Katz, co-chair of the Resources Working Group and co-chair of the PRAGMA 7 Workshop. “In addition, the lessons learned from this experience will help shape the construction of international production grids and will be valuable for hosts of applications.”
	PRAGMA 7 also highlighted the successful integration of the Grid Datafarm distributed file system (gfarm) and the Genome Annotation Pipeline (iGAP), a suite of bioinformatics software. The project not only achieves software interoperability and provides access to more users, it also illustrates the value of researchers working across disciplines and continents, which is “critical to building a community of researchers, colleagues, friends, and ultimately an extended global family,” said Dr. Jysoo Lee, deputy chair, PRAGMA Steering Committee and director of the Korea Institute for Science and Technology Information Supercomputing Center.

**PRAGMA 7: Practicing Innovative, International, and Interdisciplinary Science**
- La Jolla, Calif., U.S. —The seventh annual Pacific Rim and Grid Middleware Assembly (PRAGMA) conference celebrated the achievements of its middleware projects and the integration of new testbed resources at 10 institutions, two important steps towards PRAGMA’s chief goal of constructing a viable grid.
- 	Held on the UCSD campus from Sept. 15 through 17, PRAGMA 7 unveiled data gathered from continuously running a sample computational chemistry application over three months on 10 compute platforms in the PRAGMA testbed. “By allowing these applications to drive the underlying middleware deployment and configuration, we are learning how to share resources across international boundaries.” said Mason Katz, co-chair of the Resources Working Group and co-chair of the PRAGMA 7 Workshop. “In addition, the lessons learned from this experience will help shape the construction of international production grids and will be valuable for hosts of applications.”
- 	PRAGMA 7 also highlighted the successful integration of the Grid Datafarm distributed file system (gfarm) and the Genome Annotation Pipeline (iGAP), a suite of bioinformatics software. The project not only achieves software interoperability and provides access to more users, it also illustrates the value of researchers working across disciplines and continents, which is “critical to building a community of researchers, colleagues, friends, and ultimately an extended global family,” said Dr. Jysoo Lee, deputy chair, PRAGMA Steering Committee and director of the Korea Institute for Science and Technology Information Supercomputing Center.

## grid infrastructure enables

- Utility computing
- Virtualization
- Data center automation
- Adaptive enterprise
- Collaboratories

## ...to access resources through Standard Interfaces

- Computers

- Disk/Tape

- Specialized
-  resource

- User
- Application

- User
- Application

- User
- Application

- Database

- Storage
- Service

## Standards are needed to allow heterogeneous implementations to participate in a common infrastructure.

Can the grid  standard interfaces and services  be regarded as adding intelligence to the Network layer?

Dispatch work anywhere  - full connectivity)
Discover spare compute cycles - learn about host capabilities
Name Servers to identify grid services.

- Can the grid  standard interfaces and services  be regarded as adding intelligence to the Network layer?
- Dispatch work anywhere  - full connectivity)
- Discover spare compute cycles - learn about host capabilities
- Name Servers to identify grid services.

##  e.g. Security

## Person has single “Login” and Identity Certificate (Credit Card) e.g.:

- Issued to:
- Subject: CN=Ruth Pordes 101995, OU=People, DC=doegrids, DC=org
- Serial Number: 0E:EB
- Valid from 1/26/05 8:28 AM to 1/26/06 8:28 AM
- Purposes: Client,Server,Sign,Encrypt
- Issued by:
- Subject: CN=DOEGrids CA 1, OU=Certificate
- Authorities, DC=DOEGrids, DC=org

## Security Details

- Users

## a domain example:

## Earthquake Engineering Simulation

-  Links instruments, data,
- computers, people* *

- NEESgrid Multisite Online Simulations

## Secure, reliable, on-demand access to data,software, people, and other resources(ideally all via a Web Browser)

- Secure, reliable, on-demand access to data,software, people, and other resources(ideally all via a Web Browser)

## How it Really Happens(A Simplified View)

***Collective services**** aggregate &/or virtualize resources*

***Users**** work **with client applications*

***Application services**** organize VOs & enable access to other services*

- How it Really Happens(A Simplified View)

## How it Really Happens(with Grid Software)

***Collective services**** aggregate &/or virtualize resources*

***Users**** work **with client applications*

***Application services**** organize VOs & enable access to other services*

- How it Really Happens(**with** Grid Software)

- WebBrowser

- ComputeServer

- GlobusMCS/RLS

- DataViewerTool

- CertificateAuthority

- CHEF ChatTeamlet

- MyProxy

- CHEF

- ComputeServer

- Database
- service

- Database
- service

- Database
- service

- SimulationTool

- Camera

- Camera

- TelepresenceMonitor

- Globus IndexService

- GlobusGRAM

- GlobusGRAM

- GlobusDAI

- GlobusDAI

- GlobusDAI

| Application Developer | 2 |
| --- | --- |
| Off the Shelf | 9 |
| Globus
Toolkit | 5 |
| Grid Community | 3 |

## closer to home...

## Increasingly part of the computing for Fermilab experiments is provided off-site

Fermilab has been in the lead in Grid Computing (even before it became a household word) 

SAM-GRID is fully functional distributed computing infrastructure in use by D0, CDF and MINOS
~30 SAM stations worldwide active for D0
~20 SAM stations worldwide active for CDF

D0 successfully carried out reprocessing of data at 6 sites worldwide

And In order to better serve the entire program of the laboratory the Computing Division will place all of its production computing and storage resources in a Grid infrastructure called FermiGrid. 

- Fermilab has been in the lead in Grid Computing (even before it became a household word)
- SAM-GRID is fully functional distributed computing infrastructure in use by D0, CDF and MINOS
- ~30 SAM stations worldwide active for D0
- ~20 SAM stations worldwide active for CDF
- D0 successfully carried out reprocessing of data at 6 sites worldwide
- And In order to better serve the entire program of the laboratory the Computing Division will place all of its production computing and storage resources in a Grid infrastructure called FermiGrid.

## Slide 27

## Proto-Grid for US LHC 

- Multi-organization common shared Grid environment

** **
**35 sites **
**400-1100 concurrent jobs**
**10 applications**
**Running since October 2003**

## Plans for Production National Grid Infrastructure Open Science Grid (OSG)

- Join all of the LHC computing resources at labs and universities in the U.S.
- Add, over time, computing resources of other high energy and nuclear physics experiments and other scientific partners from Grid Projects
- Federate all of these computers and storage systems and services together into a Grid that serves the needs of all of these physics and related disciplines.

- Bioinformatics embracing “grid” concepts to change culture of the field

- 	Workflow - Unified treatment of
*	**What I want to do*  - likely to evolve in response to new knowledge
*	**What I am doing now*  - may evolve, e.g., in response to failure
*	**What I did*  - static, persistent; a source of information
- Semantic Grid - Unified treatment of	*Describing  data* - likely to evolve in response to new knowledge
- 	*Managing data* - may evolve, e.g., in response to failure
-    *Tracking data* - static, persistent; a source of information

## Exploring Williams-Beuren Syndrome using myGridHannah TipneyAcademic Department of Medical Genetics. University of Manchester. UK.

- Contiguous sporadic gene deletion disorder
- 1/20,000 live births, caused by unequal crossover (homologous recombination) during meiosis
- Haploinsufficiency of the region results in the phenotype
- Multisystem phenotype – muscular, nervous, circulatory systems
- Characteristic facial features
- Unique cognitive profile
- Mental retardation (IQ 40-100, mean~60, ‘normal’ mean ~ 100 )
- Outgoing personality, friendly nature, ‘charming’

- Example...

**Notes:**
‘Elfin-like’ face, broad forehead, wideset eyes, button nose, flat nasal bridge, long philtrum, low set checks, wide mouth, malocclusion of teeth.
Mental retardation (IQ 40-100, mean~60, ‘normal’ mean ~ 100 )
Outgoing personality, friendly nature, ‘charming’.
Unique cognitive profile
Short stature
Hoarse voice
Infantile hypercalcemia (raise blood calcium levels)
Sensitivity to noise

## myGrid

- E-Science pilot research project funded by EPSRC www.mygrig.org.uk
- Manchester, Newcastle, Sheffield, Southampton, Nottingham, EBI and RFCGR, also industrial partners.
- ‘targeted to develop open source software to support personalised *in silico *experiments in biology on a grid.’

**www.mygrid.org.uk**

- 	**Which means....**
- 	Distributed computing – machines, tools, databanks, people
- 	Personalisation
- 	Provenance and Data management
- 	Enactment and notification
- 	A virtual lab ‘workbench’, a toolkit which serves life science 	communities.

## Williams Workflow Plan

- GenBank Accession No

- GenBank Entry

- Seqret

- Nucleotide seq (Fasta)

- GenScan

- Coding sequence

- ORFs

- prettyseq

- restrict

- cpgreport

- RepeatMasker

- ncbiBlastWrapper

- sixpack

- transeq

- 6 ORFs

- Restriction enzyme map

- CpG Island locations and %

- Repetitive elements

- Translation/sequence file. Good for records and publications

- Blastn Vs nr, est databases.

- Amino Acid translation

- epestfind

- pepcoil

- pepstats

- pscan

- Identifies PEST seq

- Identifies FingerPRINTS

- MW, length, charge, pI, etc

- Predicts Coiled-coil regions

- SignalP
- TargetP
- PSORTII

- InterPro

- Hydrophobic regions

- Predicts cellular location

- Identifies functional and structural domains/motifs

- Pepwindow?
- Octanol?

- BlastWrapper

- URL inc GB identifier

- tblastn Vs nr, est, est_mouse, est_human databases.
- Blastp Vs nr

- RepeatMasker

- BLASTwrapper

- Sort for appropriate Sequences only

- Pink:     Outputs/inputs of a service
- Purple: Tailor-made services
- Green:  Emboss soaplab services
- Yellow:  Manchester soaplab services

- RepeatMasker

- TF binding Prediction

- Promotor Prediction

- Regulation Element Prediction

- Identify regulatory elements in genomic sequence

**Williams Workflow Plan**

## The Workflow Experience

- Correct and Biologically meaningful results
- Automation
- Saved time, increased productivity
- Process split into three, you still require humans!
- Sharing
- Other people have used and want to develop the workflows
- Change of work practises
*Post hoc* analysis. Don’t analyse data piece by piece receive all data all at once
- Data stored and collected in a more standardised manner
- Results amplification
- Results management and visualisation

- Have workflows delivered on their promise?

**YES!**

## The Hype or the Promise

## Grid computing appears to be a promising trend for three reasons: 

- 	(1) its ability to make more cost-effective use of a given amount of computer resources,
- 	(2) as a way to solve problems that can't be approached without an enormous amount of computing power, and
- 	(3) because it suggests that the resources of many computers can be cooperatively and perhaps synergistically harnessed and managed as a collaboration toward a common objective. ..the computers may collaborate rather than being directed by one managing computer.
- One likely area for the use of grid computing will be pervasive computing applications - those in which computers pervade our environment without our necessary awareness.

*Search CIO.com Definitions*

## An Evolutionary Change

- Not just a question of computers and disks and tapes and access to them all
- A way of working that puts emphasis on equal access for all and standardization of the way things are done
- Enable Scientists to work by creating a massive “virtual” environment  (long way to go to get to the vision)

- Democratization of Science

## Revolution in Science
Construct and mine large databases of observational or simulation data
Develop simulations & analyses
Access specialized devices remotely
Exchange information within distributed multidisciplinary teams

Revolution in Business
Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)
Business processes increasingly computing- & data-rich
Outsourcing becomes feasible => service providers of various sorts

- Revolution in Science
- Construct and mine large databases of observational or simulation data
- Develop simulations & analyses
- Access specialized devices remotely
- Exchange information within distributed multidisciplinary teams
- Revolution in Business
- Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)
- Business processes increasingly computing- & data-rich
- Outsourcing becomes feasible => service providers of various sorts

## Business Examples 2005

- 	Satellite technology used to track every     item. Inventory adjusted  in real time - *data management, prediction, real-time, wide-area synchronization.*

## Science for everyone  seti@home, einstein@home

- Science Example

## Mix of Jobs running on Grid3 for past year

**Mix of Jobs running on Grid3 for past year**