# Presentation: 939793

## FermiGrid, OSG and GIN

- Keith Chadwick
- Fermilab
- Work supported by the U.S. Department of Energy under contract No. DE-AC02-07CH11359.

## Personnel

- Eileen Berman, Fermilab, Batavia, IL 60510	_[berman@fnal.gov](mailto:berman@fnal.gov)_
- Philippe Canal, Fermilab, Batavia, IL 60510	_[pcanal@fnal.gov](mailto:pcanal@fnal.gov)_
- Keith Chadwick, Fermilab, Batavia, IL 60510	_[chadwick@fnal.gov](mailto:chadwick@fnal.gov)_	*
- David Dykstra, Fermilab, Batavia, IL 60510	_[dwd@fnal.gov](mailto:dwd@fnal.gov)_
- Ted Hesselroth, Fermilab, Batavia, IL, 60510	_[tdh@fnal.gov](mailto:tdh@fnal.gov)_
- Gabriele Garzoglio, Fermilab, Batavia, IL 60510	garzogli@fnal.gov
- Chris Green, Fermilab, Batavia, IL 60510	greenc@fnal.gov
- Tanya Levshina, Fermilab, Batavia, IL 60510	tlevshin@fnal.gov
- Don Petravick, Fermilab, Batavia, IL 60510	petravick@fnal.gov
- Ruth Pordes, Fermilab, Batavia, IL 60510	ruth@fnal.gov
- Valery Sergeev, Fermilab, Batavia, IL 60510	sergeev@fnal.gov	*
- Igor Sfiligoi, Fermilab, Batavia, IL 60510	sfiligoi@fnal.gov
- Neha Sharma Batavia, IL 60510	neha@fnal.gov	*
- Steven Timm, Fermilab, Batavia, IL 60510	timm@fnal.gov	*
- D.R. Yocum, Fermilab, Batavia, IL 60510	yocum@fnal.gov	*

- FermiGrid, OSG and GIN

- 9 May 2007

## FermiGrid - Current Architecture

- CMS
- WC1

- CDF
- OSG1

- CDF
- OSG2

- D0
- CAB1

- GP
- Farm

- VOMS
- Server

- SAZ
- Server

- GUMS
- Server

- clusters send ClassAds
- via CEMon
- to the site wide gateway

- BlueArc

- Periodic
- Synchronization

- D0
- CAB2

- Site Wide
- Gateway

- Exterior
- Interior

- FermiGrid, OSG and GIN

- 9 May 2007

## VOs hosted at FermiGrid

***VO******	******Focus Area                                        ******	******URL***
- fermilab	Fermilab Experimental Program	http://www.fnal.gov/faw/experimentsprojects/index.html
- accelerator, astro, cdms, hypercp, ktev, miniboone, minos, mipp, nova, numi, patriot, test, theory
- des	Dark Energy Survey	https://www.darkenergysurvey.org/
- dzero	D0 experiment	http://www-d0.fnal.gov
- I2u2	Interactions in Understanding The Universe	http://ed.fnal.gov/uueo/i2u2.html
- ilc	International Linear Collider	http://ilc.fnal.gov
- lqcd	Lattice QCD computations	http://lqcd.fnal.gov
- nanohub	Nanotechnology	http://www.nanohub.org
- gadu	Bioinformatics	http://compbio.mcs.anl.gov/gaduvo/gaduvo.cgi
- sdss	Sloan Digital Sky Survey	http://www.sdss.org
- osg	OSG individual researchers or small groups VO	http://www.opensciencegrid.org/

- FermiGrid, OSG and GIN

- 9 May 2007

## Computing Clusters

- Fermilab operates clusters for four major clients:
- General Purpose Grid Cluster:
- Used by smaller groups and experiments.
- Currently - 388 VM's on the grid, 240 more coming soon.
- CDF Clusters:
- Currently - 2 production grid clusters (520 WN / 2600 VM’s), 2 analysis/production (non-grid) clusters (~600 WN / ~3000 VM’s) and 1 test/integration cluster (8 WN / 40 VM’s).
- Future - 3 production grid clusters (~1200 WN / ~6000 VM’s) and 1 test/integration cluster (8 WN / 40 VM’s).
- D0 Clusters:
- Currently - 2 production grid clusters (860 WN / 3440 VM’s) and 1 farm/analysis cluster (360 WN / 1440 VM’s).
- Future - 3 production grid clusters (~1200 WN / ~4800 VM’s).
- CMS.
- Operated by the CMS Computing Facilities Department.
- Currently more than 1900 VM's on the grid.

- FermiGrid, OSG and GIN

- 9 May 2007

## FermiGrid - Storage

- BlueArc NFS Shared File Systems:
- Purchased 24 Tbytes (raw) = 14 Tbytes (formatted RAID 6)
- Currently mounted on (most) FermiGrid clusters.
- blue2:/fermigrid-home	/grid/home	1 TByte
- blue2:fermigrid-login	/grid/login	1 TByte
- blue2:/fermigrid-app	/grid/app	2 TByte
- blue2:/fermigrid-data	/grid/data	7 TBytes
- blue2:/fermigrid-state	/grid/state	1 TByte
- Public dcache (FNAL_FERMIGRID_SE):
- 7 TBytes of storage.

- FermiGrid, OSG and GIN

- 9 May 2007

## Software Stack

- Baseline:
- SL 3.0.x, SL 4.x, SL 5.0 (just released)
- OSG 0.6.0 (VDT 1.6.1, GT 4, WS-Gram, Pre-WS Gram)
- Additional Components:
- VOMS (VO Management Service)
- VOMRS (VO Membership Registration Service)
- GUMS (Grid User Mapping Service)
- SAZ (Site AuthoriZation Service)
- jobmanager-cemon (job forwarding job manager)
- MyProxy (credential storage)
- Squid (web proxy cache)
- syslog-ng (auditing)
- Gratia (accounting)
- Xen (virtualization)
- Linux-HA (high availability)

- FermiGrid, OSG and GIN

- 9 May 2007

## Authorization & Authentication

- DOEgrids Certificate Authority:
- Long lived (1 year) certificates.
- Heavy weight process (from the perspective of the typical user...).
- Fermilab Kerberos Certificate Authority:
- Service run on Fermilab Kerberos Domain Controllers.
- Most Fermilab personnel already have Kerberos accounts for single sign on.
- Lighter weight process than DOEgrids (from the users perspective).
- Short lived (1-7 day) certificates:
- kinit -n -r7d -l26h
- kx509
- kxlist -p
- voms-proxy-init -noregen -voms fermilab:/fermilab -valid 168:0
- submit grid job
- Support cron jobs through kcroninit:
-  /usr/krb5/bin/kcron <script>
- /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cron/CN=Keith Chadwick/UID=chadwick
- TAGPMA + IGTF

- FermiGrid, OSG and GIN

- 9 May 2007

## glexec

- Joint development by NIKHEF (David Groep / Gerben Venekamp / Oscar Koeroo) and Fermilab (Dan Yocum / Igor Sfiligoi).
- glexec allows a site to implement the same level of authentication, authorization and accounting for glide-in jobs on the WN as for jobs which are presented to the globus gatekeeper of the CE.
- Integrated (via “plugins”) with LCAS / LCMAPS infrastructure (for LCG) and GUMS / SAZ infrastructure (for OSG).
- glexec is currently deployed on several clusters at Fermilab.
- Will be included in Condor 6.9.x.

- FermiGrid, OSG and GIN

- 9 May 2007

## glexec block diagram

- FermiGrid, OSG and GIN

- 9 May 2007

## User Experience on FermiGrid/OSG

- VOs at Fermilab using other OSG resources:
- Our experience is not very good.
- SDSS VO - ~82 OSG sites, ~41 claim to “support”, but only 5 sites can actually run the SDSS application.
- Fermilab VO - ~82 OSG sites, ~42 claim to “support”, but only 30 sites pass the minimal acceptance tests.
- Authenticate, run already existing programs, copy file to site, copy file from site.
- The situation is slowly improving, but sites are not very responsive to problem tickets.
- VOs on OSG using Fermilab resources:
- Fermilab operates as a “universal donor” of opportunistic cycles to OSG VOs.
- In April - 18 unique OSG VOs ran on FermiGrid and used 3.9M CPU hours.
- Gratia accounting data show that ~10% of the Grid systems at Fermilab are being used opportunistically.
- But there are VOs which run into problems with our job forwarding site gateway and handling the heterogeneous cluster and sub-cluster configurations (CPU chipset - Intel vs. AMD, OS, 32 vs. 64 bit, memory, local disk and mounted filesystems).
- There are also VOs which desire significant amounts of resources beyond what we are able to opportunistically contribute.
- Hey buddy can you spare 7 terabytes of disk?  We only need it for the next year...
- VOs from other Grids using OSG/Fermilab resources:
- We participate in the GIN efforts.
- Most recently - Individuals from PRAGMA, Fermilab and OSG have demonstrated successful interoperability of PRAGMA, OSG and FermiGrid.
- Cindy Zheng, Tsutomu Ikegami, Yoshio Tanaka, Neha Sharma and Shaowen Wang
- http://fermigrid.fnal.gov/gin/pragma/osg-pragma-interoperability.html

- FermiGrid, OSG and GIN

- 9 May 2007

## fermilab VO Acceptance on OSG

- FermiGrid, OSG and GIN

- 9 May 2007

## Unique VOs on FermiGrid by Month

- FermiGrid, OSG and GIN

- 9 May 2007

## Unique VOs on OSG by Month

- FermiGrid, OSG and GIN

- 9 May 2007

## CPU Time by Organization on FermiGrid per month

- FermiGrid, OSG and GIN

- 9 May 2007

## CPU Time by Organization on FermiGrid per month (stacked)

- FermiGrid, OSG and GIN

- 9 May 2007

## CPU Time by Organization on OSG per month (stacked)

- FermiGrid, OSG and GIN

- 9 May 2007

## New Initiative - OSG User Support Campaign

- OSG has just started a campaign to address problems with grid site job submissions:
- https://twiki.grid.iu.edu/twiki/bin/view/Troubleshooting/CampaignUserRunning
- http://vdt.cs.wisc.edu/tmp/contact.pl

- FermiGrid, OSG and GIN

- 9 May 2007

## fin

- Any Questions?

- FermiGrid, OSG and GIN

- 9 May 2007