# Presentation: 819289

## FermiGrid

- Eileen Berman
- (for the FermiGrid Team)
- Fermilab
- Work supported by the U.S. Department of Energy under contract No. DE-AC02-07CH11359.

## Overview

- What Is FermiGrid?
- Running a Job on FermiGrid
- Job Accounting
- Operational Experience
- The Future
- Conclusion

- FermiGrid

- June 6, 2007

## What is FermiGrid?

- FermiGrid brings previously disjoint resources into a common infrastructure where the user communities retain control of “their” resources and also allow negotiation for opportnistic sharing.

- FermiGrid

- June 6, 2007

## What is FermiGrid?

- The Fermilab campus Grid and Grid portal.
- The site globus gateway.
- Accepts jobs from external (to Fermilab) sources and forwards the jobs onto internal clusters.
- A set of common services to support the campus Grid and interface to Open Science Grid (OSG) / LHC Computing Grid (LCG):
- VOMS, VOMRS, GUMS, SAZ, MyProxy, Squid, Gratia Accounting, etc.
- A forum for promoting stakeholder interoperability and resource sharing within Fermilab:
- CMS, CDF, D0;
- ktev, miniboone, minos, mipp, etc.

- FermiGrid

- June 6, 2007

## Software Stack

- Baseline:
- Scientific Linux 3.0.x, 4.x, 5.0
- OSG 0.6.0 (VDT 1.6.1, Globus Toolkit 4, WS-Gram, Pre-WS Gram)
- Additional Components:
- VO Management Service - VOMS
- VO Membership Registration Service - VOMRS
- Grid User Mapping Service - GUMS
- Site AuthoriZation Service - SAZ
- job forwarding job manager - jobmanager-cemon
- credential storage - MyProxy
- web proxy cache - Squid
- auditing - syslog-ng
- accounting - Gratia
- Virtualization - Xen

- FermiGrid

- June 6, 2007

## Hardware

- FermiGrid core systems:
- dual 3.6GHz Intel XEON
- 4GB of PC2100 DDRAM
- 2 mirrored 10K rpm SCSI system disks
- a single gigabit Ethernet connection to the LAN
- Cluster Worker Nodes:
- Heterogeneous commodity computers

- FermiGrid

- June 6, 2007

## Compute Elements and Worker Nodes

- CDF Gatekeeper

- D0 Gatekeeper

- D0 Gatekeeper

- CMS Gatekeeper

- CMS Gatekeeper

- GP Gatekeeper

- FermiGrid Site Gateway

- CDF
- WNs

- ~ 1140 WNs,
- ~6000 VMs

- D0
- WNs

- D0 Gatekeeper

- ~ 1200 WNs,
- ~4800 VMs

- GP
- WNs

- GP Gatekeeper

- ~ 220 WNs, ~600 VMs

- CMS WNs
- Operated by CMS

- ~ 580 WNs, ~1400 VMs

- CMS Gatekeeper

- CDF Gatekeeper

- WN: Worker Node
- VM: Virtual Machine

- FermiGrid

- June 6, 2007

## Batch Systems

- Condor:
- This is the dominant batch system at Fermilab.
- Is in use on the GP, CDF and CMS tier 1 Grid Clusters.
- PBS:
- Is in use on the D0 farms and D0 Grid Clusters, and on LQCD.
- It gives us an alternative and/or competitor to Condor...
- Sun Grid Engine:
- No direct experience at Fermilab - yet.
- We will be commissioning a small batch system test cluster with SGE later this year.

- FermiGrid

- June 6, 2007

## Storage

- BlueArc NFS Shared File Systems:
- Purchased 24 Tbytes (raw) = 14 Tbytes (formatted RAID 6)
- NFS server appliance (fiber channel)
- Failover supported
- Currently mounted on (most) FermiGrid clusters
- Temporary storage for job data, applications
- Public dcache (FNAL_FERMIGRID_SE):
- 7 TBytes of storage accessible via Storage Resource Manager.
- Access to permanent storage tape robots
- For FY07:
- Plan to closely monitor needs, user requests and utilization.
- Do not currently expect to request additional storage until FY08.

- FermiGrid

- June 6, 2007

## Virtual Organizations

- FermiGrid supports many diverse Virtual Organizations
- ... a group of individuals or institutions who share the computing resources of a "grid" for a common goal. (Wikipedia)
-  often equates to an experiment

- FermiGrid

- June 6, 2007

## Virtual Organizations

- Resource Providers (sites) or Grids establish trust relationships with a VO
- Contracts between Resource Providers & VOs govern resource usage & policies (site and VO policies)
- A VO Manager is designated by a VO to validate/approve new VO members
- A VO's member structure may include groups, subgroups and/or roles into which it divides its members according to their responsibilities and tasks
- Allows Grid level scalability for membership registration, site account management, etc.

- FermiGrid

- June 6, 2007

## VOs Hosted at Fermilab

- FermiGrid

- June 6, 2007

## Job Submission

- FermiGrid

- June 6, 2007

## Job Submission

- Obtain a certificate from an OSG recognized Certificate Authority
- DOE Grids, Fermilab Kerberos Certificate Authority
- All IGTF recognized CAs
- Register as part of a VO (Once) **(1)**
- VO Membership Registration Service (VOMRS).
- Used by individuals to apply for membership in a VO, VO subgroup, and/or Role.
- Workflow manages the approval or rejection of the individual’s application.
- Push VOMRS configuration into VOMS server **(2)**
- VO Management Service.
- Authority for DNs in VO & authorized groups and/or roles (Fully Qualified Attribute Name : FQAN).

- FermiGrid

- June 6, 2007

## Job Submission

- Periodic synchronization of VOMS information by Site GUMS Service **(3)**
- Grid User Mapping Service.
- List of VOMS servers to query is configured by the system administrator.
- Maps DN+VO extended attributes to site specific UID/GID.

- FermiGrid

- June 6, 2007

## Job Submission

- User obtains VOMS proxy certificate **(4)**
- Contains X.509 “grid” certificate  and VOMS attributes (Roles, Groups, VO)
- i.e., voms-proxy-init -voms fermilab:/fermilab/usminos/Role=softadmin
- Single initial certificate used when joining multiple VOs
- User submits job **(5)**
- condor_submit, globus-job-run, globus-job-submit, etc.

- FermiGrid

- June 6, 2007

## Job Submission

- Callout to Site Authorization Service (SAZ) **(6)**
- Site Whitelist/Blacklist Service
- Returns Allow/Deny
- Registration automatic at 1st grid job (stores DN, VO, Role, CA) (default-accept)
- voms-proxy-init vs. grid-proxy-init:
- default allow access to users who present proxy certificate generated from voms-proxy-init
- On a case-by-case basis, users who present a proxy certificate generated from grid-proxy-init can be allowed access manually
- Allows the Fermilab security authorities to impose a site-wide grid access policy
- Allows for rapid temporary suspension of a DN, VO, Role, Group, or CA during incident investigation

- FermiGrid

- June 6, 2007

## Job Submission

- Callout to Grid User Mapping Service (GUMS) **(7)**
- Returns local UID or null
- Many-to-One, One-to-One, One-to-Self Mapping Types
- The mapping scheme that is used at a Site is a decision of the site administrators based on the request of the VO using the site. At Fermilab, the [us]CMS VO has requested that users be mapped one-to-one, while the  D-Zero VO is using the many-to-one mapping.
- Implemented as a Java Servlet and run in Tomcat.
- Developed at Brookhaven  National Laboratory (BNL)

- FermiGrid

- June 6, 2007

## GUMS Mapping Types

- many-to-one:
- All members of the same VO having the same extended key attributes are mapped to a single local UID.
- In OSG the many-to-one mapping is an acceptable usage case.
- With many-to-one mappings it is possible to obtain the proxy of another user running on the same batch system.
- Currently, the OSG “prevents” this through policies set in the Acceptable Usage Policy (AUP) document that all OSG users must electronically sign when they register with their VO.
- one-to-one:
- A set of pool accounts is created on the batch system and GUMS maps a user DN with extended key attributes to a single, specific pool account.
- Maintained in the GUMS database for each consecutive request for mapping for that user+attributes+site
- –	Multiple users in the same VO cannot obtain another user’s proxy certificate, nor can they accidentally delete or overwrite another user’s data.
- Easier to trace a rogue job in the queue to the originating user.
- The site GUMS administrator must monitor the usage of the pool accounts, and the system administrators must make sure that there are enough pool accounts available on each worker node (WN) in the batch system.
- one-to-self:
- A user is mapped to their own local UID on the batch system.
- Used at Brookhaven National Laboratory (BNL).

- FermiGrid

- June 6, 2007

## Job Submission

-  Job matched to suitable cluster **(8)**
- Resource Selection Service (ReSS) uses the Condor Match-making System to match the job requirements specified by the job against information gathered by gLite CEMon information system
- Jobs matched against the various resources available *at the point in time* that the job was submitted
- Must have submitted the job to the fermigrid1 central gatekeeper
- By default, users job will be matched against clusters which support the users VO and have at least one free slot available
- Users have the ability to add additional conditions to this "requirements” attribute, using the attribute named "gluerequirements" in the condor submit file specified in terms of Glue Schema attributes.

- FermiGrid

- June 6, 2007

## Pilot Jobs

-  Pilot (glide-in) Jobs **(9)**
- Place holder jobs in the batch queue.
- “Call home” when they land on a WN to get the real, User Job
- The owner of the Pilot Job may be authorized to run but the owner of the User Job may not be
- glexec – a suexec derivative (from Apache)
- glexec is a mini gatekeeper.
- Uses the proxy certificate of the owner of the User Job and checks it against SAZ **(6)** and GUMS **(7)**
- Execs the job as the correct local UID, protecting the Pilot Job **(10)**

- FermiGrid

- June 6, 2007

## gLExec

- Joint development by NIKHEF (David Groep / Gerben Venekamp / Oscar Koeroo) and Fermilab (Dan Yocum / Igor Sfiligoi).
- glexec allows a site to implement the same level of authentication, authorization and accounting for glide-in jobs on the WN as for jobs which are presented to the globus gatekeeper of the CE.
- Allows a VO to prioritize jobs to its own policies without sacrificing security.
- Based on the LCAS/LCMAPS infrastructure (from EGEE) and integrated with GUMS/SAZ via plugin infrastructure for OSG.
- glexec is currently deployed on several clusters at Fermilab.
- glexec, with the GUMS plugin, is scheduled to be part of the next VDT release.

- FermiGrid

- June 6, 2007

## Accessing Storage

-  Mass Storage access
- Access to disk Cache (dCache) available via gridftp or Storage Resource Manager (SRM) **(8)**
- Dynamic space allocation
- File management functionality
- dCache, in collaboration with DESY is one of the leading, recognized large scale, high performance storage solutions for the LHC and other High Energy Physics experiments
- Uses gPLAZMA to interface to GUMS **(7)**
- Authorized access to tape storage

- FermiGrid

- June 6, 2007

## Gratia Accounting

- Robust, scalable, and accurate grid accounting service for OSG.
- Interoperates with EGEE accounting system.
- Utilizes a “store locally” and “forward” model to a central collector (repository).
- Assure that accounting data is not lost.
- Sites can operate a local site Gratia collector which summarizes site data and then forwards the summaries to the OSG-wide Gratia collector.
- Gratia “probes” interface between the Globus Gatekeeper and the site specific batch system to capture the accounting data.
- Condor, PBS, SGE, glexec, dCache, etc.
- psacct-probe captures system level information
- Running since Sept 2006, 9.3Million Records, current accretion rate of 90K batch records/day, 30GB (no storage records yet... soon)
- Effort now turning toward understanding the data being collected.
- Generation of “standard” dashboard accounting plots.
- Supporting ad-hoc queries against the Gratia accounting data

- FermiGrid

- June 6, 2007

## Operational Experience

- It works... well!
- Typical week in February, 2007:
- Over 110,000 jobs submitted by 18 VOs to 5 separate clusters.
- On one day:
- A peak of about 450,000 GUMS mappings occurred (~5Hz)
- System load was nominal at ~1-3 on 1, 5, and 15 min averages

- FermiGrid

- June 6, 2007

## User Experience on FermiGrid/OSG

- VOs on OSG using Fermilab resources:
- Fermilab operates as a “universal donor” of opportunistic cycles to OSG VOs.
- Gratia accounting data show that ~10% of the Grid systems at Fermilab are being used opportunistically.
- But there are VOs which run into problems with our job forwarding site gateway and handling the heterogeneous cluster configurations.
- There are also VOs which desire significant amounts of resources beyond what we are able to opportunistically contribute
- VOs from other Grids using OSG/Fermilab resources.
- We participate in the GIN (Grid Interoperability Now) efforts.
- Most recently - Individuals from PRAGMA have been able to make successful use of Fermilab Grid computing resources.

- FermiGrid

- June 6, 2007

## Unique VOs on FermiGrid

- Unique VOs

- FermiGrid

- June 6, 2007

## CPU Time per Organization on FermiGrid

- CPU  Hours

- FermiGrid

- June 6, 2007

## SAZ Operational Experience

- Monitors of the operational Site Authorization Service (SAZ) (client and server) appears to indicate that SAZ will be able to scale to the expected number of calls/day necessary to support the full deployment at Fermilab (all CEs + WNs)x(# of jobs/day).

- FermiGrid

- June 6, 2007

## Site AuthoriZaton Service (SAZ)

- FermiGrid

- June 6, 2007

## GUMS Scaling

- FermiGrid

- June 6, 2007

## The Future

- FermiGrid-High Availability Upgrades
- Linux-HA
- Active-Active or Active-Standby depending on the service.
- XEN
- Virtualization of services.
- These technologies will be used for:
- FermiGrid-HA Site Globus Gatekeeper
- Including Web Services GRAM.
- FermiGrid-HA Services
- VOMS, GUMS, SAZ, etc.
- FermiGrid-HA vobox/edge services
- FermiGrid-HA grid development platforms

- FermiGrid

- June 6, 2007

## Auditing Grid Service

- Allow assessment of overall security condition across sites and VOs
- Provides forensic analysis tools for security investigations
- Complementary to the site specific security processes
- Currently in design phase

- FermiGrid

- June 6, 2007

## Conclusion

- FermiGrid is a complex, robust, and scalable grid gateway system built using commodity hardware and community provided software.
- 15,000+ jobs submitted per day
- 400,000+ user mappings per day
- Movement of 100's of terabytes to/from mass storage
- FermiGrid Web Site & Additional Documentation:
- http://fermigrid.fnal.gov/

- FermiGrid

- June 6, 2007

## Conclusion

-                         Questions??

- FermiGrid

- June 6, 2007

## FermiGrid Strategy

- Strategy:
- In order to better serve the entire program of Fermilab, the Computing Division has undertaken the strategy of placing all of its production resources in a Grid "meta-facility" infrastructure called FermiGrid.
- This strategy is designed to allow Fermilab:
- to insure that the large experiments who currently have dedicated resources to have first priority usage of those resources that are purchased on their behalf.
- to allow opportunistic use of these dedicated resources, as well as other shared Farm and Analysis resources, by various Virtual Organizations (VO's) that participate in the Fermilab experimental program and by certain VOs that use the Open Science Grid (OSG).
- to optimise use of resources at Fermilab.
- to make a coherent way of putting Fermilab on the Open Science Grid.
- to save some effort and resources by implementing certain shared services and approaches.
- to work together more coherently to move all of our applications and services to run on the Grid.
- to better handle a transition from Run II to LHC in a time of shrinking budgets and possibly shrinking resources for Run II worldwide.
- to fully support Open Science Grid and the LHC Computing Grid and gain positive benefit from this emerging infrastructure in the US and Europe.

- FermiGrid

- June 6, 2007

## Timeline and Effort

- The FermiGrid concept started in mid CY2004 and the FermiGrid strategy was formally announced in late CY2004.
- The initial hardware was ordered and delivered in early CY2005.
- The Initial core services (Globus Gateway, VOMS and GUMS) based on OSG 0.2.1 were commissioned on April 1, 2005.
- We missed the ides of March, so we chose April Fools Day.
- Our first site gateway which used Condor-G matchmaking and MyProxy was commissioned in the fall of CY2005.
- Job forwarding based on work by GridX1 in Canada (http://www.gridx1.ca).
- Users were required to store a copy of their delegated grid proxy in our MyProxy repository prior to using the  job forwarding gateway.
- OSG 0.4 was deployed across FermiGrid in late January-February 2006.
- Followed quickly by OSG 0.4.1 in March 2006.
- The Site AuthoriZation (SAZ) service was commissioned on October 2, 2006.
- Provides site wide whitelist and blacklist capability.
- Can make decision based on any of DN, VO, Role, and CA.
- Currently operate in a default accept mode (providing that the presented proxy was generated using voms-proxy-init).
- The glexec pilot job glide-in service was commissioned on November 1, 2006.
- Provides authorization and accounting trail for Condor glide-in jobs.
- The latest version of the site job forwarding gateway (jobmanager-cemon) was commissioned in November 2006
- Eliminated the need to utilize MyProxy via “accept limited” option on the gatekeeper.
- Based on CEMon and OSG RESS, Condor Matchmaking.
- Periodic hold and release functions were added in March 2007.
- OSG 0.6.0 was deployed across FermiGrid during March & April 2007.
- Average effort over this period = ~3 FTE.

- FermiGrid

- June 6, 2007

## Authorization & Authentication

- DOEgrids Certificate Authority:
- Long lived (1 year) certificates.
- Heavy weight process (from the perspective of the typical user...).
- Fermilab Kerberos Certificate Authority:
- Service run on Fermilab Kerberos Domain Controllers.
- Most Fermilab personnel already have Kerberos accounts for single sign on.
- Lighter weight process than DOEgrids (from the users perspective).
- Short lived (1-7 day) certificates:
- kinit -n -r7d -l26h
- kx509
- kxlist -p
- voms-proxy-init -noregen -voms fermilab:/fermilab -valid 168:0
- submit grid job
- Support cron jobs through kcroninit:
-  /usr/krb5/bin/kcron <script>
- /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=cron/CN=Keith Chadwick/UID=chadwick
- TAGPMA + IGTF

- FermiGrid

- June 6, 2007

## Certificates & Mappings per Day

- FermiGrid

- June 6, 2007

## GUMS Mappings - CMS

- FermiGrid

- June 6, 2007