# Presentation: 856537

## GDM April 18th 2006

- Input:
- 	Project Status Reports from GDM projects
- 	(with some postponements).
- Format:
- 	Status Reports posted to DocDB.
- _[http:](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[//cd-docdb](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[.](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[fnal](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[.](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[gov/cgi-bin/DisplayMeeting](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[?](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[conferenceid=125](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)__[	](http://cd-docdb.fnal.gov/cgi-bin/DisplayMeeting?conferenceid=125)_
- 	Project Leaders present.
- 	Ruth and Don present summary and we discuss.
- End by ~4 pm

- Grid Activities

## Storage,  Security, Networks :  covered by Don

Grid Infrastructures, Workload Management,  Accounting & Metrics: covered by Ruth

- Grid Infrastructures, Workload Management,  Accounting & Metrics: covered by Ruth

- GDM Projects and Activities

- Grid Activities

## What is the scope of Grid Services

- Analysis

- Production
- Scripts

- Batch
-  Analysis

-  Disks + Farms
- Storage + Batch Queues
**Compute Elements **
**+ local Storage Elements**

- Permanent Storage
**Storage Elements**

-  **Local Network**

- Data Movement
- & Bandwidth
- Scheduling

- Job Scheduling
- & Priority

- Monitoring
- Information &
-  Accounting Meters

- Authentication
-  & Authorization

- Data/File Catalogs
- & Data Handling

- Job Queues &
- Workload Mgmt

- Information
- Catalogs &
- Repositories

- Event & Data
- Management &  Selection

- Resource
- Selection

- Workflow
- Management

- Virtual Organization
- (Physics Group)
- Administration

- Allocation &
- Accounting

- Experiment & User
- Interface &
- Frameworks

- System Monitoring,
- Information
- & Accounting

- Storage Access &
-  Management

- Security
- Infrastructure

- Grid Activities

## Action Items from April 4th GDM:

- Line management to find out what actions and projects people say are needed to reduce the current and/or projected operational load in the GDM area  by a factor of ~2. Plan to report these to GDMs as the information is available
- 	Reports from project leaders, department management ?
- Make a table of stakeholder requirements metrics  for Job Submission and Brokering.
- 	Volunteers  - Ruth, Adam, Igor?
- Make a table of stakeholder data caching current and future performance metrics.
- 	 Volunteers to help - Eileen, Ian, Doug, Amber  ?
- Dan Yocum/FermiGrid  will make the first draft of the requirements and plan for infrastructure to meet the Lab security requirements for GlideCAF. This will include contacting  GLexec developers, Igor Sfiligoi, etc. A first draft should be available about 1 week from now.
- 	Draft distributed internally. Will be distributed more widely on Thursday after meeting with Europeans.

- Grid Activities

## March ‘06 Effort Reporting (for reference)

- 1.55 Astro Grid applications
- 2.78 FermiGrid
- 2.19 Accounting project, beta release ~2 weeks away (reported effort moved from gridservices to gridservices/accounting)
- 0.72 D0 Resource Selection/Broker for OSG.
- 0.29 VO Management
- 1.06 Access Control, Authorization, Policy
- 1.87 Coordination, operations, DOE Certs etc
- 5.72 SAMGRID
- 0.99 Operations Support (including SVDT)
- 1.97 OSG and CMS Integration/deployment/testing /support
- ---
- 6.57 Lower Storage Admin/Support
- 1.75 Lower Storage Projects
- 1.36 LambdaStation
- 0.85 dCache Development
- 2.6 dCache Support + Administration
- 2.39 SRM Support and Development
**Total  19 + 15**= **34**

- Grid Activities

## Workload Management, Accounting and Metrics, Grid Infrastructures, covers work in:

- SAMGrid - status from Feb
- GlideCAF
- D0/OSG Resource Selection
- Grid Accounting
- Metrics
- FermiGrid
- OSG

- Grid Activities

## Workload Management 

- SAMGrid
- jobs run on LCG and OSG using forwarding node (SamGrid - Adam)
- Upgrading to V7 of the infrastructure.
- SAMGrid (JIM) – Soon will integrate with Runjob
- MC Requests – Technical problems with integration DB are overcome. Testing continues.
- OSG + SAM Integration – start working to incorporate ReSS for D0
- GlideCAF-
- Single site version working: Web based, all downloads verified by SHA1: Optional Squid cache can reduce network load : User has both krb5 and X509 proxy
- Still relying on CDFSOFT being on shared FS
- Krb5 binaries and compatibility libraries delivered
- Multi site version still in prototype stage
- Scalability problems found with GCB Being addressed by the Condor team
- Both versions now run all jobs under the same UID

- Grid Activities

## Workload Management cont;

- Project to make Resource Selection (RESS - Gabriele) service for D0 and other VOs on OSG.
- Milestone in August:
- CEMon Development: old_classad dialect (gLite) *(DONE)*
- ReSS Central Services development (Information Gatherer: CEMon to Condor MMS adapter) *(DONE)*
- Integration and Small scale Testing *(DONE)*
- Preparing deployment of ReSS to OSG ITB (May 06). GOAL: Production deployment on OSG v0.6.0 *(Jul 06?)*
- Integration of ReSS with FermiGrid
- End to end Deployment for DZero: SAM-Grid integration with OSG *(By Aug 06).*
- Discussing extensions of ReSS for ATLAS and LIGO
- Joint document: GLUE schema to OLD classad mapping (ATLAS, DZero, FNAL, UIowa, GLUE)
- Future directions (after Aug)
- Production deployment and operations of SAM-Grid / OSG interoperability system.
- Modifications to ReSS for ATLAS, LIGO, et al.
- Complete by FY07?

- Grid Activities

## Accounting - US CMS, Fermilab and OSG

- Goal to Deploy V0 3 months ago! Nixed by staff leaving.
- Include capabilities of existing Fermi/CD farm accounting.
- Grid + Fermi Prototype being integrated/deployed now:
- Collector and Reporting are in place
- Will develop more reports once the condor probe is in regular use
- Final touch to the condor probe
- PSACCT probe should be available in a week or so.
- Ongoing Tasks:
- Deploy at BNL
- Enable the authentificated connection
- Implement the missing data detection
- configuration and monitoring of probe
- General Deployment (in collaboration with VDT team).
- Consolidate condor probe
- Implement/request Role information
- Implement/request way of determining the job history/provenance

- Grid Activities

## Effort and Timeline

- Distribution of v0.0 in VDT *- next few weeks*
- General distribution of v0.1 in VDT *- end of June*
- Deploy Gratia Prototype in Open Science Grid - *July 2006*
- Support use of Gratia the >40 OSG sites - *September 2006*
- Replacement of the Computing Division Farm Usage Reports  *- June 2006*
- Integrate new probes for PBS and LSF into Gratia  - *Sept 2006*
- Verify, Analyze, Cross-check the data *(on going)*
- Staffing
- 2 1⁄4 FTE until end of September
- likely not including time to develop PBS and LSF probes
- Ramp down from September to December
- Production maintenance requirements not known yet (but planned to be *low*)

- Grid Activities

## Monitoring and Metrics

- FermiGrid doing good “service access” metrics.
- Accounting will give us job metrics
- CAF monitoring pages
- SAMGrid use SAMTV. New MIS is being deployed on Clued0 for live shakedown
- How do we present and review ?
- Will need a concerted effort to review and audit these?

- Grid Activities

## Slide 12

- Grid Activities

## OSG Monitoring/Metrics

- Default currently is MonaLisa information.
- _[http://grid02.uits.indiana.edu:8080/reports/daily/20060418/index.html](http://grid02.uits.indiana.edu:8080/reports/daily/20060418/index.html)_
- Gathering existing information from all VOs and Sites
- _[http:](http://osg.ivdgl.org/twiki/bin/view/Deployment/MetricsPlace)__[//osg](http://osg.ivdgl.org/twiki/bin/view/Deployment/MetricsPlace)__[.](http://osg.ivdgl.org/twiki/bin/view/Deployment/MetricsPlace)__[ivdgl](http://osg.ivdgl.org/twiki/bin/view/Deployment/MetricsPlace)__[.org/twiki/bin/view/Deployment/MetricsPlace](http://osg.ivdgl.org/twiki/bin/view/Deployment/MetricsPlace)_
- Operations also gathering metrics on support issues.

- Grid Activities

## Experiment / VO  Grids 

- D0 / CDF / Minos SAMGrid and CMS
- Reported in previous and future GDMs and CMS Coordination meetings.
- Patriot starting to run on FermiGrid, (OSG?)
- LQCD grid shows up on OSG but supports single VO

- Grid Activities

## Grid Infrastructure: SAMGRid

- V7 upgrade at D0:
- Online - Sucessful tests. More testing to do with full system
- SAMGrid (JIM) – Soon will integrate with Runjob
- MC Requests – Technical problems with integration DB are overcome. Testing continues.
- Oracle 10g Testing
- Testing and Monitoring
- Incorporating Station Tests into Test Harness
- Writing Station Testing Requirements Document
- DBServer
- Randolph Herber is scaling the DB Server learning curve. Will address some small improvement needs first
- Longer term study of DB Server performance and improvements under consideration
- GRID
- Andrew has WBS for SRM + SAM Integration
- LCG + SAM Integration deployment to production complete. Will integrate into operations

- Grid Activities

## Grid Infrastructures : FermiGrid

- Storage Element:
- The FermiGrid OSG storage element FNAL_FERMIGRID_SE has been commissioned as part of the public dcache.  The IA group is in the process of adding the necessary dcache user accounts.
- Requisition CD99397, totaling ~$64K, for 24 Tbytes of HDS SATA disk for attachment to the CD BlueArc NFS server appliance was submitted for approvals on Wednesday April 12, 2006._x0013_
- FermiGrid
- Recoded SAZ implementation is almost ready to go. Deployment Plan:
- Have a formal code review.
- Install SAZ server on FermiGrid4
- Install SAZ client on fgtest1-5 and test for at least a week.

- Grid Activities

## FermiGrid roadmap

- Grid Activities

## FermiGrid – Milestones, Time Line, Effort

- Apr 2006:	CDF Glide-CAF Integration.
- May 2006:	SAZ Testing & Deployment.
- 	glexec software design.
- 	Participation in OSG 0.3.6, 0.3.7 ITB, OSG 0.4.1 release.
- 	Revised FermiGrid border gateway job forwarding mechanism using OSG cemon and resource selection.
- 	FermiGrid OSG SE & public dcache deployment.
- June-July 2006	Participation in OSG 0.5.0 ITB, OSG 0.6.0 release.
- 	glexec coding, test and deployment.
- 	BlueArc storage integration.
- Ongoing:	Service redundancy / high availability configuration.
- 	Work with other Fermilab resources to integrate them into FermiGrid.
- 	Work with OSG to validate FermiGrid interoperability.

- Grid Activities

## Grid Infrastructures: GlideCAF

- Support Issues for the Future?
- 	FNAL On-site: Doug Benjamin
- 	Offsite: ?

- Grid Activities

## Grid Infrastructures: Open Science Grid

- Grid Activities

## OSG Roadmap

- OSG 0.4.1 about to be released.
- Addressing headnode overload problems
- Reducing effort on “provisioning”
- Security patches
- Full support for WLCG Job Submission
- OSG 0.6.0
- Will support updates
- Include Accounting, RESS, - Fermilab deliverables

- Grid Activities

## OSG cont. Recognised by WLCG

- Grid Activities

## OSG Status

- Submitted the same OSG proposal  to NSF (January) and DOE SciDAC-2 (March)
- Scope of program of work is
**Sustaining a Production Distributed Facility** including: Operations and maintenance; Distribution and testing of common middleware; A scalable integration grid; Inter-operation with EGEE, TeraGrid and Campus’s; and Engagement.
**Education, Outreach and Training** including: Technical training of administrators and support personnel; Grid schools; Attention to areas with reduced resources (network, storage, compute) and expertise.
**Science driven Extensions** including: Delivering to LHC requirements and schedule; Meeting needs of LIGO, Run II, Biology stakeholders; Interoperability to support global collaboration.
- New organization structure reflecting program of work with Council Chair (Bill Kramer, NERSC), Executive Director (Ruth Pordes, Fermilab), Facility Manager (Miron Livny, UWisconscin), etc. now in effect.
**Consortium continuing its work to meet users needs.**

- Grid Activities

## Over to Don...

- Grid Activities