# Presentation: 268032

## Progress on Release, API Discussions,Vote on APIs, and PI mtg

**Progress on Release, API Discussions,****Vote on APIs, and PI mtg**

- Al Geist
- January 14-15, 2004
- Chicago, ILL

## Participating Organizations

**Coordinator: Al Geist**

**Participating Organizations**

**ORNL**
**ANL**
**LBNL**
**PNNL**

**PSC**
**SDSC**
**IBM**
**SGI**

**SNL**
**LANL**
**Ames**
**NCSA**

**Cray**
**Intel**
**Unlimited Scale**

**Participating Organizations**

**Changes**

## Scalable Systems Software

- IBM
- Cray
- Intel
- SGI

**Scalable Systems Software**

**Participating Organizations**

- ORNL
- ANL
- LBNL
- PNNL

- NCSA
- PSC
- SDSC

- SNL
- LANL
- Ames

- Collectively (with industry) define standard interfaces between systems components for interoperability
- Create scalable, standardized management tools for efficiently running our large computing centers

**Problem**

**Goals**

- Computer centers use incompatible, ad hoc set of systems tools
- Present tools are not designed to scale to multi-Teraflop systems

**www.scidac.org/ScalableSystems**

**To learn more visit**

## Potential Impact of Project

- Fundamentally change the way future high-end systems software is developed and distributed
- Reduced facility management costs
-  reduce need to support ad hoc software
-  better systems tools available
-  able to get machines up and running faster and keep running
- More effective use of machines by scientific applications
-  scalable launch of jobs and checkpoint/restart
-  job monitoring and management tools
-  allocation management interface

## Scalable Systems Software Suite

**Grid Interfaces**

**Accounting**

**Event **
**Manager**

**Service**
**Directory**

**Meta**
**Scheduler**

**Meta**
**Monitor**

**Meta**
**Manager**

**Scheduler**

**Node State**
**Manager**

**Allocation**
**Management**

**Process **
**Manager **

**Usage**
**Reports**

**Meta Services**

**System &**
**Job Monitor**

**Job Queue**
**Manager**

**Node**
**Configuration **
**& Build**
**Manager**

**Standard XML**
**interfaces**

**Working Components and Interfaces (bold)**

**authentication **
**communication**

**Components written in any mixture of C, C++, Java, Perl, and Python can be integrated into the Scalable Systems Software Suite**** **

**Checkpoint /**
**Restart**

**Scalable Systems Software Suite**

**Validation & Testing**

**Hardware**
**Infrastructure**
**Manager**

**First Release at SC2003**

**Packaging**
**&**
**Install**

## Scalable Systems Software Center

- September 11-12
- Washington DC

**Review of Last Meeting**

- Details in
- Main project notebook

## Highlights from Sept. mtg

**Rusty Lusk **– Using SSS as the production systems software on Chiba City for a number of months now. Use restriction syntax for everything. Got blessing of ANL sysadmin group.
**Scott Jackson** – Standard  Error reporting and codes across components. Discuss dividing up code space in consistent way.
**Eric  Debenedictus** –  Issues for peta-scale systems
- Redstorm and Bluelight mesh rather than switch means that topology is important consideration for SSS to consider:
- XML attribute to specify topology and I/O resources
- XML attribute to specify data arrangement on disk
- OS functionality hints to help auto placement
**Thomas Naughton** – SSS deployment using OSCAR
- A release of OSCAR that contains all SSS software
- Roll SSS components into OSCAR packages – RPM format
- Create repository for OSCAR package uploads

## Highlights from Sept. mtg (cont.)

**Al Geist **– Plans for SC2003
**Working Group Leaders** –
- What areas their working group is addressing
- Progress report on what their group has done
- Present problems being addressed
- Next steps for the group
- Discussion items for the larger group to consider
**Long Term Strategy – **
- Get Computer Centers involved and using suite
- Get vendors to be compliant with APIs

- Slides can be found in Main Notebook

## Consensus and Voting:

**Communication Infrastructure Spec**
- Wire protocols – need to add security envelope protocol
- Added service location. Bootstrapped using /etc/sss/
- Vote to Accept as spec for
- Wire Protocol definition to get new ones accepted
- Service Directory interface
- Event Manager interface
- Second vote: 16 yes   2 abstaining   0 no
- Agreement for having **common error objects** with 3 digit codes and messages. Message is human readable string.
- Two special ones 000 success 999 unknown
- Straw vote: 15 no 1 Abs 0
- Al suggests these general error classes –
- 	success, warning, temp failure, partial failure, failure
- People need to come up with counter proposal if they care

## Scalable Systems Software Center

- September-January

**Progress Since Last Meeting**

## Systems Software Suite Release

**Open Source License **– Fred asks that we come up with one general text that all organizations can agree on and then he will bless it. ***DONE***
**SSS-OSCAR** – Packaging done of all components (working around those components with license issues)
**First Release **– Announced at SC2003.
-     Available from project web site
-     **www.scidac.org/ScalableSystems**

## SC2003 Scalable System Demos and Talks

- Rusty – fancy dancing meatball in wxpython
- Thomas – SSS-OSCAR working
- Will – fancy graphic  demonstration of APITest ????
- Brett – demonstrate swapping components in SSS architecture
- Paul – chkpoint interacting with PM on chiba
- Locations: All Across the show floor
- SciDAC booth – Talks by Rusty, Craig
- OSCAR BOF on Tuesday 5:00-6:00 mentions SSS-OSCAR

## Five Project Notebooks

- A main notebook for general information
- And individual notebooks for each working group
- Over 297 total pages – 16 added since last meeting
- BC and PM groups need to get specs into their notebooks
- Add Telecom meeting notes even if short

- Get to all notebooks through main web site
**www.scidac.org/ScalableSystems**
- Click on side bar or at “project notebooks” at bottom of page** **

## Bi-Weekly Working Group Telecoms

**Starting back up after Holidays**

**Resource management, scheduling, and accounting**** **
**     **Tuesday 3:00 pm (Eastern)**  1-800-664-0771   keyword “SSS mtg”**
**Validation and Testing Group**
**     **No need for telecoms recently
**Proccess management, system monitoring, and checkpointing **
**     **Thursday 1:00 pm (Eastern) ** 1-877-252-5250  mtg code 160910**
**Node build, configuration, and information service**
**     **Thursday  3:00 pm (Eastern)  **1-888-469-1934  mtg code ****(changes)**

## Scalable Systems Software Center

- January 14-15, 2004

**This Meeting**

## Major Topics this Meeting

**Stability of Systems Software Suite **– first release is out. Are we ready for a more robust second release
**Large Scale test run **– NCSA has dedicated some time tonight to run our suite on their 1250 dual node cluster
**Quarterly Report Due **–  would like to get one to Fred by end of January. Will need text from WG leaders.
**Formal API presentations and voting**  - it is that time in the project when we are finalizing on some APIs.
**SciDAC PI Mtg**  - March 22-24 in Charleston SC. We will need poster(s), talk, and 2 page summary document

## Agenda - January 15

-   8:30     Al Geist – Project Status.
-   9:15     Thomas Naughton – SSS OSCAR software suite release
-   **Working Group Reports**
**Progress report on what their group has done **
**API Proposals for adoption by the group**
**Progress on software suite improvements**
-   9:30	Narayan Desai – Node Build, Configure
- 10:30     Break
- 11:00 	Will McClendon – Validation and Testing
- 12:00     Lunch  (on own – cafeteria room B)
-   1:00     Paul Hargrove – Process Management
-   2:00 	Scott Jackson – Resource Management
-   3.00      Break
-   3:30      Narayan - Review of "restriction syntax" style of XML
-   4:00	 Rusty - Discussion of restriction syntax for scheduler and queue mgr
-   4:30      Craig – Brief on on big testbed run
-   5:00      Eric – competitive system to SSS
-   5:30      Adjourn
-   Evening Working groups may want to help with large NCSA test run

## Agenda – January 16

-   8:30     **Discussion, proposals, votes**
- 	Rusty -  Process Manager API (discussion/vote)
-               Narayan -  Node state API (discussion/vote)
-               Scott – Allocation Manager API (discussion/vote)
-               Brett – Queue manager API (discussion/vote)
-               Scott – SSSRMAP interface
-               Al - Progress report
-               Al - SciDAC mtg 2 pager, posters, talks
**	****	**
- 10:30     Break
- 11:00     Al Geist – Summary
-               SciDAC PI Mtg March 22-24, Charleston SC
-               next meeting date: May 13-14
- 	location: Argonne
- 12:00     meeting ends

## Meeting notes

- Al presents his slides
- Thomas Naughton – SSS deployment using OSCAR
- Good – RPMs created for all SSS components!
-    OSCAR packaging (varying levels)
-    SourceForge project supplied central CVS location
- Bad – not all scripts are created equal – new untested submissions
-    Some pain getting SF accounts.
-    Time constraints forced script hacks
- OSCAR testing framework
- Status – Tarball available fairly toxic but builds full working cluster w/ SSS
-    Updated OSCAR pkg HowTo
- ToDo – clean up hacks, integrate remaining SSS components (qbank)
-    Add SSS interface to OSCAR itself
- Would like to establish release schedule – March 1
- Not clear that anyone has downloaded yet
- Discussion of how many orgs in our group could shakedown the tarball
- Group feels better to have few very reliable components than all components

## Meeting notes

- Narayan – node build progress report
- Only had a few minor bug fixes
- Infrastructure has been reliable for 6 month
- Library updates:
-      Portability - OSX support, 64-bit  tested, Tru64 support
-      Thread-safety
-      SSL wire-protocol module – soon to be the default protocol in ssslib
- Node state manager – reliable
- Build System – building vs configuration interface/conflict issues
- Hardware infrastructure – model needs refinement WRT topology info
- Restriction Syntax augmentations
-      New operators added – negations, numeric, regular expression
-      Integrated into all python components
- Next steps – work on new models for hardware infrastructure
-      Work on multiple implementations of BCM components
-      Performance tuning – for ssslib, event manager, service directory

## Meeting notes

- Will McLnedon – Component Interface testing report
- Description of his work for the new folks
- SC2003 demo of APItest v.1 in ASCI booth (GUI HTTP interface)
-     built on Twisted Framework _[www.twistedmatrix.com](http://www.twistedmatrix.com/)_
-     Db interfacing, distributed component testing, HTTPD mode
- APItest development. Lessons learned. V.2
-     new test file formats – collab with Jackson
-     separate individual tests from batch grouping
-     Runs through some examples.
- Feedback is encouraged
- Hope to get some real test suites going this quarter
- Ron Oldfield – introduced
- Shows graphical APItest demo that was given at SC2003

## Meeting notes

- Paul Hargrove – Process management report
- SSS-OSCAR release
- Coming to a point where components have to interact more eg. Chkpt
- Real deployment/testing on Chiba (ANL), XTORC (ORNL)
- Checkpoint manager – progress
-    ported to RH9 (hard – Red Hat kernel’s...)
-    checkpoint using LAM/MPI
-    stand-alone package w/ LAM/MPI for chkpt
-    suspend/resume interface working with queue manager
- Outstanding issues – need to design restart-time interactions
-    need to implement a full interface
-     - restriction syntax, event generation, error reporting
-    basic ideas on file management
- Monitoring progress
-   in SSS-OSCAR
-   Scalability work – thread pool, internal protocol changes
-   fix  service directory connections
-   write documentation

## Meeting notes

- Process manager (cont)
- Rusty Lusk – Process Manager functionality overview
- Show Schematic of process management components
- Various commands that are in the syntax
- Progress – already a stable component, fixed several bugs at SC03
- Improved queries and error codes
- Future
-     INTEGRATION!  Stable software makes this possible
-     Chiba production use has forced the issue
-     Continued development

## Meeting notes

- Scott Jackson – Resource Manager report
- Short overview for new attendees
- Progress – released in SSS-OSCAR Bamboo, Maui, Gold, Warehouse
-   Updated RM web page for new components being available
-   Deployed user oriented problem response system
-   Created SSSRMAP C-implementation module
-   Completed per-component interface documents
- Schedule Progress
-   - Completed chkpt/restart based SSS calls.
-     blocked until can test with checkpoint guys
-   - support for dynamic jobs
-     blocked until support provided in PM and QM
-     discussion of feature of dynamic jobs how/if we should work on it
-   - resource limit enforcement and tracking
-     need rusage on process exit
-     blocked until support from PM and QM progress
- Too much blocking seems RM group lacks coordination with other groups.

## Meeting notes

- Scott Jackson – Resource Manager report (cont)
- Initial release of Bamboo and wrote API document
- Accounting and allocation
- Qbank was an initial solution replaced by Gold
- Gold – released under BSD open source licence
-   packaged as tarball. And initial OSCAR rpm created
-   added support for Service Directory registration
-   implemented status codes
-   implemented instance-level role-based authorization
- Gold running on 11 TF cluster at PNNL
- GUI improved to include user, project, machine management views
- Meta-scheduler – added thread support
-   improved Silver installation procedure
-   testing of (grid level) data staging
- Future- draft of SSSRMAP v3 protocol spec (chunking)
-   release alpha versions of Bamboo, Maui, Gold, Warehouse
-   complete design spec documents for above components.

## Meeting notes

- Discussion of having two XML syntax styles (functional, object)
- Al says he would like to see one common one across the suite
- that he didn’t care which one as long as the whole group could agree.
- Rusty brought up a second issue, wire protocol, and having a
- single library that has all the protocols used by the components in the
- SSS suite.
- Narayan – Restriction Syntax Overview
- Command syntax –
-   incorporates imperative and database operations
-   allows uniform data queries across components
-   easy to process
-   improves atomicity of operations
- Semantics – Examples given
-   going across attributes are ANDed and multple lines are Ored
- An issue of uniqueness was brought up and will be taken into
- consideration by Narayan.

## Meeting notes

- Rusty – Restriction Syntax on Chiba City
- David would like to see a paper of the requirements that the Chiba
- effort required.
- Narayan – Hack of quick interfaces for Queue Manager
- Restriction Interface has 4 commands (add, del, run, get)
- Doesn’t show Scheduler Interface
- Craig – 1280 dual xeon cluster “Titanium” is available this evening
- To test the scalability of SSS suite. One node will be used as
- Head node to install our suite and run on entire cluster.
- Could build everything but Bambo and ssslib due to Xerses
- Will begin to be available at 6pm
- Eric – A competing package. From his Russian “secret city” trip Oct. 03
- Package for - Distributed calculations, metacomputing, Grid.
- System is based on XML, web-based user interface,
- Configure, manage, and submit jobs. Challenges auto load balance.

## Meeting notes

- Late night session on 1280 node testbed
- PM ran at 1280 worked at 4000, hung at 6000
- Warehouse had a problem at 1280 and took out head node
- RM components ran on head node OK until Warehouse crashed it
- Rusty – Process Manager Spec for first vote
- Presentation and discussion...
- Who is responsible for limited enforcement PM or QM? I.e.
- Must use certain amount of memory, must not execute OS command
- (in general - things that happen after fork)
- Rusty says the question is good and he needs to think about
- How this may affect the interface.
- Other items to think about
-  - use of wildcard as “to be returned” operator – OK
-  - Inclusion but don’t show me.
-  - Dynamic jobs and PM.
-  - improve readability
- Delay vote until we have a written proposal.

## Meeting notes

- How to write spec to describe how XML should be extended to future needs.
- Narayan – Node State Manager spec (no written doc so no vote)
- Presentation and lots of discussion...
- Scott – Allocation Manager spec (has written doc in notebook)
- Goes through examples in the document. Discussion.
- Switches to discussion of comparison between both XML syntax
- And Andrew Lusk thinks that a translator could be created for queries
- (but not for output) Rusty thinks it is a bad idea and feels
- It is not  problem to have two syntax.
- David says the translation is good because it could buy time to switch syntax
- Andrew and Paul and Craig offer to help build a prototype translator
- To see how / if it is possible.
- Investigate standardization of tokens across the two syntax

## Meeting notes

- How