# Presentation: 131820

## Review report, Vote on APIsQuarterly report, and SW release

**Review report, Vote on APIs****Quarterly report, and SW release**

- Al Geist
- June 5-6, 2003
- Chicago, IL

## Participating Organizations

**Coordinator: Al Geist**

**Participating Organizations**

**ORNL**
**ANL**
**LBNL**
**PNNL**

**PSC**
**SDSC**
**IBM**

**SNL**
**LANL**
**Ames**
**NCSA**

**Cray**
**Intel**
**Unlimited Scale**

**Participating Organizations**

**External reviewers want to see more vendors involved **

**Have begun working with Don Mason and John Lawson to**** ****set up a presentation to a vendor forum.**
**Will need your participation when logistics are known**

## Scalable Systems Software

- IBM
- Cray
- Intel
- Unlimited Scale

**Scalable Systems Software**

**Participating Organizations**

- ORNL
- ANL
- LBNL
- PNNL

- NCSA
- PSC
- SDSC

- SNL
- LANL
- Ames

- Collectively (with industry) define standard interfaces between systems components for interoperability
- Create scalable, standardized management tools for efficiently running our large computing centers

**Problem**

**Goals**

**Impact**

- Computer centers use incompatible, ad hoc set of systems tools
- Present tools are not designed to scale to multi-Teraflop systems

- Reduced facility  mgmt costs.
- More effective use of machines by scientific applications.

**www.scidac.org/ScalableSystems**

**To learn more visit**

## Scalable Systems Software Center

- February 24-25
- Chicago ILL

**Review of Last Meeting**

- Details in
- Main project notebook

## Progress Reports at Feb. mtg

**Al Geist **– preparation for external review,
-      SciDAC PI meeting, posters, and demos
**Working Group Leaders** –
- What areas their working group is addressing
- Progress report on what their group has done
- Present problems being addressed
- Next steps for the group
- Discussion items for the larger group to consider
**Discussion of Prototype Components**
- Prep for external review demo

- Slides can be found in Main Notebook

## Consensus and Voting:

**None at last meeting.**
**Something we need to start doing again.**

## Scalable Systems Software Center

- February-June

**Progress Since Last Meeting**

## SciDAC PI mtg – all 50 projects

**SciDAC PI mtg – all 50 projects**

**March10-11, 2003 **– Napa California
- Attending for Scalable Systems – Al Geist, Brett Bode
**20 minute talk **– presented by Al
- Scalable Systems, CCA, PERC, SDM
**Poster Presentation**

## External SciDAC Review mtg

**March12-13, 2003 **– Napa California
- Attending for Scalable Systems – Al Geist, Brett Bode, Paul Hargrove, Narayan Desai, Mike Showerman. (Rusty)
**Four ISIC Projects were reviewed separately **–
- Scalable Systems, CCA, PERC, SDM
**External review panel** (9 members)
- Bob Lucas, Jim McGraw, Jose Munoz, Lauren Smith, Richard Mount, Ricky Kendall, Rod Oldehoeft, and Tony Mezzacappa John Grosh
**Day 1 – We had 1 3⁄4  hours to present project**
**Day 2 – We got grilled by panel for 11⁄2 hrs**

## External Review mtg Agenda

**Wednesday, March 12**
** **
**  ****7:45****	****	**Welcome, charge to reviewers
**  ****8:15**		Plenary session for Common Component Architecture ISIC
**10: 00**		Break
**10:15**		**Plenary session for Scalable Systems Software ISIC**
**                             ****Al Geist gives 1 hr project overview, vision, goals**
**                             ****Last 45 minutes team gives demos, answer questions**
**12:00****	****	**Reviewer caucus
**12:15** 		Lunch
**  ****1:15**		Plenary session for Scientific Data Management ISIC
**  ****3:00****	****	**Break
**  ****3:15**		Plenary session for Performance Engineering ISIC
-   **5:00**   	    Reviewer caucus
**  ****5:30**		Adjourn

## External Review Demo

**Grid Interfaces**

**Accounting**

**Event **
**Manager**

**Service**
**Directory**

**Meta**
**Scheduler**

**Meta**
**Monitor**

**Meta**
**Manager**

**Scheduler**

**Node State**
**Manager**

**Allocation**
**Management**

**Process **
**Manager **

**Usage**
**Reports**

**Meta Services**

**System &**
**Job Monitor**

**Job Queue**
**Manager**

**Node**
**Configuration **
**& Build**
**Manager**

- Standard XML
- interfaces

**Working Components and Interfaces (bold)**

**authentication **
**communication**

**Components written in any mixture of C, C++, Java, Perl, and Python can be integrated into the Scalable Systems Software Suite**** **

**Checkpoint /**
**Restart**

**External Review Demo**

**Validation & Testing**

**Hardware**
**Infrastructure**
**Manager**

## External Review mtg Agenda Day 2

**Thursday, March 13**
**  ****8:00****	****	****Meetings between reviewers and ISIC members**
-                               A.     Common Component Architecture
-                               **B.     Scalable Systems Software (Jim McGraw)**
**  ****9:45****	****	**Break
** ****10:00****	****	**Meetings between reviewers and ISIC members
-                                                C.     Scientific Data Management
-                                                D.     Performance Engineering
**11:45****	****	****Reviewer Caucus/End of ISIC Reviews**
**12:15****	****	**Lunch (**McGraw gives initial assessment**)
**Team brain storms on the response to the initial comments**
**These sent to McGraw that same day.**
**  **

## External Review Initial Comments

- Response on top two issues:
-  1. Lack of large-scale testbed for scalable systems    Mike Showermann of NCSA says they will have a 900 processor system by late summer that Scalable Systems software could be tested on. He also said there are plans to get an additional 1300 node system. CPlant has also been thrown out as a possible large scale (~1200 processor) test platform. 2. Get more vendors involved and more "buy-in"    I will redouble my efforts to get SGI to get back involved in Scalable Systems. HP has been a tough nut to crack, both PSC and PNNL have tried to get them to engage. I'll see if PSC and PNNL are willing to try again.    By late summer we will have a beta release of the suite that I can use to demonstrate to vendors our progress and advantages of going the scalable systems path.

## Official External Review Report

**Arrived in May 2003**

- Organizationally the project has developed effective working units
- The project appears to be on schedule for technical issues
- It has made several noteworthy accomplishments
- Recommendations:
- The two greatest obstacles to the success of this project are the availability of an adequate testbed for proving scalability of the interface design and the willingness of vendors to adopt the design for future systems
- Secondary considerations
- Investigate relationship with CCA
- Investigate File system plan
- Investigate security plan
- Importance of fault tolerance at smaller cluster sizes
- Develop test workloads

## Five Project Notebooks filling up

- A main notebook for general information
- And individual notebooks for each working group
- Over 270 total pages – few added since last meeting
- Add Telecon meeting notes even if short
- Have had several web server problems this quarter

- Get to all notebooks through main web site
**www.scidac.org/ScalableSystems**
- Click on side bar or at “project notebooks” at bottom of page** **

## Bi-Weekly Working Group Telecons

**Have been sparse since March review**** **

**Resource management, scheduling, and accounting**** **
**     **Tuesday 3:00 pm (Eastern)**  1-800-664-0771   keyword “SSS mtg”**
**Validation and Testing**** (****hasn’t met since last year)**
**     **Wednesday 1:00 pm (Eastern)**  1-877-540-9892   mtg code 999157**
**Proccess management, system monitoring, and checkpointing **
**     **Thursday 1:00 pm (Eastern) ** 1-877-252-5250  mtg code 160910**
**Node build, configuration, and information service**
**     **Thursday  3:00 pm (Eastern)  **1-888-469-1934  mtg code ****(changes)**

## Scalable Systems Software Center

- February 24-25, 2003

**This Meeting**

## Major Topics this Meeting

**MICS request for Highlights **– Fred sent out a call for 2 page highlights due to MICS by June 12.
- Has anyone responded?  I sent in our 2 pager
**Response to Reviewers Report **– need feedback from the team on our official response to the points in the report
**Quarterly Report Due **–  would like to get one to Fred by end of June. Will need text from WG leaders.
**Formal API presentations and voting**  - it is that time in the project when we should be settling on some APIs.
**SC2003 Tutorial**  - proposal submitted at Fred’s request. Have a software suit released before SC2003

## Agenda – June 5

-   8:30     Al Geist – Project Status. Qtr report coming up
-               and External review report
-   9:00     Matt Sottile – Using Scalable Systems API
-   **Working Group Reports**
-   9:30	Scott Jackson – Resource Management
- 10:30     Break
- 11:00 	Erik Debenedictis – Validation and Testing
- 12:00     Lunch  (on own - walk to cafeteria)
-   1:00     Paul Hargrove – Process Management + Rusty slides
-   1:30     Craig Stefan – Warehouse Monitoring framework
-   2:00 	Narayan Desai – Node Build, Configure
-               Stephen Scott – OSCAR release with SSS inside
-   3.00      Break
-   3:30      **Presentation of formal APIs for discussion**
-   5:00      Rusty, Scott, Narayan, Paul?
-   5:30      Adjourn
-   Working groups may wish to prepare material for voting Friday

## Agenda – June 6

-   8:30     **Discussion, proposals, straw votes**
**	**Discussion of review report
-               API proposals for envelope
**	****	**
- 10:30     Break
- 11:00     Al Geist – Summary
-               Qtr Report.
-               next meeting date: .
- 12:00     meeting ends

## Meeting notes

- Matt Sottile – bproc (bstat_sss) software integrated with cluster status component
- Good (was able to do it in a day) bad (shouldn’t take 8 hours) ugly (python)
- Example with distribution didn’t help much.
- XML isn’t well documented
- But it is a prototype distribution so some of these issues are expected
- Major gripes had to write code for Socket code and XML parsing and creation
- These should be APIs – He then talks about Linux TCP being a hack
- XML parsing – the schema and associated parser are intimately related
- Noted that code had some constructs that could be made more robust
- CCA thoughts on relation to our project
- His expertise is language interoperability and runtime frameworks
- Law of least surprises. Consistency is good
- Insulate developers from the support structure
- Components the wheel everyone continues to reinvent
- But is SSS there aren’t components – just XML and wire protocol
- CCA provides: SIDL, standard interfaces to runtimes – CCAFFEINE, CCAT, Dune, ...
- Suggests: Could try to leverage CCA messaging layer, Define interfaces in SIDL, Build services that conform to SIDL. CCA provides no security
- Concentrate on interfaces and problem of mapping concrete services into the interface space of SSS
- Conclusion: Clean up APIs to minimize possibilities for version skew. Too late to adopt CCA model
- Overall things worked – a good accomplishment
- Showed demo

## Meeting notes

- Scott Jackson – RM wg report
- Progress – SSS front end created for Qbank,
- Soon Release v1.0 Open PBS, Maui, and Qbank all with SSS XML front end.
- Created Job Object specification v2.0
- Created SSSRMAP v 2.0 – in notebook
- Scheduler progress: 40% of clients now using SSSRMAP, supports dynamic reservations to support growing and shrinking MPI jobs
- Security- support for a user specified keyfile
- Fault tolerance – implemented a fallback server
- Ease of use- initial web-GUI developed
- Oueue Manager Progress – updated service directory and event manager interfaces
- Accounting and allocation manager progress – GOLD
- All functionality of Qbank plus support for deposits, support for hierarchical accounts, support for refunds, guaranteed quotes, negotiation of options
- Added role-based access control, authentication, and encryption
- Got PNL OK to open source as BSD, sent email to Fred for DOE OK
- Will talk about SSSRMAP v2 details this afternoon, in particular interfaces to other working group components.

## Meeting notes

- Will McLendon – Validation and testing WG update
- Strategies for distributed runtime system testing – users expect high quality
- ESP benchmark – out of NERSC used in procurement to predict the effectiveness of a system before it is purchased. Could be used to test the SSS suite
- Consider putting ESP on the SSS testbed(s)
- APItest – most of the work this quarter is going on here.
- Recoded in Python  for portability (C++ version had portability problems)
- Integrated into SSSlib as part of the distribution
- Tests well under develoment for ssslib components
- Status slide shows working, prototype, and planned features
- Black box testing – does component support the API
- White box testing – coverage tests, internal states of component, unreachable states
- Encoding XML inside XML is a problem
- Ran real demos of APItest running on Chiba City
- MySQL database support – used to store raw test results
- Work still to do – see status slide

## Meeting notes

- Paul Hargrove – used my laptop for presentation – see slides
- Checkpoint/restart progress is stalled because person has been pulled off our project by Bill McCurdy to work on NERSC projects.
- Craig Steffan – Warehouse Monitoring Software Infrastructure
- Describes the old way cluster monitor worked and scalability issues with it
- Presents new design – each node is a peer each can be root of subtree
- They can be grouped into “information storehouses” w/ multiple sources and sinks
- Showed how it can be used to monitor multiple clusters in a compute center
- Information storehouse infrastructure is done.
- Sources and Sinks – next step will be to write simple ones, then more complex
- Lots of questions about the design. Good answers from Craig
- Only update changing information
- In next 6 months - Self balancing systems by tuning update intervals and
- Message passing to request information through the tree

## Meeting notes

- Narayan Desai – BCWG report
- All APIs changed to restriction syntax – draft spec
- Service directory – new schema and new implementation
- Event manager – same
- SSSlib – more wire protocol modules – SSL, SSSRMAP
- OSX port in progress
- Build and configuration now has diagnostic services
- Hardware infrastructure issues discussed – what does system look like right now?
- Open issues
- 	specification formats – what tests does it need to pass
- 	release formats – see OSCAR slides
- 	XML interface formats
- 	multiple implementations
- Thomas Naughton – SSS deployment using OSCAR
- How users download and install SSS suite? Propose leverage OSCAR framework
- OSCAR core – SIS, C3, ODA, Env-Switcher
- OSCAR package facility – RPMs and other package classes
- OSCAR package loader
- Seems to be consensus of group to do this for SC2003

## Meeting notes

- Rusty Proposal – an API for the Process Mangement Component
- He says the material is not quite in the form needed to vote on, but here is the process we should follow to vote in standard APIs
- Voting should be on a document that has
- 	descriptions
- 	examples both simple and complex
- 	Details of XML schema
- See his slides for details of his process manager interface proposal
- Much discussion.
- Scott Jackson – SSSRMAP v2 proposal
- Have taken an object oriented approach to jobs and attributes
- Goes over Basic examples in proposal (found in RM notebook)
- Discuss of the differences between RM Schema and BC Schema
- Part of the difference is the incorporation of security
- Another part is functional vs object oriented
- Discussion of outer (envelope, signature, body) framing and put in SSSlib (vote)

## Meeting notes Day 2

- Al Geist – action items
- 1. Need Working group leaders to send me a couple pages for the Qtr rpt
- Status and Progress from Feb-June
- 2. Any comments on points in the external reviewers report.
- Paragraph or two is fine.

## Meeting notes

- Narayan Desai – Restriction syntax proposal
- Goes over basic command syntax where an attribute can be “*” wildcarded
- Goes over complex command syntax
- Matching semantics – especially for wildcards
- Benefits of this approach – compact, powerful, simple syntax, validatable, data ownership is explicit
- Uses MySQL on the backend
- This syntax has Constructive Normal Form
- Discussion that need to add negation before this is true
- What about regular expression support? –
- More discussion on how to do various things like “join” and “union”
- Discussion of the Communication Infrastructure Spec Draft (hardcopy handed out)
- We should be able to hardwire components together.
- Existence of static file to define where things are – may just have service directory
- Uunix Domain socket protocol for SMP servers
- Vote – accept the spec pending  Yes 15, No, 0  abstaning 0

## Meeting notes

- Paul – Discusses the idea of hiding the socket code in a library
- Matt says he would be happy to contribute such a server.
- Discussion of scalability of the event manager – not a problem because the
- Number of meatballs does not increase with system size.
- Question about the Ordering of events  notification
- Scott – Lively discussion of the two XML variants
- What are the strengths and weakness of both
- Agreement for having common error objects with 3 digit codes and messages
- Message is human readable string. Two special ones 000 success 999 unknown
- Straw vote: 15 no 1 Abs 0
- Add “supported scheme version” to Service directory Vote: 15 no 0 Abs 0
- Next meeting September 9-10 in DC so Fred can attend?