**White paper February, 2005**

**What are the System Software Challenges for SciDAC?**

Prepared by SciDAC Scalable Systems Software Team

[www.scidac.org/ScalableSystems](http://www.scidac.org/ScalableSystems)

## System Software Challenges

The nation's premiere scientific computing centers are facing a crisis
where they are having to rewrite all their home-grown systems software
to scale to the multi-teraflops systems that are being installed in
their centers. Over the past few years the DOE, along with others, have
continued to expand the size of high-end systems. Today there are many
systems with more than 1000 processors and several systems with more
than 4000 processors. We fully expect this trend to continue in the
future with increasing node and processor counts. Indeed the first full
scale BlueGene/L system will be delivered in the next year providing a
single system with 64 thousand nodes and 128 thousand processors! While
this growth has been largely successful on the hardware side, the
software to needed to manage the growing complexity that comes with
larger node counts has continued to lag far behind. In large part the
systems management software continues to be based on software developed
a decade ago to manage a single system or at most a handful of systems.
While many people have worked at patching the software to get it to work
on larger systems, there have not been many efforts to design a systems
software management infrastructure that is designed for scalability from
day one.

The systems software problems for tera-op class computers with thousands
of processors are significantly more difficult than for small-scale
systems with respect to fault-tolerance, reliability, manageability, and
ease of use for systems administrators and users. Layered on top of
these are issues of security, heterogeneity and scalability found in
today's large computer centers. The computer industry is not going to
solve these problems because business trends push them towards smaller
systems aimed at web serving, database farms, and departmental sized
systems. In the longer term, the operating system issues faced by next
generation petaop class computers will require research into innovative
approaches to systems software that must be started today in order to be
ready when these systems arrive.

Goals of the Scalable Systems Software ISIC

The goal of the SciDAC Scalable Systems Software ISIC is to
fundamentally change the way future high-end systems software is
developed to make it more cost effective and robust. The research
involves three efforts. First, collectively getting the DOE centers, NSF
centers, and vendors (IBM, Cray, SGI, Intel) to define standardized and
flexible interfaces between system components. Second, designing a
modular software architecture that is portable across all major
supercomputer platforms. Third, producing an open source reference
implementation of a fully integrated suite of systems software that can
be used across all the terascale computer centers for the cost effective
management and utilization of their computational resources. The scope
of our suite encompasses the aspects of system management illustrated in
Figure 1.

![](media/image1.png){width="3.042361111111111in"
height="2.3340277777777776in"}

Figure 1. The mission of the Scalable Systems Software center is the
development of an integrated suite of systems software and tools for the
effective management and utilization of terascale computational
resources particularly those at the DOE facilities.

## Highlights

**Designed Modular Architecture:** A critical component of our design is
its modularity. The ability to plug and play components is important
because of the diversity in both high-end systems and in the individual
site management policies. For example, an individual site may have some
very specialized requirements for job scheduling that may require them
to write their own scheduling component. Figure 2 shows the systems
software architecture and how all the components interface with each
other. The Event Manager, Service Directory, and Communication library
shown in the middle connect to all the other components and allow
individual components to announce their participation in the system,
find one another, and communicate both synchronously (send/receive) and
asynchronously (register/notify). They play a key role in the
portability of the architecture to different terascale computing
facilities and hardware.

![](media/image2.jpeg){width="3.3097222222222222in"
height="2.4819444444444443in"}

> Figure 2. System Software Architecture and component interfaces.

**Defined XML based interfaces that are independent of language and wire
protocol:** The interfaces between all the components have been fully
documented and made publicly available allowing others to write
replacement components (or wrap their existing components) as needed.
Components communicate through XML messages and can use one or more of
the extensible set of wire protocols supplied by the Communication
library. The API imposes no restrictions on the language a particular
component is written in. Our reference implementation has a mixture of
C, C++, Perl, Java, and Python components. This allows a great deal of
flexibility to the component author and allows the same interface to
work on a wide range of hardware architectures and facility polices.

**Reference Implementation Released:** Version 1.0 of our fully
integrated systems software suite was released at SC2005 and will be
followed by quarterly updates for the next year. The suite is available
in both source form and as a precompiled, integrated package in the form
of an OSCAR (Open Source Cluster Application Resources) distribution.
OSCAR distributions (of which there are several) have been adopted by
many cluster vendors and have been downloaded by thousands all around
the world. By leveraging the popularity of OSCAR, we raise our software
suites profile and availability.

**Adoption:** The success of a CS ISIC can be measured in part by the
adoption of the software into the research community and/or industry.
The systems software suite has several key features that will make it
attractive for others to adopt. First, the suite is modular, which
allows others to easily replace a component that doesn't meet their
needs or use only the parts of the suite they need. Second, standard
interfaces allows components to be shared across facilities, reducing
development and support costs. Finally, the reference systems software
is all provided under an open source license to allow others, including
commercial interests, to use, modify, and ship it as part of their own
offering.

**Production Users:** Full system software suites are now in production
use on clusters at Ames laboratory and Argonne National Laboratory. The
Chiba City 200-node cluster at Argonne has been operated for over a year
by a collection of components based on the architecture described here.
Ames has been running a full suite for several months. Pacific Northwest
National Laboratory and the National Center for Supercomputer
Applications have adopted one or more components from the suite to use
on their production systems. The information gathered from the
production use is currently being used as feedback into further
developments. We are also in discussion with DOD-HPCMP sites about use
of some of our system software components.

**Adoption of API:** The suite's scheduler component is the widely used
Maui Scheduler. the public Maui release (as well as the commercial Moab
scheduler) has been updated to use the public XML interfaces and has
added new capabilities for fairness, higher system utilization, and
improved response time. All new Maui and Moab installations worldwide
(more than 3000/month) now use the system software interfaces developed
in this ISIC. This includes: 75 or the top 100 supercomputers in the TOP
500 list and commercial industries such as Amazon and Ford.

In February 2005 we held discussions with Cray about their software
roadmap for the Leadership-class systems. Cray also has a plan to use
XML messages to connect their systems components. We have exchanged
information on our XML format, process manager API, and API test
harness.

**View to the Future---CS ISIC Gaps:**

Although much has been accomplished in the CS ISICs, several important
gaps have been identified requiring research and development over the
next few years in order to provide capable Leadership-class resources to
the SciDAC scientists

-   Given the heterogeneous nature of the multiple leadership-class
    machines, science teams need to have a robust environment that
    presents the similar programming interfaces and tools across the
    different machines.

-   There is a lack of understanding of the fault tolerance requirements
    in the OS and systems software, particularly as systems scale up to
    petascale in the 2010 time frame.

-   Support for application users submitting interactive jobs is
    presently not addressed, but will become more important as
    scientists begin to use computational steering for scientific
    discovery.

-   Security has been addressed in the communication infrastructure, but
    not at higher levels where one wishes to restrict access to certain
    components while allowing access to others.

-   There are increasingly demanding needs on the file systems including
    security, scalability, and fault tolerance. It is critical to the
    success of the NLCF that these issues be addressed.

The National Leadership Computing Facility (NLCF) will incorporate
several different systems including a Cray X1 vector systems, a RedStorm
system, and an IBM BlueGene system. While this diversity will be a
benefit for many applications, it will add significant complexity to the
system management. This is an area where the systems software effort can
offer significant benefits. If the suite is optimized to run on each of
the architectures it would provide the system administrators as well as
the scientists with a uniform interface to each system. Many of the
underlying components would have different implementations, but the
modularity of the system should allow much of those differences to be
hidden

One of the challenges of such a heterogeneous environment is that while
the application may be developed in a standard language such as FORTRAN
or C that is available on most machines, the underlying algorithms
implemented in the software typically do not translate as easily to "yet
another" architecture. Thus, at best you can hope for an application
that does not run optimally and at worst, the application will simply
not run at all. Research into unified computing environments that employ
techniques to simplify leveraging of the underlying architecture will go
a long way in improving this situation. Here the hope is to provide a
common look and feel for application development across the varying
hardware platforms used to construct NLCF class computing. This will
include everything from common libraries, with their underlying
implementation hidden yet taking advantage of specific architectural
traits, to common system administration suites for the operators.\
\
Today batch scheduling is the norm, but there is an increasing desire by
science teams to be able to do things more interactively, one example is
to steer a computation while visualizing the live results of a run.
Another problem that is emerging is how to dynamically expand, or
scale-up, a computation in an efficient manner during runtime.

Research in survivability and resiliency of hardware/software systems
will go far to increase the usability and scalability of our NLCF class
machines. In those situations where a failure can not be circumvented,
an extended recovery capability with a transparent failure recovery is
needed in order to provide more future options to better enable the
requisite scale-out for NLCF class computing. Some of these options
include reliability-aware intelligent checkpointing, a sophisticated
rule-based recovery, diskless checkpointing on non-volitile memory, and
a common interface layer available to the various message passing and
shared memory schemes.

High-end, parallel file systems are evolving as the de-facto standard
for supercomputing centers due to their attractive price-performance
ratio, high/sustained throughput required for data intensive
applications and tight integration with computing nodes. However, they
lack strong availability semantics, often times creating \"islands\" of
file system hierarchies due to head node failure, etc. In addition,
seamlessly scaling to several hundreds of terabytes is also an issue. To
this end, exploring low-cost alternatives that can complement these
systems to transparently and reliably make available several orders of
terabytes through storage aggregation can be worthwhile. Local cluster
storage or workstation storage can be transparently accrued to offer a
highly-available, replicated store for these high-end parallel file
systems. Further, to address seamless application functioning, client
accesses to partitioned (unavailable) portions of the parallel file
system can be transparently channeled to such aggregate stores and
eventually to archival stores, thereby creating a seamless hierarchical
pipeline of differentiated storage services. These research areas can
help improve the capacity, availability and serviceability of
supercomputer storage systems.