# Presentation: 403840

## Allocation Management Solutionsfor High Performance Computing

- Scott M. Jackson
- Workshop on Scheduling and Resource Management for Parallel and Distributed Systems (SRMPDS)
- Parallel and Distributed Processing Techniques and Applications (PDPTA'05)
- June 27, 2005
- Las Vegas, USA

## Allocation Manager

**Allocation Manager**

- $

- Charges for resource usage

**Notes:**
When a large number of projects and users share access to High Performance Computing (HPC) resources, it becomes necessary to manage the allocation of these resources in a controlled and coordinated fashion. The site management needs a means to fairly distribute the underlying computing resources (processors, memory, disk, etc.) to the various users or projects that should have access to them.

These and other problems can be resolved by using a resource allocation management tool, referred to as an allocation manager or an allocation bank. An allocation manager does much more than an accounting system that simply tracks resource usage. It places controls on the use of shared resources. The allocation manager associates a cost (measured in resource credits) with computing resources. It works like a bank, allocating resource credits via deposits, and charging user accounts for the resources utilized by jobs after they complete. Site managers can decide how resource credits are to be allocated among projects, users and machines. Allocation management allows site policy makers to control how resources are used rather than simply tracking the resource usage. 

## Typical problems faced by HPC managers

| Problem | Solution |
| --- | --- |
| Resources are hogged by the projects or project members that submit jobs most aggressively | Fair share, Fine-grained allocations |
| It’s difficult to know how much additional work to accept on the machine or how big of an upgrade is needed | Expiring allocations (The Brick Approach) |
| Projects start slow then simultaneously demand their entire allocations be fulfilled at end of project cycle | Expiring allocations (use-it-or-lose-it) |
| Users can exceed their allocations until accounting is synchronized with resource management system | Dynamic Charging, Reservations |
| Would like to delegate the management and distribution of credits to established project management hierarchy | Role-based AC,          Nested accounts |
| Want to share resources with other organizations but face issues of security, autonomy and accountability | Strong security, Distrd accounting, Charge quotes |
| Each site wants to track and perform historical queries on site-specific accounting statistics | Custom accounting, Flexible charging, Journaling |
| You want the accounting system to be transparent to the user | Default projects, Credit accounts, Auto-generation |

**Notes:**
HPC managers face unique challenges relating to the allocation of their resources. Typical allocation problems include:
Resource hogging -- aggressive users use an unfair share of the resources
Capacity planning -- deciding how many new projects to accept
Year-end resource exhaustion -- underspent projects want their full allocations fulfilled at end of year
End of term spending spree – (If you do use an offline allocation system) delayed settlement permits abuse of allocations

## 	Gold is an open source allocation system that controls project usage on High Performance Computers.

**Allocation Manager**

- Ensures resources are used according to mission plan
- Tracks resource utilization
- Allows for insightful capacity planning
- Facilitates resource sharing between organizations
- Used in production at PNNL and other sites

**Notes:**
The Gold Allocation Manager [1] will be presented as a test case that implements many of the proposed solutions. Gold is an open source accounting system and allocation manager that tracks and allocates resource usage on high performance computers. It acts much like a bank in which accounts are established in order to allocate computing resources to projects and users over designated timeframes. Gold has been developed at Pacific Northwest National Laboratory (PNNL) as part of the U.S. Department of Energy Scalable Systems Software Project [2]. It manages allocations on PNNL’s 11.8TF production Linux Cluster and is currently in testing at several other government labs and universities.

## Resource Hogging

- Perhaps the greatest single resource allocation challenge on a High Performance Computer is that of fairly distributing the computing resources among the parties that want to use them.
- Without some form of allocation control, projects and users would consume computing resources based solely on how aggressively they submit their workload rather than based on any site-managed policies or priorities.

**Notes:**
	Perhaps the greatest single allocation challenge on a High Performance Computing resource is that of fairly distributing the computing resources among the parties that want to use them. Without allocation management or some form of fair-share scheduling, projects and users would consume computing resources based solely on how aggressively they submitted their workload rather than based on any site-managed policies or priorities.

## Fair Share Scheduling

- Job priorities are increased or decreased based on short term historical usage.
- Systems that employ fair share scheduling include LSF and the Maui Scheduler.
- Limitations: Does not factor in distant prior usage, cannot enforce absolute allocation limits.

**Notes:**
One solution to this problem is to use fair share scheduling [3]. Fair share scheduling allows a site to steer resource usage towards a per-user or per-group target by adjusting job priorities in the queue. Job priorities are increased or decreased based on short term historical usage. Systems that employ fair share scheduling include LSF [4] and the Maui Scheduler [5].

## Allocations

- An *allocation* is an allotment of computational resource credits for use by specific projects or users during a specified time period.
- When jobs complete, projects are charged and resource usage recorded. Jobs are prevented from starting unless they have an active allocation with a sufficient balance.
- Limitations: Does manage short-term usage, does not impact scheduling priority.

| Project | Users | Machine | Time Period | Balance |
| --- | --- | --- | --- | --- |
| Biology | Amy, Bob | Colony | FY2005 | 100,000 |
| chemistry | Bob, Dave | Colony | FY2005 | 150,000 |

**Notes:**
Another solution is to use allocations via an allocation manager. An allocation is an allotment of computing resources that is made available for use by an account during a specific time period. Resource credits are allocated to projects and users for use on specified machines. Jobs are prevented from starting unless they have an active allocation with a sufficient balance. Whereas fair share can help direct usage over short term windows, allocations can be used to control usage over long term periods. Allocations can be used in conjunction with fair share scheduling for the best of both worlds.

Gold employs allocations within a unique account architecture that allows a site to dedicate or share HPC resources in a highly flexible manner. Resource credits can be tied to projects, users and machines and are valid for designated time periods. The most common configuration is to allocate resource credits to individual projects and their members. With the Gold Allocation Manager, it is possible to dedicate portions of a project’s resource credits to one or more individual project members, with the remainder being shared among the other members. Resource credits may even be shared by multiple projects. Gold also permits the nesting of accounts. A hierarchical arrangement of accounts allows deposits to trickle down the hierarchy and charges to trickle up the hierarchy.

## Capacity and Workload Planning:The Brick Approach

- Controlling project usage is key to capacity and workload planning.
- The use of regularly expiring allocations allows you to establish a project cycle and use the brick approach to allocation management.

- Now

- - 1 Qtr

- + 3 Qtr

- + 2 Qtr

- + 1 Qtr

- - 3 Qtr

- - 2 Qtr

- + 4 Qtr

- 100 % Capacity

**Notes:**
It becomes a veritable balancing act to keep a High Performance Computing resource fully utilized while ensuring adequate availability to the work proposals you have accepted on your system. It can be difficult to judge whether your system will support yet another project.

One remedy to this problem is to establish an allocation project cycle with the aid of an allocation manager. First, an estimate of the total computational resources within a regular (e.g. yearly) project cycle is made. Each accepted work proposal is granted an allocation that expires at the end of the cycle. By imposing allocation limits, a site can determine when the project load has met capacity limits. If projects consistently underspend their allocations, a site can analyze historical allocation and usage statistics to learn how much it can “overbook” the system to properly balance utilization and availability. 

The Gold Allocation Manager allows time-bounded allocations to be created through deposits. It tracks resource usage and stores historical transactions. Historical allocation state information is preserved via journaling. Capacity planning can be assisted by reports such as project usage summaries and account statements.

## Year-end Resource Exhaustion

- It is common to see projects get off to a slow start and then have a surge of activity at the end when results are due.
- Without careful management of expectations, this can be a cause of considerable anxiety.

- 100% Machine Capacity

- Project A’s Allotment

- Project A’s Demand

- Project Period

**Notes:**
When running an HPC center where projects are granted resources in periodic (e.g. yearly) cycles, it is common to see projects get off to a slow start. However, near the end of the cycle, when results are due, there is a surge of activity and everyone wants their unused allocations fulfilled. Clearly, if many projects are allowed to do this, nobody’s allocations are going to be fulfilled at the end. Without careful management of expectations, this can become a cause of considerable consternation, as it can become impossible to fulfill all the outstanding requests.

## Expiring Allocations

**Notes:**
One solution to this problem is to implement a use-it-or-lose-it policy in which resource credits are allocated in smaller chunks that expire (become inactive) at regular intervals within the project cycle. In some systems, resource credits may also be made to start (become active) in the future. Resource credits that are not used within their validity period are lost to the project. In this way, a proscribed usage pattern can be established by the resource provider.

Gold supports the creation of allocations for use within arbitrary time periods. For example, four quarterly disbursements may be apportioned within a yearly project cycle. At the end of the each quarter, any unused resource credits designated for use during that period might be made to expire.
Figures 1-4 illustrate some possible allocation strategies. In Figure 1, all disbursements are made into a single monolithic allocation. In Figure 2, all of the credits are available for use at the beginning of the usage period. Portions of the resource credits are made to expire at regular intervals implementing a use-it-or-lose-it policy. If the main concern was to prevent overuse up front, a site could instead periodically increase the allocations as shown in Figure 3. Staggered allocations, depicted in Figure 4, can be used to ensure projects use their funds at an even proscribed rate.

## End of Life Spending Spree

- Offline banks cannot strictly enforce allocations
- Without reservations, more jobs can get started than allocation balance can support.

- Job

- Debit
- Accounts

- Job

- Debit
- Accounts

**Notes:**
Many sites have recognized the need for some type of Allocation Management and have developed home-grown systems that periodically (weekly or daily) parse the resource management logs in order to make debits to a master accounts file or database. Such offline banks suffer exposure to abuse since the projects nearing completion (with a negligible balance) can get a large number of jobs started just before the account is disabled resulting in overuse of their allocation and a negative balance.

## Reservations and Dynamic Charging

**Accounting Manager (Gold)**

**Deposit**

**0   Deposits are made in Account**

**1   A Job is Submitted**

**2   A Quote is Requested**

**Quote**

**3   A Reservation is Made**

**Scheduler**
**(Maui)**

**4   The Job is allowed to Start**

**Reservation**

**5   The Job Completes**

**Resource Manager**
**(PBS, LL, LSF)**

**6   The Reservation is Removed**

**Reservation**

**    ****and a Charge is Issued**

**Notes:**
[The use of reservations and dynamic charging (referred to as online banking) allows you to strictly enforce usage limits – which is impossible with the offline banking approach where accounts are rectified long after the job completes.]

This problem can be overcome through the use of reservations and dynamic charging. Before a job runs, the bank will attempt to place a reservation or hold against the account for the estimated number of resource credits based on the estimated wallclock time (wall clock limit) for the job. Subsequent jobs will also place reservations while the available balance (allocations-reservations) allows. As soon as a job completes, the reservation is removed and the actual charge is made to the account. This procedure ensures strict enforcement of the allocations. 

The Gold Allocation Manager functions as an online bank, facilitating the placement of a reservation before a job starts and the charging of a job immediately following its termination, thereby preventing projects from using more resources than were allocated to them. Reservations are given an end time causing them to automatically expire after the wallclock limit so that they do not impinge against the account balance if they are not properly cleaned up at the end of the job. 

## 	Gold is an open source allocation system that controls project usage on High Performance Computers.

**Allocation Manager**

- Ensures resources are used according to mission plan
- Tracks resource utilization
- Allows for insightful capacity planning
- Facilitates resource sharing between organizations
- Used in production at PNNL and other sites

**Notes:**
Conclusion:

High Performance Computing centers frequently encounter allocation problems such as resource hogging, capacity planning, year-end resource exhaustion, and end-of-term spending sprees. An allocation manager can be effectively employed as part of a solution to these challenges by using mechanisms such as regularly expiring allocations, reservations and dynamic charging.

The Gold Allocation Manager demonstrates a real world implementation of these mechanisms. When used in conjunction with appropriate scheduling and queue policies, sites can transition from merely tracking the computing resources used to actually controlling that usage in a manner that is fair and consistent with local site priorities.

## Background

- SciDAC
- Scientific Discovery through Advanced Computing – A DOE initiative to improve the impact of scientific computing.
- Scalable Systems Software Project (SSS)
- Research, develop and support an integrated suite of systems software and tools for the effective management and utilization of DOE’s terascale computational resources. It is a 5 year project involving over a dozen sites.
- QBank
- Gold is based on a successful program, QBank, that has been used for years on dozens of government and university computing systems.

**Notes:**
The Scalable Systems Software Center is a 5-year multi-institutional SciDAC project funded by DOE ASCR to produce a fully integrated suite of scalable open-source software for the management and effective utilization of DOE’s terascale computers.

PNNL has partnered with a dozen other sites to improve the manageability and utilization of DOE’s largest systems. As lead of the Resource Management and Accounting Working Group, I work very closely with Ames Lab and Oak Ridge National Laboratory.

QBank is in production use on dozens of sites including DOE labs like Argonne National Laboratory, DOD labs like Maui High Performance Computing Center and universities such as the University of Utah.

## Allocation Architecture

| Time Period | Allocation |
| --- | --- |
| F1Q04 | 1,000,000 |
| F2Q04 | 1,000,000 |
| F3Q04 | 1,000,000 |
| F4Q04 | 1,000,000 |

- Universe
- of
- Projects

- Account: 1234
- Name:	Chemistry

- Universe
- of
- Users

- Universe
- of
- Machines

- Account: 1234
- Name:	Chemistry

| Time Period | Allocation | Credit Limit |
| --- | --- | --- |
| F1Q04 | 1,000,000 | 10,000 |
| F2Q04 | 1,000,000 | 10,000 |
| F3Q04 | 1,000,000 | 10,000 |
| F4Q04 | 1,000,000 | 10,000 |

## Other Gold Features

**Flexible Charging**

**Credit and Debit Models**

**Guaranteed Quotes**

**Powerful Querying**

- CPU

- Memory

- Disk

- QOS

**Notes:**
Other sites use features in Gold such as overdraft, or a bottom-feeder account that is charged less in return for reduced priority in the queue. 
The University of Utah is using Gold to allow jobs that run out of cycles to have their priority reduced and charge from a freecycle account.

## Other Gold Features

**Strong Security**

**Historical Journaling**

**Transparency Features**

- Customizable
- Accounting

**Dynamically Customizable**

**Notes:**
Since accounting needs vary greatly from organization to organization, Gold has been designed to be extremely flexible, featuring customizable accounting and supporting a variety of accounting models.

Attention has been given to scalability, security and fault tolerance.

## Web-based GUI

- Facilitates Resource Sharing Between Organizations

**Notes:**
Distributed Allocation and Accounting enables resource sharing between sites.

## Status

- Status:
- In production use at PNNL 11.8 TF system
- 	and at least one other site – being evaluated at others
- Currently in second beta release
- General release targeting 3rd quarter of 2005
- Download:
- _[http://www.emsl.pnl.gov/docs/mscf/gold](http://www.emsl.pnl.gov/docs/mscf/gold)_
- For more information:
- _[Scott.Jackson@pnl.gov](mailto:Scott.Jackson@pnl.gov)_
- _[http://www.emsl.pnl.gov/docs/mscf/gold](http://www.emsl.pnl.gov/docs/mscf/gold)_
- _[http://](http://sss.scl.ameslab.gov/home.shtml)__[sss.scl.ameslab.gov/home.shtml](http://sss.scl.ameslab.gov/home.shtml)_
- http://www.scidac.org/ScalableSystems

**Notes:**
Been running at PNNL for a year and a half to two years now. It has been extremely stable and I don’t hear much about it because it just works.
The GUI is still receiving final touches – a beta will be out for the GUI in the production release.

## Additional Slides

## Nested Accounts

- University

- Biology

- Actinides

- Chem 201

- Physics

- Chem 101

- Chemistry

- Workshop

- 25

- 25

- 25

- 40

- 40

- 50

- 40

**Notes:**
A hierarchical relationship may be created between accounts. This allows for the delegation of management responsibilities, the establishment of automatic rules for the distribution of downstream resource credits, and the option of making higher level credits available to lower level accounts.

## Gold Welcome Screen

## List Accounts

## Modify Projects

## Account Statement