**Best practices for the procurement process of high performance
computers by the Federal government**

+-----------------+---------------+-------------------+----------------+
| R. Scott        | Arthur S.     | William T.C.      | Robin          |
| Studham         | Bland\        | Kramer            | Goldstone      |
|                 | Director of   |                   | Group Leader,  |
| Technical Group | Operations\   | Head of High      | Production     |
| Leader          | ORNL Center   | Performance       | Linux Group\   |
|                 | for           | Computing and     | LLNL\          |
| MSCFO - PNNL    | Computational | General Manager   | 7000 East Ave. |
|                 | Sciences\     | National Energy   | L-073\         |
| P.O. Box 999\   | P.O. Box      | Research          | Livermore, CA  |
| MSIN: K8-91\    | 2008\         | Scientific        | 94550\         |
| Richland WA     | Oak Ridge, TN | Computing Center  | (925)          |
| 99352           | 37831-6203\   | (NERSC) - LBNL    | 422-5832\      |
|                 | +1-           |                   | r              |
| +1-509-376-8430 | 865-576-6727\ | MS 50B-4230       | obing@llnl.gov |
|                 | Bla           |                   |                |
| studham@pnl.gov | ndAS@ornl.gov | Berkeley CA,      |                |
|                 |               | 94720             |                |
|                 |               |                   |                |
|                 |               | \(510\) 486-7577  |                |
|                 |               |                   |                |
|                 |               | kramer@nersc.     |                |
+-----------------+---------------+-------------------+----------------+
| Cheryl Wampler  | David A.      | Bob Ayars         |                |
| Group Leader,   | Dixon         |                   |                |
|                 |               | Deputy Group      |                |
| CCN-7 LANL      | Battelle      | Leader, CCN-7     |                |
|                 | Fellow        | LANL              |                |
| High            |               |                   |                |
| Performance     | Pacific       | High Performance  |                |
|                 | Northwest     | Computing Systems |                |
| Computing       | National      |                   |                |
| Systems         | Laboratory,   | MS T084           |                |
|                 |               |                   |                |
| MS T084         | P.O. Box      | Los Alamos, New   |                |
|                 | 999/MS K9-90, | Mexico 87544      |                |
| Los Alamos, New | Richland, WA  |                   |                |
| Mexico 87544    | 99352,        | \(505\) 667-9047  |                |
|                 |               |                   |                |
| \(505\)         | 509-372-4999  | rsa@lanl.gov      |                |
| 667-0147        |               |                   |                |
|                 | david.        |                   |                |
| clw@lanl.gov    | dixon@pnl.gov |                   |                |
+-----------------+---------------+-------------------+----------------+

This white paper is being submitted in response to question #8: "As
input to HECRFT charge (3) overall, please provide information about how
the Federal government can improve the process of procuring HEC systems
and tools." The institutions and agencies represented by the authors
have been involved in platform procurements of various sizes and
natures, including some of the largest HEC platform procurements
conducted by U.S. agencies, such as the Cray T3E system at NERSC, the
IBM Power 3 and Power 4 systems at NERSC, ORNL, and LLNL, the Cray X1 at
ORNL, the compaq HPQ system at LANL, the IBM Power 5 and Blue Gene
systems at LLNL, and the large HP Linux cluster at PNNL. Different
procurement methods have been used, including: (1) System specification
based procurement, (2) Science driven (Greenbook) procurement with
design criteria, (3) Performance driven procurement based on benchmarks
of actual applications, and (4) Direct partnerships with industry. Often
the actual procurement process is a combination of these methods. In all
cases, it is critical that the eventual end-users and stakeholder of the
systems play a role in defining specifications and requirements for the
system and when feasible, participate in the procurement, integration,
and evaluation processes.

In today\'s environment, clusters of symmetric multi-processor (SMP)
computers are the most common choice to build a high-end computer
system. The monolithic architectures such as the Cray computers that
dominated the 1980s and early 1990s are no longer prevalent. Each new
system pushes the previous limits in scale and represents a breakthrough
in technology, requiring innovation and development to integrate the
system. In order to accomplish mission objectives, i.e., advance the
solution of complex scientific and engineering problems through the use
of simulation, at minimum cost and maximize the overall system value, it
is necessary to stay at the leading edge of the performance/price curve
reflected in Moore's Law. Thus, it is very common at this scale that HEC
procurements are for systems and technologies that do not exist at the
time of the procurement evaluation or contract signing. In all cases,
there are tradeoffs between the length of time for the procurement, the
costs involved both for the vendor and the procuring institution, the
performance levels, the solution costs, the reliability of the system,
and the risks of not getting what is contracted for. In view of the
rapidly evolving nature of this field, the procurement and contracting
process needs to be flexible and adapt readily to changes in technology
directions in order to be successful. This white paper focuses on two of
the primary procurement methods: (1) Technology-driven procurements, and
(2) Science driven procurements. A key conclusion is the need for
empowerment and flexibility within the context of best management
practices for structuring procurements.

# Technology Driven Procurements

Technology-driven procurements provide the ability to rapidly procure a
new technology for evaluation or in the case where the procuring agency
has designed a unique solution to meet the mission requirements of the
users and is looking for a vendor to build the system to specification.
This method is often a sole-source procurement as only one vendor has
the right technology/capability at the time, leading to reduced cost for
both the vendor and the procuring agency.

The DOE Office of Advanced Scientific Computing Research has supported a
project called "Evaluation of Early System". The goals of the project
are to select promising new technologies, acquire early versions of
these technologies, rapidly deploy the new systems so that application
scientists can get early access to the latest systems, and to publish
the results of the evaluation. During the evaluation of the system, a
suite of low level benchmarks, for example MPI ping-pong and the streams
memory bandwidth test, kernel applications, for example, the Parallel
Spectral Method Shallow Water Transport test and the NAS Parallel
Benchmarks, and full applications are tested and compared to other
available systems to understand the strengths and weaknesses of the
various machines in terms of both hardware and software. This
information is then made available to the public as openly as possible
(subject to non-disclosure agreements) to aid others in the procurement
process and to help in defining future directions for the procurement of
high-end architectures

As HEC procurements shift from the vendor-proprietary (e.g., IBM SP with
Power microprocessors and AIX) solution space to commodity hardware and
Open Source software solutions, there are many more choices that can be
made by the purchaser from Intel/AMD ecosystems for cluster components.
That is, the purchaser can choose, among other things, the processor
(e.g., IA-32, IA-64, IA32-64), node (e.g., dual or quad processors),
interconnect (e.g., Quadrics, Infiniband or Myrinet), cluster management
hardware and software, and packaging solutions. The procuring center
would use a technology-driven procurement process to acquire a large
commodity cluster built around a center-specific design based on a
myriad of choices from the space of potential solutions. In so doing,
the center can maximize performance/cost, but inherently assumes more
risk and responsibility in the success of the HEC system.

A main advantage of this style of procurement is that it yields a system
that is highly tuned to the technical requirements of the procuring
center and can be accomplished quickly in a rapidly changing
marketplace. By using this strategy, centers have been able to go from a
technology announcement to having a world class computer running in as
few as three months (although the average time varies). A second
advantage is that the price can be very attractive because the vendors
incur less pre-sales and proposal costs as well as much lower component
and integration costs. Higher discounts have been obtained from
commodity cluster procurements designed for the commodity market because
of economies of scale and the volume distribution method increases
competition and decreases margins at every level of the supply chain.
Another advantage of this style of procurement is that it becomes
feasible for centers to offset costs and increase the possibility of
success by entering into close collaborations with vendors. By
collaborating with the manufacturers, important information is passed to
the system designers about the unique ways that scientific and technical
applications stress the processors, memory, storage, and network
subsystems of these machines. This technical exchange is beneficial to
the center, the ultimate users in the scientific and engineering
community, and the manufacturers, as it allows them to develop products
better suited to HEC as well as to commercial customers.

# Science Driven Procurements

Whereas the technology-driven procurement is tailored towards rapid
evaluation of new systems or systems where the computer center has done
substantial system design work, it is essential that major investments
in computing technology be driven by scientific, programmatic, and/or
mission requirements. Open, competitive procurements based on these
requirements are one of the best ways to acquire a solution that meet
the needs and requirements of the user community. Open procurements are
often appropriate when there is the possibility of more than one
effective solution, the scale of the system(s) is large, and/or it is
mission-critical.

Regardless of the specific process chosen, open procurements start with
assessing the scientific mission requirements, which involves forming a
working team of scientific users, stakeholders, and the selection team.
The requirements are translated into technical specifications that match
the scientific mission requirements to the best technical options within
the available budget. This matching is often reflected in tests or
benchmarks that suppliers perform. Often there is interaction with the
scientific user community, and possibly the suppliers, as the derived
requirements and tests get refined. The principal task of the
acquisition team is to decide the best alternative among the available
choices based on current data. The effort involved both by the selection
team and the suppliers depends less on the type of process chosen than
on the sophistication of the testing. Due to the complexity of
predicting behavior on future architectures it is important to have a
balanced procurement team with expertise in computer hardware, operating
systems and management, computer scientists who understand advanced
architectures and algorithms, applied mathematicians who understand the
implementation of algorithms, and user software developers who know the
science to be done as well as the scaling of the algorithms.

Current Best Value Source Selection (BVSS) style procurements vary
widely in implementation, are very flexible, and do not force the
selection of the lowest bidder but result in the solution with the best
value. The results of such procurements range from cooperative research
and development contracts to firm fixed price contracts.

Procurements should be focused on fulfilling the requirements of the
user community and the stakeholder customers as opposed to prescribing
implementation details and features that are more the purview of the
vendor. Traditional system specification based procurements focus on the
detailed design of a system by specifying many of the internal
parameters of the potential system. The BVSS procurement process,
currently used for most HEC systems, accomplishes this by describing a
relatively small set of key requirements. This provides an opportunity
for vendor innovation in addressing the requirements and recognizes that
widely differing solutions cannot be encompassed in a detailed manner,
but must be broader. Best Value methods provide more flexibility and are
very cost effective. The evaluation of vendor responses is necessarily
qualitative with the emphasis on the total solution as opposed to
detailed feature-by-feature comparisons. The most common metric by which
to compare disparate vendor responses is Time-To-Solution (TTS). TTS
includes the development, pre- and post-processing times for an
application. By having more than one supplier involved in a full or
limited competitive procurement, the purchasing centers have the
advantage of more information and options from which to choose. There is
also the possibility that good solutions may come from sources that are
not originally expected. The supplier competition often provides
substantially lower costs and a more effective solution. Should the
preferred option not perform as expected, this competition will already
have identified an alternative solution.

Due to the scale of the largest system procurements being undertaken
today, full-scale benchmarking for the performance requirements of
targeted scientific applications is difficult if not impossible.
Nevertheless, the sheer scale of the systems may introduce design
considerations that will not surface using prototype benchmarking at
smaller scale. Recent successes in performance benchmarking suggest an
approach to a solution for this problem. For example, at Los Alamos,
accurate performance models for applications from the ASCI workload have
been developed and used for system design, optimization and maintenance,
and have even been used in the procurement process for the ASCI Purple
machine at LLNL, predicting the performance of the code SAGE on all the
competing machines. These results suggest that it is possible to
accurately predict the performance achieved by a future system (much
larger in size and employing a distinct design from hardware currently
in operation) running a future scientific application (much larger in
problem size than currently being run), thereby allowing the decision
makers to have at their disposal not only performance information but
more importantly being able to analyze "what if" scenarios.

# Expectations for success vary and must be clear 

Whether implicit or explicit, with large-scale procurements there are a
wide range of groups with expectations -- the suppliers, the purchasers,
the end users and the stakeholders who fund the systems. The end users
can have widely different goals ranging from the design, evaluation, and
testing of new architectures to the development of application software
to the solution of complex domain-specific problems. It is essential
that accurate and meaningful expectations be set during the procurement
process. Whereas procurement costs can be significant for both
purchasers and the suppliers, they are relatively small compared to the
resulting value and costs for the purchaser and supplier on large scale
systems. No matter what method is used to enter a contract, much of the
cost associated with the procurement is in defining the parameters of
the contract. It is in the purchaser's as well as the supplier's best
interest to have contracts that set clear expectations and parameters
for success. If the solution is loosely- or under-specified, the
technical solutions will not live up to the expectations of the
purchasers and their clients. A poorly written contract can also become
a problem for suppliers who then face large uncertain risk and to the
purchasers who then have to continually renegotiate the terms of the
contract. Even in the case of very fast procurements, there have been
examples where the contract negotiations have taken significant
additional time in order to set the proper expectations.

Thus it is important for all parties to fully understand and accept the
requirements, the expected performance and the risks. The more
uncertainty there is in the proposed solution, the more important it is
to fully document the requirements and commitments. This effort requires
resources and cost on both sides, but is essential.

# Conclusion

Methods used to purchase existing standardized commodity systems simply
do not scale to serve HEC procurements. There are some clear ways to
improve and streamline the process. Laboratories have developed
innovative methods that have been proven effective and should be allowed
to continue these approaches. If all the Federal Acquisition Regulations
applied directly to the Laboratories, the current BVSS and other
processes discussed above would need to be completely redesigned or even
totally abandoned.

*How can the Federal Government improve the process*? The goal is to be
able to do performance driven procurements instead of ones that design
the system *a priori*. Specific suggestions for improvements include:
allowing for different solutions to unique situations including
simultaneous negotiations with multiple vendors; enabling more open and
iterative communications about true needs and requirements with the
vendors; allowing multiple and flexible options for bigger/extra systems
that can be exercised quickly; and allowing unscored and/or unweighted
evaluations. Furthermore, it is clear that adjusting the traditional
guidance for issues such as Intellectual Property, Indemnification, and
Limitation of Liability would make the procurement process, or at least
contract negotiations, faster and more effective. Potential vendors
should be encouraged to get directly involved with the science mission
themselves. Once the science mission collaboration is established it
will produce a team where the vendor/center has a system defined to
better meet the scientific needs. In order to improve the state of High
End Computing, it is critical that the methods used to acquire the
systems and services be revitalized as well. There needs to be room for
flexibility, creativity and cooperation within the procurement process,
while still assuring the systems obtained meet the science mission
driven requirements.