**MRC Committee Minutes, 24 May 2001**

Where: 50B-6208

When: Thursday, 24 May 2001

Who: Sandy Merola, Ken Downing, Michel Van Hove, Dan Hawkes, Jon Bashor,
Jim Leighton, Alessandra Ciocio, Doug Olson, John Staples, Shaneen
Tonse, Ali Belkacem, Gary Jung

Issue: Consider Funding Models, particularly Ali\'s position paper

Ali Belkacem presented a well-thought-out paper, which will be known
always as the *Goat and Grain of Salt* paper.

It is essential that we start formulating a financial model now even
before we get a response from potential users. Getting funding is a long
and drawn-out process and we must be ready with a definite plan, and
users must be part of this plan. Any fuzziness will reduce the
effectiveness of this proposal.

Ali recommends that the most effective entry into MRC for LBNL is to
start with a user community built around the present Alvarez cluster.
The original goal of Alvarez is for NERSC to test the cluster concept
with a local set of users. This experiment will end in the summer of
2002, at which point the Alvarez cluster would be handed over for other
users. This would be the right time to transition Alvarez over to the
first phase of an LBNL machine, with appropriate funding for continued
operation and support.

The operation would require some amount of Lab overhead, as well as some
flavor of recharge by either the users themselves or their divisions,
most likely in the form of hardware purchases to expand an existing
cluster. Alvarez would be seen as a stepping-stone to a next machine,
not necessarily a cluster, if the users find that another architecture
is better.

Alvarez is almost a stepping-stone to NERSC, as the parallelization
effort for NERSC is close to that required by an Alvarez-like machine,
so most of that effort would not be lost, and the goal of training LBNL
NERSC users would be satisfied.

Whatever machine is selected must be brought in to most divisions to be
viable. Some divisions may decide to opt out, but could be brought in
later with specific contributions. All divisions would contribute to
overhead. The use of capital funds to support the hardware upgrades of
the MRC machine avoids the cost burden imposed on operating funds, and
should be seen as a more efficient way of supporting MRC than the
customary recharge model.

At the start of an MRC facility, avoid a recharge funding model
altogether, as it is seen as a killer. Alvarez already exists, and would
be the entry machine, and a following machine would be assembled using
capital funds as mentioned above. Some recharge would probably be
necessary, and those contributing capital funds would pay a lower
recharge rate.

Some divisions would contribute by the individual user or by the
program; in other divisions, the support would be division-wide. Some
divisions are largely non-DOE funded, and Sandy will check on the use of
non-DOE dollars for MRC.

The operating cost of an MRC, independent of the hardware cost, will be
on the order of one million dollars per year. We will need at least a
three year commitment of operating money to get started, and the model
would work for at least five years, after which the nature of computing
may change enough to require the development of a new funding model.

The Laboratory must partially subsidize an MRC facility. The Laboratory
will accrue several benefits from this facility:

A high-end computing mentality will develop among more users at LBNL

A high-end computing facility will be available to smaller groups

LDRD funds should be made available to subsidize an MRC

Even non-contributing divisions will benefit, as there will be available
at LBNL a high-end computing facility that would be available to them at
\"reasonable\" recharge rates. However, it is in our best interest to
have as many divisions as possible to buy in to MRC.

Procurement of a new machine will take on the order of a year, followed
by continuous upgrades in \"real time\". The procurement of the \$750K
Alvarez machine took about \$225K to do (6 staff 0.5 FTE for 6 months).
We can benefit from procurement work by NERSC to reduce this for an MRC
machine. NERSC had to satisfy national requirements, and an MRC should
be less expensive, as the audience is local.

In the short term, a request to earmark funds at the next division
director retreat will be made. The amount is not yet known, but depends
on which of two most likely scenarios will be followed: use Alvarez, or
go with an all-new facility. Both scenarios will be costed out, and by
October 2001 one of the two scenarios will be selected. The Alvarez
scenario may need approximately \$200K seed money, the other on the
order of a million in FY02. We will need working money by January 2002,
and will ask for it by June 15^th^ of this year as a placeholder.

Sandy will need a set (about 10) viewgraphs for the division director\'s
retreat, which Jon Bashor will help provide. We have almost all the
information we need for the informational document that Bashor will
prepare.

In the short term, we will clarify the financial model, establishing the
obligations for those who buy in, and for those divisions that don\'t.
Ali will work this out.

Gary will further refine the details of the acquisition of the local
clusters.

The following is the 2.5 pages "MRC financial model" document that Ali
submitted to this committee for discussion.

MRC financial model

May 22, 2001

**[The goat and the grain of salt]{.underline}**: The first part of the
present draft reviews in a few sentences the rationale behind the MRC
effort before presenting the financial model. In the likelihood a real
need for an MRC facility is identified, the tough question to answer is
what can the laboratory do about it? And how should we proceed from
there? The real challenge is finding a solution and financial model that
will work. In the process of doing that, it should be kept in mind that
a move ahead with an MRC facility would mean that the laboratory is
committing itself to a substantial investment (which is called spending
by those who don\'t subscribe to the idea). This investment has to be
sustained every year of the lifetime of the facility. Since the
laboratory doesn\'t operate in vacuum and new initiatives mushroom
regularly in DOE and elsewhere, institutional priorities shift and any
financial model should be able to shield the MRC funding from these
fluctuations.

This draft went through a filtered (meaning incomplete and biased) view
of the author, thus should be considered as such - A starting point for
a meaningful discussion of the financial model.

**[Rationale and goal]{.underline}**

The rationale behind the discussions and ongoing effort on mid-range
computing is the possibility that an easily accessible high performance
computing facility could become key for accomplishing 21^st^ century
science at the Laboratory. So far the Laboratory has been lacking a road
map and strategic planning in this area. This lack of strategic planning
has in the past led to some haste decisions and failed attempts, which
in turn had the adverse effect of steering the Laboratory away from
mid-range computing. Strategic planning in mid-range computing should
address two separate but essential issues.

\- The first one has to do with usefulness. To be useful and succeed the
mid-range computing facility:

a)  Should respond to the needs of a broad range of users.

b)  Should provide much more than a user-owned cluster-machine can
    provide. It should be available, it should have a high-turn-around,
    it should have a configuration that responds to the needs of the
    user and it should not require a rocket scientist to run codes on it
    unless a rocket-scientist helper is provided with it.

c)  Should be perceived by a scientist owning a small cluster-machine as
    a major step-up in terms of computer-power when needed and cutting
    edge software.

d)  Should be upgradeable and upgraded regularly to keep up with
    advances in technology

e)  Should be much more cost effective dollar-wise than owning a
    small-cluster machine.

f)  Should be operated by experts and with expertise

g)  Should be operated with responsiveness to user needs, requests and
    input.

\- The second one has to do with commitment. There should be a clearly
expressed need by scientists and their involvement. There should be a
strong commitment of the scientific divisions. There should be a strong
commitment of the Lab management. Finally, the timing should be right.

The list of requirements listed above obviously will put a major burden
on the design, operation and sustained funding of a mid-range computing
facility even when a strong need is identified. It will be extremely
difficult to achieve this goal overnight with a turnkey solution and a
more gradual approach may make much more sense. Since the Laboratory
will be starting from ground zero it will take a few years before
achieving the desired goal of a well-oiled facility that truly responds
to the needs and a facility that evolves in step as these needs evolve
with time.

# Financial model

The financial model should be driven by a vision that takes into account
the realities of the laboratory.

\- First and foremost the laboratory has been bringing down overhead and
that will not change any time soon. We should keep in mind that funding
something completely from overhead is tantamount to recharging everybody
at the Laboratory - there is going to be a major resistance because the
commitment is not a one time thing but year after year.

\- Relying in a large part on recharge to fund the operation and
upgrades of a facility is a non-starter. It killed the Masspar (although
not the only factor) and will likely do the same thing here.

\- Some scientific divisions spend every year a substantial amount of
money on scientific computing (hardware + software + support) - The
hardware part is usually purchased on capital equipment and is decided
at the division director level. The divisions will continue spending
that capital equipment money any way and for them it doesn\'t make sense
to go to recharge mode.

\- The laboratory needs some institutional computing resource that can
be used for LDRD projects and strategic hires.

When all this is put together it becomes clear that the MRC financial
model has to involve a strong commitment and funding from the scientific
programs, the divisions, and laboratory overhead. The ultimate goal
would be a facility that belongs to the scientific divisions, that is
configured with input from the users, operated by ITSD (leveraging some
resources from NERSC). The operation and system management is funded
through overhead.

[Haste is the enemy of success]{.underline}: A step-by-step approach is
necessary to achieve the desired goal. The first step is to establish
that ITSD (with leverage from NERSC) can operate a MRC facility. It is
very wise and very good politics to use the Alvarez cluster as the
starting point. For the first two years the funding should come from the
Laboratory to make Alvarez a MRC machine.

**Very important**: While this is ongoing the next machine or upgrade of
Alvarez should move into the design stage. This will involve
representatives of divisions (identified users) and programs. The goal
is to get a consensus on what to do next. A lot of handholding is needed
during this two-year phase at the end-of-which hardware is purchased by
the divisions (or some divisions). The handholding should be provided by
experts from ITSD and NERSC (and maybe from outside).

What the division gets from is a free-access and use in proportion to
the contributions. This could present major benefits to the
divisions: 1) they don\'t have to pay for the operation and management
of the system. They don\'t have to pay for the operating system and the
major software packages. They don\'t have to pay for a maintenance
contract. And they don\'t have to deal with computer security issues.
When something doesn\'t work instead of just complaining about it they
can actually decide what should be done about it and get it done.

What the laboratory gets out of it is institutional computing resource
that can be used for LDRD projects and strategic hires.

The ideal scenario would of course be to get all the divisions buy in.
This is unlikely to happen but it is essential to get a majority of them
buy in. The division that doesn\'t contribute will be recharged for
usage. The recharge user should get a very low priority for usage.