**LQCD Computing BY 2008 EXHIBIT 300**

**[Part I: Summary Information and Justification]{.underline}**

In Part I, complete sections A, B, C, and D for all capital assets (IT
and non-IT). Complete sections E anf F for IT capital Assets.

**Section A: Overview**

+-----------------+----------------------------------------------------+
| The following   |                                                    |
| series of       |                                                    |
| questions are   |                                                    |
| to be completed |                                                    |
| for all         |                                                    |
| investments.    |                                                    |
| These questions |                                                    |
| will be used by |                                                    |
| OMB to identify |                                                    |
| which agency    |                                                    |
| and bureau is   |                                                    |
| responsible for |                                                    |
| managing each   |                                                    |
| capital asset,  |                                                    |
| which OMB MAX   |                                                    |
| budget account  |                                                    |
| funds the       |                                                    |
| project, the    |                                                    |
| kind of         |                                                    |
| project, who to |                                                    |
| contact with    |                                                    |
| questions about |                                                    |
| the information |                                                    |
| provided in the |                                                    |
| Exhibit 300,    |                                                    |
| and whether or  |                                                    |
| not it is an IT |                                                    |
| or a non-IT     |                                                    |
| capital asset.  |                                                    |
+-----------------+----------------------------------------------------+
| 1\. Date of     |                                                    |
| Submission      |                                                    |
+-----------------+----------------------------------------------------+
| 2\. Agency      | (Department of Energy)                             |
+-----------------+----------------------------------------------------+
| 3\. Bureau      | High Energy Physics, Nuclear Physics               |
+-----------------+----------------------------------------------------+
| 4\. Name of     | SC Lattice Quantum ChromoDynamics (LQCD) Computing |
| this Capital    |                                                    |
| Asset           |                                                    |
+-----------------+----------------------------------------------------+
| 5\. Unique      | 019-20-01-21-01-1032-00-109-026                    |
| (Investment)    |                                                    |
| Identifier      |                                                    |
+-----------------+----------------------------------------------------+
| 6\. What kind   | Mixed Life Cycle                                   |
| of investment   |                                                    |
| will this be in |                                                    |
| FY 2008?        |                                                    |
+-----------------+----------------------------------------------------+
| 7\. What was    | FY2006                                             |
| the first       |                                                    |
| budget year     |                                                    |
| this investment |                                                    |
| was submitted   |                                                    |
| to OMB?         |                                                    |
+-----------------+----------------------------------------------------+
| 8\. Provide a   | The LQCD Computing Investment acquires, operates,  |
| brief summary   | and maintains computational systems that will      |
| of this         | serve as significant resources for the national    |
| investment      | LQCD user community. Given the known computational |
|                 | requirements, configurable commercial              |
|                 | off-the-shelf components will be acquired through  |
|                 | firm-fixed-price contracts. Using these resources, |
|                 | LQCD theorists can better provide theoretical      |
|                 | insight and guidance to the community of           |
|                 | approximately 4,500 particle and nuclear           |
|                 | physicists.\                                       |
|                 | \                                                  |
|                 | The performance of LQCD on a given computational   |
|                 | resources is measured by the floating point        |
|                 | operations per second (flop/s) using the actual    |
|                 | production scientific codes. The total capacity of |
|                 | a given system is then rated as million (mega) or  |
|                 | trillion (tera) flop/s of sustained performance.   |
|                 | The price performance for LQCD of a given          |
|                 | computational resource is measured as dollars per  |
|                 | megaflop/s (\$ per megaflop/s, sustained).         |
|                 | Throughout this document computer performance or   |
|                 | price/performance, refers to performance sustained |
|                 | by production codes, not to theoretical peak       |
|                 | rates.\                                            |
|                 | \                                                  |
|                 | Milestones are specified by giving the average     |
|                 | performance of the sparse matrix inversion routine |
|                 | that consumes the bulk of the floating point       |
|                 | operations in any LQCD calculation. By the end of  |
|                 | FY 2007, it is anticipated that the project will   |
|                 | be operating facilities with an aggregate capacity |
|                 | of 11.4 teraflop/s. The objective for FY 2008 is   |
|                 | to acquire an additional 4.2 teraflop/s for        |
|                 | production calculations with an average            |
|                 | price/performance of \$0.33 per sustained          |
|                 | megaflop/s. Thereafter, the objective is to take   |
|                 | advantage of the improvements in technology        |
|                 | implied by Moore\'s law, as well as the specific   |
|                 | nature of QCD calculations to advance a series of  |
|                 | increasingly powerful capabilities for science.    |
|                 | The hardware will be maintained at Brookhaven      |
|                 | National Laboratory (BNL), Fermi National          |
|                 | Accelerator Laboratory (FNAL), and the Thomas      |
|                 | Jefferson National Accelerator Facility (TJNAF).   |
|                 | The deployment location(s) of new hardware will be |
|                 | determined each year by peer review and            |
|                 | availability of funding. The effort will be highly |
|                 | leveraged, benefiting from the expertise of the    |
|                 | existing computer staffs and infrastructure of     |
|                 | these laboratories. The facilities will be         |
|                 | available to the entire U.S. lattice QCD           |
|                 | community.                                         |
|                 |                                                    |
|                 | In each year, additional hardware will be acquired |
|                 | and operated together with the existing systems.   |
|                 | Accordingly, the fraction of the budget for SS     |
|                 | activities will increase relative to DME           |
|                 | activities in each year.                           |
+-----------------+----------------------------------------------------+

  ------------------------------------------------------ -------- -------
  9\. Did the Agency's Executive/Investment Committee    Yes      
  approve this request?                                           

  ------------------------------------------------------ -------- -------

a\. If "yes" what was the date of this approval?

7/15/2006

  ------------------------------------------------------ -------- -------
  10\. Did the Project Manager review this Exhibit?      Yes      

  ------------------------------------------------------ -------- -------

11\. Contact Information of Project Manager?

Name Jehanne Simon-Gillo

Phone Number 301-903-1455

Email Jehanne.Simon-Gillo@science.doe.gov

  -------------------------------------------------------- ------- -------
  13\. Does this investment support one of the PMA         Yes     
  initiatives?                                                     

  -------------------------------------------------------- ------- -------

If "yes" check all that apply:

-   Human Capital

-   Budget Performance Integration

-   Financial Performance

-   Expanded E-Government

-   Competitive Sourcing

-   Faith Based and Community

-   Real Property Asset Management

-   Eliminating Improper Payments

-   Privatization of Military Housing

> X Research & Development Investment Criteria

-   Housing & Urban Development Management & Performance

-   Broadening Health Insurance overage through State Initiatives

-   "Right Sized" Overseas Presence

-   Coordination of VA and DoD Programs and Systems

  ------------------------------------------------------ ------- --------
  14\. Does this investment support a program assessed           No
  using OMB's Assessment Rating Tool (PART)?                     

  ------------------------------------------------------ ------- --------

  ----------------------------------------------------- -------- ---------
  a\. If "yes" does this investment address a weakness           
  found during the PART review                                   

  ----------------------------------------------------- -------- ---------

> b\. If "yes" what is the name of the PART program assessed by OMB's
> Program Assessment
>
> Rating Tool?

c\. If "yes" what PART rating did it receive?

  ------------------------------------------------------ ------- --------
  15\. Is this investment for information technology?    Yes     

  ------------------------------------------------------ ------- --------

> If the answer to the question above was "Yes," complete questions
> 16-23 below. If the answer is "No," do not answer questions 16-23.

16\. What level is the IT Project (Check One)?

> X Level 1

-   Level 2

-   Level 3

17\. What project management qualifications does the Project Manager
have?

> X Project manager has been validated as qualified for this investment

-   Project manager qualification is under review for this investment

-   Project manager assigned to investment, but does not meet
    requirements

-   Project manager assigned but qualification status review has not yet
    started

-   No Project manager has yet been assigned to this investment

  ---------------------------------------------------- -------- ---------
  18\. Is this investment identified as "high risk" on          No
  the Q4 -- FY 2006 agency high risk report?                    

  ---------------------------------------------------- -------- ---------

  ------------ ------------ -------------------------------------- ------- --------
  19\. Is this                                                             No
  a financial                                                              
  management                                                               
  system?                                                                  

               a\. If "yes"                                                
               does this                                                   
               investment                                                  
               address a                                                   
               FFMIA                                                       
               compliance                                                  
               area?                                                       

                            1\. If "yes" which compliance area?            

                            2\. If "no" what does it address?              

               b\. If "yes"                                                
               please                                                      
               identify the                                                
               system                                                      
               name(s) and                                                 
               system                                                      
               acronym(s)                                                  
               as reported                                                 
               in the most                                                 
               recent                                                      
               financial                                                   
               systems                                                     
               inventory                                                   
               update                                                      
               required by                                                 
               Circular                                                    
               A-11 section                                                
               52.                                                         
  ------------ ------------ -------------------------------------- ------- --------

20\. What is the percentage breakout for the total FY2008 funding
request for the following?

> 66 Hardware

-   Software

> 34 Services

-   Other

  -------------------------------------------------- ------ ------- ------
  21\. If this project produces information                         N/A
  dissemination products for the public, are these                  
  products published to the Internet in conformance                 
  with OMB Memorandum 05-04 and included in your                    
  agency inventory, schedules and priorities?                       

  -------------------------------------------------- ------ ------- ------

22\. Contact Information of individual responsible for privacy related
questions:

-   Name Don Holmgren

-   Phone Number 630-840-2745

-   Title Contract Project Manager

-   E-mail djholm@fnal.gov

  --------------------------------------------------------- ------ ------
  23\. Are the records produced by this investment          Yes    
  appropriately scheduled with the National Archives and           
  Records Administration' approval?                                

  --------------------------------------------------------- ------ ------

**Section B: Summary of Funding**

Provide the total estimated life-cycle cost for this investment by
completing the following table. All amounts represent budget authority
in millions, and are rounded to three decimal places. Federal personnel
costs should be included only in the row designated "Government FTE
Costs," and should be excluded from the amounts shown for "Planning,"
"Full Acquisition," and "Operation/Maintenance." The total estimated
annual cost of the investment is the sum of costs for "Planning," "Full
Acquisition," and "Operation/Maintenance." For Federal buildings and
facilities, life-cycle costs should include long term energy,
environmental, decommissioning, and/or restoration costs. The costs
associated with the entire life-cycle of the investment should be
included in this report.

  ------------- ---------- ------- ------- ------- ------- ------ ------ -------- --------------
                                                                                  

  Table 1:                                                                        
  SUMMARY OF                                                                      
  SPENDING FOR                                                                    
  PROJECT                                                                         
  PHASES                                                                          
  (REPORTED IN                                                                    
  MILLIONS) All                                                                   
  amounts                                                                         
  represent                                                                       
  Budget                                                                          
  Authority                                                                       
  (Estimates                                                                      
  for BY+1 and                                                                    
  beyond are                                                                      
  for planning                                                                    
  purposes only                                                                   
  and do not                                                                      
  represent                                                                       
  budget                                                                          
  decisions)                                                                      

                PY-1 &     PY 2006 CY 2007 BY 2008 BY +1   BY+2   BY+3   BY+4     Total
                Spending                           2009    2010   2011   2012 and 
                Prior to                                                 beyond   
                2006                                                              

  Planning                 0.025   0.114   0.119   0.123                          0.381

  Acquisition              1.850   1.616   1.540   0.668                          5.674

  Subtotal                 1.875   1.730   1.659   0.791                          6.055
  Planning &                                                                      
  Acquisition                                                                     

  Operations &             0.625   0.770   0.841   0.909                          3.145
  Maintenance                                                                     

  TOTAL                    2.500   2.500   2.500   1.700                          9.200

  Government                                                                      
  FTE Costs                                                                       
  should not be                                                                   
  included in                                                                     
  the amounts                                                                     
  provided                                                                        
  above.                                                                          

  Government               0.050   0.050   0.050   0.100                          0.250
  FTE Costs                                                                       

  Number of FTE            0.25    0.24    0.23    0.44                           
  represented                                                                     
  by cost                                                                         
  ------------- ---------- ------- ------- ------- ------- ------ ------ -------- --------------

***Note:*** For the cross-agency investments, this table should include
all funding (both managing partner and partner agencies).

Government FTE Costs should not be included as part of the TOTAL
represented.

  --------------------------------------------------------- ------- ------
  2\. Will this project require the agency to hire                  No
  additional FTE's?                                                 

  --------------------------------------------------------- ------- ------

a\. How many and in what year?

3\. If the summary of spending has changed from the FY2007 President's
budget request, briefly explain those changes.

> From our operations experience in the first half of FY2006, we
> determined that additional manpower for operations was required in
> FY2007-FY2009. Funding was shifted accordingly from Acquisition to
> Maintenance & Operations. The amounts shifted by year in \$million
> were \$0.08, \$0.09, and \$0.13, respectively, in FY2007, FY2008, and
> FY2009.
>
> **Section C: Acquisition/Contract Strategy (All Capital Assets)**

1\. Complete the table for all contracts and/or task orders in place or
planned for this investment:

  --------------- ------------ ----------- ----------- ----------- -------------- ------------- --------------- ------- ----------- ----------
  Contract or     Type of      Has the     Start & end Total Value Is this an     Is it         Competitively   SIS,    Is EVM in   Does the
  Task Order      Contract/    contract    date of     of          Interagency    performance   awarded? (Y/N)  ESPC or the         contract
  Number          Task Order   been        Contract/   Contract/   Acquisition?   based? (Y/N)                  UESC?   contract?   include
                               awarded, if Task Order  Task Order  (Y/N)                                        (Y/N)   (Y/N)       the
                               so what is              (\$M)                                                                        required
                               the date of                                                                                          security &
                               award? If                                                                                            privacy
                               not, what                                                                                            clauses?
                               is the                                                                                               (Y/N)
                               planned                                                                                              
                               award date?                                                                                          

  SURA-06-C0350   Firm-fixed   2/28/2006   2/28/2006   \$280K      N              N             Y               N       N           N\*
                  price                                                                                                             

  FNAL-BEC 186004 Firm-fixed   5/17/2006               \$1430K     N              N             Y               N       N           N\*
                  price                                                                                                             

  Planned FY07 at Firm-fixed   2/1/2007                \$1480K     N              N             Y               N       N           N\*
  TJNAF           price                                                                                                             

  Planned FY08 at Firm-fixed   2/1/2008                \$1400K     N              N             Y               N       N           N\*
  FNAL            price                                                                                                             

  Planned FY09 at Firm-fixed   2/2/2009                \$830K      N              N             Y               N       N           N\*
  FNAL            price                                                                                                             
  --------------- ------------ ----------- ----------- ----------- -------------- ------------- --------------- ------- ----------- ----------

**\*** These task orders cover the purchase of computer hardware and
some physical integration only. Systems integration and operation of
this computer hardware are performed by the host laboratories. The host
laboratories' M&O contracts do cover security and privacy, and these
requirements are satisfied by the laboratories' staff, not through the
task orders in the table.

+-----------------------------------------------------------------------+
| 2\. If earned value is not required or will not be a contract         |
| requirement for any of the contracts or task orders above, explain    |
| why:                                                                  |
|                                                                       |
| The DOE has determined that this investment does not meet the         |
| criteria requiring EVM.                                               |
+-----------------------------------------------------------------------+

  ----------------------------------------------- ------- ------- --------
  3\. Do the contracts ensure Section 508         Yes             
  compliance?                                                     

  ----------------------------------------------- ------- ------- --------

  -------------------------------------------------------- ------- ------
  4\. Is there an acquisition plan which has been approved Yes     
  in accordance with agency requirements?                          

  -------------------------------------------------------- ------- ------

a\. If "yes," what is the date?

May 25, 2005

  -------------------------------------------------------- ------ -------
  b\. If "no," will an acquisition plan be developed?             

  -------------------------------------------------------- ------ -------

1\. If "no," briefly explain why:

**I.D. Performance Information (All Assets)**

Agencies must use Table 1 below for reporting performance goals and
measures for all existing IT investments that were initiated prior to FY
2005. The table should be extended to include measures beyond FY 06.

Table 1

+----+---------+----------+-----------+-------------+----------------+
| Fi | St      | Per      | Actual    | Planned     | Performance    |
| sc | rategic | formance | /baseline |             |                |
| al | Goal(s) | Measure  | (from     | Performance | Metric Results |
|    |         |          | Previous  |             | (Actual)       |
| Ye | Su      |          | Year)     | Metric      |                |
| ar | pported |          |           | (Target)    |                |
+====+=========+==========+===========+=============+================+
| 20 |         |          |           |             |                |
| 06 |         |          |           |             |                |
+----+---------+----------+-----------+-------------+----------------+
| 20 |         |          |           |             |                |
| 06 |         |          |           |             |                |
+----+---------+----------+-----------+-------------+----------------+
| 20 |         |          |           |             |                |
| 07 |         |          |           |             |                |
+----+---------+----------+-----------+-------------+----------------+
| 20 |         |          |           |             |                |
| 07 |         |          |           |             |                |
+----+---------+----------+-----------+-------------+----------------+
| 20 |         |          |           |             |                |
| 08 |         |          |           |             |                |
+----+---------+----------+-----------+-------------+----------------+

Table 2

+----+--------+--------+-----------+--------+------------+-----------+
| Fi | Measu  | Measu  | Me        | Ba     | Planned    | Actual    |
| sc | rement | rement | asurement | seline | I          | Results   |
| al | Area   | Gr     | Indicator |        | mprovement |           |
| Ye |        | ouping |           |        | to the     |           |
| ar |        |        |           |        | Baseline   |           |
+====+========+========+===========+========+============+===========+
| 20 | M      | G      | Improved  | 40\^3  | 40\^3 x 96 |  (2/      |
| 06 | ission | eneral | staggered | x 96   | completed, | 16/2006)\ |
|    | and    | S      | confi     | la     | 48\^3 x    | 40\^3x96: |
|    | Bu     | cience | gurations | ttices | 144 50%    | 2802 of   |
|    | siness | and    | generated | gene   | completed  | 3400      |
|    | R      | Inno   | for       | ration | at one     | tra       |
|    | esults | vation | studies   | 1/3    | quark      | jectories |
|    |        |        | of CKM    | com    | mass, 25%  | done.     |
|    |        |        | matrix    | pleted | completed  | 4         |
|    |        |        | and       |        | at a       | 8\^3x144: |
|    |        |        | hadron    |        | second     | 1443 of   |
|    |        |        | structure |        | (1.0       | 1900      |
|    |        |        | \[SC      |        | TF-Yrs)    | tra       |
|    |        |        | Goals 4,  |        |            | jectories |
|    |        |        | 6\]       |        |            | at one    |
|    |        |        | \[NP-1\]  |        |            | mass      |
|    |        |        |           |        |            | done. 12  |
|    |        |        |           |        |            | of 269 at |
|    |        |        |           |        |            | a second  |
|    |        |        |           |        |            | mass      |
|    |        |        |           |        |            | done. On  |
|    |        |        |           |        |            | pace.     |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Ca        | Ba     | C          |  (2/      |
| 06 | ission | eneral | lculation | seline | alculation | 16/2006)\ |
|    | and    | S      | of CKM    | calcu  | with 40\^3 | D-mesons: |
|    | Bu     | cience | matrix    | lation | x 96       | 28\^3x96  |
|    | siness | and    | elements  | with   | lattices   | complete, |
|    | R      | Inno   | with      | 28\^3  | completed  | 37% done  |
|    | esults | vation | improved  | x 96   | (1.0       | on        |
|    |        |        | staggered | la     | ter        | 40\^3x96, |
|    |        |        | quarks    | ttices | aflops-yr) | to finish |
|    |        |        | \[SC      | com    |            | April 30. |
|    |        |        | Goals 4,  | pleted |            |           |
|    |        |        | 6\]       | (\<    |            |           |
|    |        |        | \[NP-1\]  | 0.5    |            |           |
|    |        |        |           | t      |            |           |
|    |        |        |           | eraflo |            |           |
|    |        |        |           | ps-yr) |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Hybrid    | \< 0.5 | 0.8 TF-yrs |  (2/      |
| 06 | ission | eneral | ca        | TF-Yrs | completed  | 16/2006)\ |
|    | and    | S      | lculation | ba     |            | 0.61 of   |
|    | Bu     | cience | of quark  | seline |            | 0.8       |
|    | siness | and    | structure | calcu  |            | TF-yrs    |
|    | R      | Inno   | of        | lation |            | complete  |
|    | esults | vation | nucleon   | com    |            | (on       |
|    |        |        | in chiral | pleted |            | pace).    |
|    |        |        | regime,   |        |            |           |
|    |        |        | including |        |            |           |
|    |        |        | GPDs \[SC |        |            |           |
|    |        |        | Goals 4,  |        |            |           |
|    |        |        | 6\]       |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Ca        | \< 0.5 | 1.0 TF-Yrs |  (2/      |
| 06 | ission | eneral | lculation | TF-Yrs | c          | 16/2006)\ |
|    | and    | S      | of        | ba     | alculation | 0.54      |
|    | Bu     | cience | p         | seline | completed  | TF-yrs    |
|    | siness | and    | roperties | calcu  |            | complete, |
|    | R      | Inno   | of hot    | lation |            | ahead of  |
|    | esults | vation | hadronic  | com    |            | pace.     |
|    |        |        | and quark | pleted |            |           |
|    |        |        | matter in |        |            |           |
|    |        |        | chiral    |        |            |           |
|    |        |        | regime    |        |            |           |
|    |        |        | \[SC      |        |            |           |
|    |        |        | Goals 5,  |        |            |           |
|    |        |        | 6\]       |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | Pl     | Resources | 80% of | 90% of     |  (2/      |
| 06 | ission | anning | are being | time   | time used  | 16/2006)\ |
|    | and    | and    | used in   | used   | by         | 90% of    |
|    | Bu     | Re     | a         | by     | approved   | time used |
|    | siness | source | ccordance | ap     | projects   | by        |
|    | R      | Allo   | with      | proved |            | approved  |
|    | esults | cation | a         | pr     |            | projects. |
|    |        |        | llocation | ojects |            |           |
|    |        |        | committee |        |            |           |
|    |        |        | decisions |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Pro    | Produc | Meta      | A      | Average    |  (2/      |
| 06 | cesses | tivity | -facility | verage | machine    | 16/2006)\ |
|    | and    | and    | achieves  | m      | uptime of  | 93%       |
|    | Acti   | Effi   | high      | achine | 88%        | uptime.   |
|    | vities | ciency | uptime    | uptime |            |           |
|    |        |        | and       | of 80% |            |           |
|    |        |        | e         |        |            |           |
|    |        |        | fficiency |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Tech   | E      | Aggregate | 5.9    | \+ 2.7 =   |  (2       |
| 06 | nology | ffecti | resource  | TF\    | 8.6 TF\    | /16/2006) |
|    |        | veness | sustained | This   | This would |           |
|    |        |        | teraflops | capa   | establish  | Hardware  |
|    |        |        | for       | bility | sufficient | acq       |
|    |        |        | Asqtad +  | allows | capability | uisitions |
|    |        |        | DWF (1:1) | the    | for the    | in        |
|    |        |        |           | comp   | planned    | progress. |
|    |        |        |           | letion | 2007       |           |
|    |        |        |           | of the | physics    |           |
|    |        |        |           | p      | program.   |           |
|    |        |        |           | hysics |            |           |
|    |        |        |           | p      |            |           |
|    |        |        |           | rogram |            |           |
|    |        |        |           | p      |            |           |
|    |        |        |           | lanned |            |           |
|    |        |        |           | for    |            |           |
|    |        |        |           | 2006.  |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Domain    | 16\^3  | 24\^3 x 64 |  (2/      |
| 06 | ission | eneral | wall      | x 32   | at         | 16/2006)\ |
|    | and    | S      | lattice   | gene   | generation | G         |
|    | Bu     | cience | ge        | ration | one quark  | enerating |
|    | siness | and    | nerations | com    | mass       | at 3      |
|    | R      | Inno   | \[SC      | pleted | completed  | masses,   |
|    | esults | vation | Goals     | , 0.5  | (1.0 TF -  | 2540 of   |
|    |        |        | 4,6\]     | TF     | years)     | 5000      |
|    |        |        |           | -years |            | tra       |
|    |        |        |           |        |            | jectories |
|    |        |        |           |        |            | complete. |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | Time   | Response  | 80% of | 85% of     |  (2/      |
| 06 | stomer | liness | to        | t      | tickets    | 16/2006)\ |
|    | R      | and    | service   | ickets | closed     | Estimate  |
|    | esults | Re     | requests  | closed | within 48  | 90% of    |
|    |        | sponsi |           | within | hours      | tickets   |
|    |        | veness |           | 48     |            | closed    |
|    |        |        |           | hours  |            | within 48 |
|    |        |        |           |        |            | hours.    |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Improved  | 40\^3  | 48\^3 x    |           |
| 07 | ission | eneral | staggered | x 96   | 144        |           |
|    | and    | S      | confi     | la     | lattices   |           |
|    | Bu     | cience | gurations | ttices | completed  |           |
|    | siness | and    | generated | com    | (1.65      |           |
|    | R      | Inno   | for       | pleted | TF-Yrs)    |           |
|    | esults | vation | studies   | 48\^3  |            |           |
|    |        |        | of CKM    | x 144  |            |           |
|    |        |        | matrix    | la     |            |           |
|    |        |        | and       | ttices |            |           |
|    |        |        | hadron    | in     |            |           |
|    |        |        | structure | pr     |            |           |
|    |        |        | \[SC      | ogress |            |           |
|    |        |        | Goals 4,  | (1.75  |            |           |
|    |        |        | 6\]       | TF-    |            |           |
|    |        |        | \[NP-1\]  | Years) |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Ca        | Ba     | C          |           |
| 07 | ission | eneral | lculation | seline | alculation |           |
|    | and    | S      | of CKM    | calcul | with 48\^3 |           |
|    | Bu     | cience | matrix    | ations | x 144      |           |
|    | siness | and    | elements  | with   | lattices   |           |
|    | R      | Inno   | with      | 40\^3  | completed  |           |
|    | esults | vation | improved  | x 96   | (1.5       |           |
|    |        |        | staggered | la     | TF-Yrs)    |           |
|    |        |        | quarks    | ttices |            |           |
|    |        |        | \[SC      | com    |            |           |
|    |        |        | Goals 4,  | pleted |            |           |
|    |        |        | 6\]       | (1.0   |            |           |
|    |        |        |           | T      |            |           |
|    |        |        |           | F-Yrs) |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Hybrid    | 0.8    | 1.8        |           |
| 07 | ission | eneral | ca        | te     | tera       |           |
|    | and    | S      | lculation | raflop | flops-year |           |
|    | Bu     | cience | of quark  | s-year | c          |           |
|    | siness | and    | structure | calcu  | alculation |           |
|    | R      | Inno   | of        | lation | (total)    |           |
|    | esults | vation | nucleon   | com    | completed  |           |
|    |        |        | in chiral | pleted |            |           |
|    |        |        | regime,   |        |            |           |
|    |        |        | including |        |            |           |
|    |        |        | GPDs \[SC |        |            |           |
|    |        |        | Goals 4,  |        |            |           |
|    |        |        | 6\]       |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | P         | 0.5    | 1.25       |           |
| 07 | ission | eneral | entaquark | te     | tera       |           |
|    | and    | S      | and N\*   | raflop | flops-year |           |
|    | Bu     | cience | spe       | s-year | c          |           |
|    | siness | and    | ctroscopy | ba     | alculation |           |
|    | R      | Inno   | in the    | seline | (total)    |           |
|    | esults | vation | chiral    | calcu  | completed  |           |
|    |        |        | regime    | lation |            |           |
|    |        |        | \[SC      | com    |            |           |
|    |        |        | Goals 4,  | pleted |            |           |
|    |        |        | 6\]       |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Ca        | 1.0    | 2.25       |           |
| 07 | ission | eneral | lculation | te     | tera       |           |
|    | and    | S      | of        | raflop | flops-year |           |
|    | Bu     | cience | p         | s-year | c          |           |
|    | siness | and    | roperties | calcu  | alculation |           |
|    | R      | Inno   | of hot    | lation | (total)    |           |
|    | esults | vation | hadronic  | com    | completed  |           |
|    |        |        | and quark | pleted |            |           |
|    |        |        | matter in |        |            |           |
|    |        |        | chiral    |        |            |           |
|    |        |        | regime    |        |            |           |
|    |        |        | \[SC      |        |            |           |
|    |        |        | Goals 5,  |        |            |           |
|    |        |        | 6\]       |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Improved  | 40\^3  | 48\^3 x    |           |
| 07 | ission | eneral | staggered | x 96   | 144        |           |
|    | and    | S      | confi     | la     | lattices   |           |
|    | Bu     | cience | gurations | ttices | completed  |           |
|    | siness | and    | generated | com    | (1.65      |           |
|    | R      | Inno   | for       | pleted | TF-Yrs)    |           |
|    | esults | vation | studies   | 48\^3  |            |           |
|    |        |        | of CKM    | x 144  |            |           |
|    |        |        | matrix    | la     |            |           |
|    |        |        | and       | ttices |            |           |
|    |        |        | hadron    | in     |            |           |
|    |        |        | structure | pr     |            |           |
|    |        |        | \[SC      | ogress |            |           |
|    |        |        | Goals 4,  | (1.75  |            |           |
|    |        |        | 6\]       | TF-    |            |           |
|    |        |        | \[NP-1\]  | Years) |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | Domain    | 24\^3  | 24\^3 x 64 |           |
| 07 | ission | eneral | wall      | x 64   | generation |           |
|    | and    | S      | lattice   | gene   | at a       |           |
|    | Bu     | cience | g         | ration | second     |           |
|    | siness | and    | eneration | at one | quark mass |           |
|    | R      | Inno   | \[SC      | quark  | completed  |           |
|    | esults | vation | Goals     | mass   | 32\^3 x 64 |           |
|    |        |        | 4,6\]     | com    | generation |           |
|    |        |        |           | pleted | at one     |           |
|    |        |        |           | (1.0   | quark mass |           |
|    |        |        |           | T      | 25%        |           |
|    |        |        |           | F-Yrs) | completed  |           |
|    |        |        |           |        | (3.0       |           |
|    |        |        |           |        | TF-Yrs)    |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | S      | Number of | 20     | 25         |           |
| 07 | stomer | ervice | users     |        |            |           |
|    | R      | Co     | running   |        |            |           |
|    | esults | verage | jobs each |        |            |           |
|    |        |        | month     |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | Time   | Response  | 85% of | 90% of     |           |
| 07 | stomer | liness | to        | t      | tickets    |           |
|    | R      | and    | service   | ickets | closed     |           |
|    | esults | Re     | requests  | closed | within 2   |           |
|    |        | sponsi |           | within | business   |           |
|    |        | veness |           | 2      | days       |           |
|    |        |        |           | bu     |            |           |
|    |        |        |           | siness |            |           |
|    |        |        |           | days   |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | Cu     | Customer  | A      | Average    |           |
| 07 | stomer | stomer | sat       | verage | score of   |           |
|    | R      | B      | isfaction | score  | 8.0        |           |
|    | esults | enefit | survey    | of 7.0 |            |           |
|    |        |        | (         |        |            |           |
|    |        |        | Customers |        |            |           |
|    |        |        | rate      |        |            |           |
|    |        |        | sat       |        |            |           |
|    |        |        | isfaction |        |            |           |
|    |        |        | with the  |        |            |           |
|    |        |        | service   |        |            |           |
|    |        |        | provided  |        |            |           |
|    |        |        | on a      |        |            |           |
|    |        |        | scale of  |        |            |           |
|    |        |        | 1 to 10)  |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | P      | Q      | Percent   | 10% of | 8% of node |           |
| 07 | rocess | uality | of        | node   | hours at   |           |
|    | &      |        | delivered | hours  | each site  |           |
|    | Acti   |        | node      | at     | (BNL,      |           |
|    | vities |        | hours     | each   | FNAL,      |           |
|    |        |        | consumed  | site   | TJNAF)     |           |
|    |        |        | by jobs   | (BNL,  |            |           |
|    |        |        | with an   | FNAL,  |            |           |
|    |        |        | "error"   | TJNAF) |            |           |
|    |        |        | exit      |        |            |           |
|    |        |        | status.   |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | P      | Se     | Frequency | "N     | "Nessus"   |           |
| 07 | rocess | curity | of        | essus" | or         |           |
|    | &      | &      | vuln      | or     | equivalent |           |
|    | Acti   | P      | erability | equi   | deep scan  |           |
|    | vities | rivacy | scans on  | valent | performed  |           |
|    |        |        | nodes     | deep   | every      |           |
|    |        |        | visible   | scan   | month at   |           |
|    |        |        | from the  | per    | each site  |           |
|    |        |        | Internet  | formed | on all     |           |
|    |        |        | performed | every  | nodes      |           |
|    |        |        | at each   | other  | connected  |           |
|    |        |        | site.     | month  | directly   |           |
|    |        |        |           | at     | to the     |           |
|    |        |        |           | each   | public     |           |
|    |        |        |           | site   | internet.  |           |
|    |        |        |           | on all |            |           |
|    |        |        |           | nodes  |            |           |
|    |        |        |           | con    |            |           |
|    |        |        |           | nected |            |           |
|    |        |        |           | di     |            |           |
|    |        |        |           | rectly |            |           |
|    |        |        |           | to the |            |           |
|    |        |        |           | Int    |            |           |
|    |        |        |           | ernet. |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Tech   | Relia  | Meta      | A      | Average    |           |
| 07 | nology | bility | -facility | verage | machine    |           |
|    |        | and    | achieves  | m      | uptime of  |           |
|    |        | Availa | high      | achine | 92%        |           |
|    |        | bility | uptime    | uptime |            |           |
|    |        |        | and       | of 88% |            |           |
|    |        |        | e         |        |            |           |
|    |        |        | fficiency |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Tech   | E      | Aggregate | 8.6    | \+ 3.2     |           |
| 07 | nology | ffecti | resource  | TF\    | (new) -    |           |
|    |        | veness | sustained | This   | 0.2        |           |
|    |        |        | teraflops | capa   | (retired)  |           |
|    |        |        | for       | bility | = 11.6 TF\ |           |
|    |        |        | Asqtad +  | allows | This would |           |
|    |        |        | DWF (1:1) | the    | establish  |           |
|    |        |        |           | comp   | sufficient |           |
|    |        |        |           | letion | capability |           |
|    |        |        |           | of the | for the    |           |
|    |        |        |           | p      | planned    |           |
|    |        |        |           | hysics | 2008       |           |
|    |        |        |           | p      | physics    |           |
|    |        |        |           | rogram | program.   |           |
|    |        |        |           | p      |            |           |
|    |        |        |           | lanned |            |           |
|    |        |        |           | for    |            |           |
|    |        |        |           | 2007.  |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | M      | G      | C         | 7.0    | 9.5 TF-Yrs |           |
| 08 | ission | eneral | ompletion | TF-Yrs | applied to |           |
|    | and    | S      | of 2008   | a      | program    |           |
|    | Bu     | cience | S         | pplied | set out by |           |
|    | siness | and    | cientific | to     | Scientific |           |
|    | R      | Inno   | Program   | p      | Program    |           |
|    | esults | vation |           | rogram | Committee. |           |
|    |        |        |           | set    |            |           |
|    |        |        |           | out by |            |           |
|    |        |        |           | Scie   |            |           |
|    |        |        |           | ntific |            |           |
|    |        |        |           | P      |            |           |
|    |        |        |           | rogram |            |           |
|    |        |        |           | Comm   |            |           |
|    |        |        |           | ittee. |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | S      | Number of | 25     | 30         |           |
| 08 | stomer | ervice | users     |        |            |           |
|    | R      | Co     | running   |        |            |           |
|    | esults | verage | jobs each |        |            |           |
|    |        |        | month     |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | Time   | Response  | 90% of | 92% of     |           |
| 08 | stomer | liness | to        | t      | tickets    |           |
|    | R      | and    | service   | ickets | closed     |           |
|    | esults | Re     | requests  | closed | within 2   |           |
|    |        | sponsi |           | within | business   |           |
|    |        | veness |           | 2      | days       |           |
|    |        |        |           | bu     |            |           |
|    |        |        |           | siness |            |           |
|    |        |        |           | days   |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Cu     | Cu     | Customer  | A      | Average    |           |
| 08 | stomer | stomer | sat       | verage | score of   |           |
|    | R      | B      | isfaction | score  | 8.5        |           |
|    | esults | enefit | survey    | of 8.0 |            |           |
|    |        |        | (         |        |            |           |
|    |        |        | Customers |        |            |           |
|    |        |        | rate      |        |            |           |
|    |        |        | sat       |        |            |           |
|    |        |        | isfaction |        |            |           |
|    |        |        | with the  |        |            |           |
|    |        |        | service   |        |            |           |
|    |        |        | provided  |        |            |           |
|    |        |        | on a      |        |            |           |
|    |        |        | scale of  |        |            |           |
|    |        |        | 1 to 10)  |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | P      | Q      | Percent   | 8% of  | 6% of node |           |
| 08 | rocess | uality | of        | node   | hours at   |           |
|    | &      |        | delivered | hours  | each site  |           |
|    | Acti   |        | node      | at     | (BNL,      |           |
|    | vities |        | hours     | each   | FNAL,      |           |
|    |        |        | consumed  | site   | TJNAF)     |           |
|    |        |        | by jobs   | (BNL,  |            |           |
|    |        |        | with an   | FNAL,  |            |           |
|    |        |        | "error"   | TJNAF) |            |           |
|    |        |        | exit      |        |            |           |
|    |        |        | status.   |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | P      | Se     | Frequency | "N     | "Nessus"   |           |
| 08 | rocess | curity | of        | essus" | or         |           |
|    | &      | &      | vuln      | or     | equivalent |           |
|    | Acti   | P      | erability | equi   | deep scan  |           |
|    | vities | rivacy | scans on  | valent | performed  |           |
|    |        |        | nodes     | deep   | every two  |           |
|    |        |        | visible   | scan   | weeks at   |           |
|    |        |        | from the  | per    | each site  |           |
|    |        |        | Internet  | formed | on all     |           |
|    |        |        | performed | every  | nodes      |           |
|    |        |        | at each   | month  | connected  |           |
|    |        |        | site.     | at     | directly   |           |
|    |        |        |           | each   | to the     |           |
|    |        |        |           | site   | public     |           |
|    |        |        |           | on all | internet.  |           |
|    |        |        |           | nodes  |            |           |
|    |        |        |           | con    |            |           |
|    |        |        |           | nected |            |           |
|    |        |        |           | di     |            |           |
|    |        |        |           | rectly |            |           |
|    |        |        |           | to the |            |           |
|    |        |        |           | Int    |            |           |
|    |        |        |           | ernet. |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Tech   | Relia  | Meta      | A      | Average    |           |
| 08 | nology | bility | -facility | verage | machine    |           |
|    |        | and    | achieves  | m      | uptime of  |           |
|    |        | Availa | high      | achine | 93%        |           |
|    |        | bility | uptime    | uptime |            |           |
|    |        |        | and       | of 92% |            |           |
|    |        |        | e         |        |            |           |
|    |        |        | fficiency |        |            |           |
+----+--------+--------+-----------+--------+------------+-----------+
| 20 | Tech   | E      | Aggregate | 11.5   | \+ 4.1     |           |
| 08 | nology | ffecti | resource  | TF\    | (new) =    |           |
|    |        | veness | sustained | This   | 15.6 TF\   |           |
|    |        |        | teraflops | capa   | This would |           |
|    |        |        | for       | bility | establish  |           |
|    |        |        | Asqtad +  | allows | sufficient |           |
|    |        |        | DWF (1:1) | the    | capability |           |
|    |        |        |           | comp   | for the    |           |
|    |        |        |           | letion | planned    |           |
|    |        |        |           | of the | 2009       |           |
|    |        |        |           | p      | physics    |           |
|    |        |        |           | hysics | program.   |           |
|    |        |        |           | p      |            |           |
|    |        |        |           | rogram |            |           |
|    |        |        |           | p      |            |           |
|    |        |        |           | lanned |            |           |
|    |        |        |           | for    |            |           |
|    |        |        |           | 2008.  |            |           |
+----+--------+--------+-----------+--------+------------+-----------+

**Section E: Security and Privacy (IT Capital Assets only)**

For all investments, please respond to the questions below and verify
the system owner took the following actions:

  -------------------------------------------------------- ------- ------
  1\. Identified the IT security costs for the system(s)   Yes     
  and have integrated those costs into the overall costs           
  of the investment:                                               

  -------------------------------------------------------- ------- ------

> a\. If "yes," provide the "Percentage IT Security" for the budget
> year: (whole numbers plus 2 decimals)
>
> 3.32%

+--------------------------------------------------------+------+-----+
| 2\. Is identifying and assessing security and privacy  | Yes  |     |
| risks a part of the overall risk                       |      |     |
|                                                        |      |     |
| management effort for each system supporting or part   |      |     |
| of this investment.                                    |      |     |
+--------------------------------------------------------+------+-----+

3\. Systems in Planning - Security:

  ----------- ------------- ------------- --------------------------------
  Name of     Agency/ or    Planned       Planned or Actual C&A Completion
  System      Contractor    Operational   Date
              Operated      Date          
              System?                     

  JLab 7n     Contractor    June 30, 2007 July 1, 2005 \*

  FNAL 8      Contractor    June 30, 2008 October 2006

  FNAL 9      Contractor    June 30, 2009 October 2006
  ----------- ------------- ------------- --------------------------------

\* The LQCD systems acquired by this investment will be part of existing
cyber enclaves at their respective laboratories (BNL, FNAL, TJNAF). The
cyber enclaves at each of the three labs (BNL, FNAL, TJNAF) have been
C&A. The certification reviews were performed utilizing information and
checklists from NIST SP 800-18 (Security Plans), NIST SP 800-26 (Self
Assessments), NIST SP 800-37 (Certification and Accreditation), NIST SP
800-53 (Recommended Security Controls), FIPS 199, and published interim
OMB guidance to review the system security plan and to determine the
sensitivity, confidentiality, integrity, and availability levels for
this application. Based on this self-assessment certification
documentation, they received authority to operate (i.e. accreditation)
from senior management. The current C&As based on this process are:
FNAL\'s last C&A was issued in September, 2004, and the FNAL Authority
to Operate (ATO) is valid until October 2006. TJNAF\'s last C&A was
issued on July 1, 2005, and TJNAF has an ATO until July 2008. BNL
submitted its C&A package on February 21.

4\. Operational Systems -- Security:

+---------+------+------+-------+-------+----------+-------+-------+
| Name of | Spe  | NIST | Has   | Date  | What     | Date  | Date  |
| System  | cify | FIPS | C&A   | C&A   | s        | Co    | the   |
|         | whe  | 199  | been  | Comp  | tandards | mplet | C     |
|         | ther | Risk | Compl | leted | were     | e(d): | ontin |
|         | ag   | Im   | eted, |       | used for | Sec   | gency |
|         | ency | pact | using |       | the      | urity | Plan  |
|         | or   | l    | NIST  |       | Security | Co    | T     |
|         | co   | evel | 80    |       | Controls | ntrol | ested |
|         | ntra | (H   | 0-37? |       | tests?   | Te    |       |
|         | ctor | igh, | (Y/N) |       | (FIPS    | sting |       |
|         | oper | M    |       |       | 200/NIST |       |       |
|         | ated | oder |       |       | 800-53,  |       |       |
|         | sy   | ate, |       |       | NIST     |       |       |
|         | stem | Low) |       |       | 800-26,  |       |       |
|         |      |      |       |       | Other,   |       |       |
|         |      |      |       |       | N/A)     |       |       |
+---------+------+------+-------+-------+----------+-------+-------+
| BNL     | Co   | Low  | Y     | Feb,  | FIPS 200 | March | April |
| LQCD,   | ntra |      |       | 2006  |          | 23,   | 24,   |
| BNL     | ctor |      |       |       | NIST SP  | 2006  | 2006  |
| R       |      |      |       |       | 800-53   |       |       |
| esearch |      |      |       |       |          |       |       |
| Enclave |      |      |       |       |          |       |       |
+---------+------+------+-------+-------+----------+-------+-------+
| FNAL    | Co   | Low  | Y     | Sept, | FIPS 200 | April | Ja    |
| LQCD,   | ntra |      |       | 2004  |          | 25,   | nuary |
| General | ctor |      |       |       | NIST SP  | 2006  | 31,   |
| Co      |      |      |       |       | 800-53   |       | 2006  |
| mputing |      |      |       |       |          |       |       |
| Enclave |      |      |       |       |          |       |       |
+---------+------+------+-------+-------+----------+-------+-------+
| JLab    | Co   | Low  | Y     | July, | FIPS 200 | A     | April |
| LQCD,   | ntra |      |       | 2005  |          | pril, | 25,   |
| HPC/S   | ctor |      |       |       | NIST SP  | 2006  | 2006  |
| ci-Comp |      |      |       |       | 800-53   |       |       |
| Pr      |      |      |       |       |          |       |       |
| otected |      |      |       |       |          |       |       |
| Zone    |      |      |       |       |          |       |       |
+---------+------+------+-------+-------+----------+-------+-------+

  -------------------------------------------------------- ------- ------
  5\. Have any weaknesses related to any of the systems    Yes     
  part of or supporting this investment been identified by         
  the agency or IG?                                                

  -------------------------------------------------------- ------- ------

  -------------------------------------------------------- ------- ------
  a\. If "yes," have those weaknesses been incorporated    Yes     
  into the agency's plan of action and milestone process?          

  -------------------------------------------------------- ------- ------

BNL-LQCD shares a vulnerability with other BNL Research Enclave systems,
namely that it permits the use of static passwords for SSH access. BNL
is investigating correcting this by using single-use passwords instead.
The same situation exists at Jefferson Lab.

  -------------------------------------------------------- ------- ------
  6\. Indicate whether an increase in IT security funding          No
  is requested to remediate IT security weaknesses?                

  -------------------------------------------------------- ------- ------

a\. If "yes," specify the amount, a general description of the weakness,
and how the funding request will remediate the weakness.

7\. How are contractor security procedures monitored, verified, and
validated by the agency for the contractor systems above?

> The lattice gauge computing systems are managed at BNL, FNAL, and
> TJNAF, each of which is a government-owned, contractor-operated
> facility. Performance is monitored by the DOE site office at each
> laboratory, in accordance with the requirements specified in the
> contracts between the DOE and the respective contracting agencies
>
> (Brookhaven Science Associates (BSA) for BNL, Universities Research
> Association (URA) for FNAL, and Southeastern Universities Research
> Association (SURA) for TJNAF). These contracts include requirements
>
> for compliance with pertinent government and DOE Computer Security
> policies (e.g. DOE O 205.1 Department of Energy Cyber Security
> Management Program).
>
> At each laboratory, contractor security procedures are monitored,
> verified, and validated by numerous external entities including: 1)
> DOE-OCIO, 2) DOE Office of Performance Management and Oversight
> Assessment, 3) annual site reviews by the Office of Science, 4) the
> DOE-IG, and 5) external reviews.
>
> The dates of recent and planned monitoring, verification, and
> validation activities by the DOE at each of the host laboratories
> involved in this investment are:
>
> FNAL: November 2005 Site Assist visit; March/April 2006 DOE IG visit
> and audit
>
> BNL: IG visit scheduled for October 2006
>
> JLAB: DOE OA (Office of Independent Oversight and Performance
> Assurance) site visit May 2004. Additional OA visit planned for
> summer, 2006.
>
> 8\. Planning & Operational Systems -- Privacy:

  -------- ---------- ------------- ----------- ------------ ---------------
  Name of  Is this a  Is there a    Is the PIA  Is a System  Was a new or
  System   new        Privacy       available   of Records   amended SORN
           system?    Impact        to the      Notice       published in FY
           (Y/N)      Assessment    public?     (SORN)       06? (1,2,3,4,5)
                      (PIA) that    (1,2,3)     required for 
                      covers this               this system? 
                      system?                   (Y/N)        
                      (1,2,3,4,5)                            

  BNL:     N          3             2           N            5
  QCDOC                                                      

  FNAL:    N          3             2           N            5
  LQCD                                                       

  JLab:    N          3             2           N            5
  LQCD                                                       
  -------- ---------- ------------- ----------- ------------ ---------------

Section F: Enterprise Architecture (EA) (IT Capital Assets only)

  -------------------------------------------------------- ------- ------
  1\. Is this investment included in your agency's target  Yes     
  enterprise architecture?                                         

  -------------------------------------------------------- ------- ------

a\. If "no," please explain why?

  -------------------------------------------------------- ------- ------
  2\. Is this investment included in the agency's EA       Yes     
  Transition Strategy?                                             

  -------------------------------------------------------- ------- ------

> a\. If "yes," provide the investment name as identified in the
> Transition Strategy provided in the agency's most recent annual EA
> Assessment.
>
> SC Lattice Quantum ChromoDynamics Computing (LQCD)

b.  If "no," please explain why?

3\. Identify the service components funded by this major IT investment
(e.g., knowledge management, content management, customer relationship
management, etc.). Provide this information in the format of the
following table. For detailed guidance regarding components, please
refer to <http://www.whitehouse.gov/omb/egov/>.

  --------------- ---------------- ------------ ---------------- ----------- -------- ---------- ------------
  Agency          Agency Component FEA SRM      FEA SRM          FEA Service          Internal   Funding
  Component Name  Description      Service Type Component        Component            or         Percentage
                                                                 Reused               External   
                                                                                      Reuse?     

                                                                 Component   UPI                 
                                                                 Name                            

  Lattice QCD     The means to     Knowledge    Simulation                                       90%
  Simulation      perform lattice  Discovery                                                     
  Hardware        QCD simulations.                                                               
                  Dedicated                                                                      
                  computing                                                                      
                  hardware                                                                       
                  designed to                                                                    
                  execute lattice                                                                
                  QCD computer                                                                   
                  codes in the                                                                   
                  most cost                                                                      
                  effective                                                                      
                  manner. Supports                                                               
                  the LQCD                                                                       
                  hardware                                                                       
                  abstraction                                                                    
                  software                                                                       
                  libraries                                                                      
                  developed by the                                                               
                  SciDAC Lattice                                                                 
                  Gauge Computing                                                                
                  Project.                                                                       

  Lattice QCD     Organization and Document     Library /                                        2.5%
  Vacuum Gauge    archival storage Management   Storage                                          
  Configuration   (disk and tape)                                                                
  Archives        of vacuum gauge                                                                
                  configuration                                                                  
                  data generated                                                                 
                  on Lattice QCD                                                                 
                  Simulation                                                                     
                  Hardware.                                                                      

  Lattice QCD     Databases that   Data         Data                                             0.3%
  Metadata        relate the       Management   Classification                                   
  Catalogs        simulation                                                                     
                  parameters                                                                     
                  (quark masses,                                                                 
                  interaction                                                                    
                  constants,                                                                     
                  action, lattice                                                                
                  spacing, lattice                                                               
                  size) used to                                                                  
                  generate gauge                                                                 
                  configurations                                                                 
                  and quark                                                                      
                  propagators to                                                                 
                  data file series                                                               
                  stored in                                                                      
                  various                                                                        
                  archives.                                                                      

  Lattice QCD     Databases that   Knowledge    Information                                      2.5%
  Replica         relate lattice   Management   Retrieval                                        
  Catalogs        QCD data file                                                                  
                  series to                                                                      
                  physical storage                                                               
                  locations.                                                                     

  Lattice QCD     Strong           Knowledge    Information                                      0.4%
  Authenticated   authentication   Management   Sharing                                          
  Data Access     mechanisms                                                                     
                  permitting                                                                     
                  access to                                                                      
                  Lattice QCD data                                                               
                  and simulation                                                                 
                  hardware by                                                                    
                  authorized                                                                     
                  users.                                                                         

  Lattice QCD     Defined schema   Data         Data Exchange                                    0.3%
  Metadata        used to describe Management                                                    
  Specification   key parameters                                                                 
                  of Lattice QCD                                                                 
                  simulations.                                                                   

  Lattice QCD     Mechanisms to    Systems      Remote Systems                                   1.0%
  Hardware Remote enable remote    Management   Control                                          
  Management      management of                                                                  
                  Lattice QCD                                                                    
                  simulation                                                                     
                  hardware.                                                                      
                  Examples include                                                               
                  mechanisms for                                                                 
                  power cycling                                                                  
                  computer                                                                       
                  hardware,                                                                      
                  reloading                                                                      
                  operating                                                                      
                  systems, data,                                                                 
                  and firmware,                                                                  
                  and resetting                                                                  
                  computer and                                                                   
                  network                                                                        
                  hardware.                                                                      

  Lattice QCD     Mechanisms for   Systems      System Resource                                  1.0%
  Hardware        monitoring the   Management   Monitoring                                       
  Monitoring      status of                                                                      
                  Lattice QCD                                                                    
                  simulation                                                                     
                  hardware.                                                                      
                  Includes the                                                                   
                  gathering,                                                                     
                  storage,                                                                       
                  analysis, and                                                                  
                  presentation of                                                                
                  machine health                                                                 
                  and status                                                                     
                  information.                                                                   
  --------------- ---------------- ------------ ---------------- ----------- -------- ---------- ------------

4\. To demonstrate how this major IT investment aligns with the FEA
Technical Reference Model (TRM), please list the Service Areas,
Categories, Standards, and Service Specifications supporting this IT
investment.

  ---------------- ---------------- ------------------ ---------------- ---------------------------
  FEA SRM          FEA TRM Service  FEA TRM Service    FEA TRM Service  Service Specification
  Component        Area             Category           Standard         (i.e., vendor and product
                                                                        name)

  Library /        Service Platform Database / Storage Storage          Anacapa "XTORE" NAS
  Storage          and                                                  (network attached storage)
                   Infrastructure                                       

  Library /        Service Platform Database / Storage Storage          StorageTek Tape Silos
  Storage          and                                                  ("Powderhorn") and Tape
                   Infrastructure                                       Drives (T9940A, T9940B)

  Information      Service Access   Delivery Channels  Internet         ESNET
  Retrieval        and Delivery                                         

  Information      Service Access   Service Transport  Supporting       SSH (OpenSSH)
  Sharing          and Delivery                        Network Services 

  Information      Service Access   Service Transport  Supporting       Kerberos (MIT krb5)
  Sharing          and Delivery                        Network Services 

  Knowledge        Service Access   Service Transport  Service          SFTP (OpenSSH)
  Distribution and and Delivery                        Transport        
  Delivery                                                              

  Knowledge        Service Access   Service Transport  Service          SCP (OpenSSH)
  Distribution and and Delivery                        Transport        
  Delivery                                                              

  Knowledge        Service Access   Service Transport  Service          Hyper Text Transfer
  Distribution and and Delivery                        Transport        Protocol (HTTP)
  Delivery                                                              

  Knowledge        Service Platform Delivery Servers   Web Servers      Apache (www.apache.org)
  Distribution and and                                                  
  Delivery         Infrastructure                                       

  Simulation       Service Platform Hardware /         Servers /        Custom QCDOC Supercomputer
                   and              Infrastructure     Computers        
                   Infrastructure                                       

  Simulation       Service Platform Hardware /         Servers /        Intel Processor-based
                   and              Infrastructure     Computers        Clusters (Xeon, Pentium 4)
                   Infrastructure                                       

  Simulation       Service Platform Hardware /         Local Area       Ethernet
                   and              Infrastructure     Network (LAN)    
                   Infrastructure                                       

  Simulation       Service Platform Hardware /         Local Area       Myricom Myrinet 2000
                   and              Infrastructure     Network (LAN)    
                   Infrastructure                                       

  Simulation       Service Platform Hardware /         Local Area       Mellanox Infiniband
                   and              Infrastructure     Network (LAN)    Switches and Host Channel
                   Infrastructure                                       Adapters

  Simulation       Service Platform Supporting         Platform         Linux (Scientific Linux)
                   and              Platforms          Independent      
                   Infrastructure                                       

  Simulation       Service Platform Supporting         Platform         Custom QOS Operating System
                   and              Platforms          Dependent        
                   Infrastructure                                       

  Simulation       Component        Business Logic     Platform         C/C++ (GNU compilers, Intel
                   Framework                           Independent      C/C++ compilers, Portland
                                                                        Group C/C++ compilers)

  Simulation       Component        Business Logic     Platform         Perl
                   Framework                           Independent      

  Simulation       Component        Business Logic     Platform         Python
                   Framework                           Independent      

  Data Exchange    Service          Interoperability   Data Format /    XML (World Wide Web
                   Interface and                       Classification   Consortium, w3.org)
                   Integration                                          

  Data             Component        Data Interchange   Data Exchange    XQuery (World Wide Web
  Classification   Framework                                            Consortium, w3.org)

  Remote System    Component        Business Logic     Platform         IPMI (Intelligent Platform
  Control          Framework                           Independent      Management Interface)

  System Resource  Component        Presentation /     Dynamic          MRTG
  Monitoring       Framework        Interface          Server-Side      
                                                       Display          

  System Resource  Component        Presentation /     Dynamic          Ganglia
  Monitoring       Framework        Interface          Server-Side      (sourceforge.ganglia.net)
                                                       Display          
  ---------------- ---------------- ------------------ ---------------- ---------------------------

  ------------------------------------------------------- ------- -------
  5\. Will the application leverage existing components           No
  and/or applications across the Government (i.e.,                
  FirstGov, Pay.Gov, etc)?                                        

  ------------------------------------------------------- ------- -------

a\. If "yes," please describe.

+-------------------------------------------------------+------+------+
| 6\. Does this investment provide the public with      |      | No   |
| access to a government automated information system?  |      |      |
+-------------------------------------------------------+------+------+
| a\. If "yes," does customer access require specific   |      | No   |
| software (e.g., a specific                            |      |      |
|                                                       |      |      |
| web browser version)?                                 |      |      |
+-------------------------------------------------------+------+------+

> 1\. If "yes," provide the specific product name(s) and version
> number(s) of the required software and the date when the public will
> be able to access this investment by any software (i.e. to ensure
> equitable and timely access of government information and services).

**[Part II: Planning, Acquisition, and Performance]{.underline}**

Part II should be completed *only* for investments which in FY 2008 will
be in "Planning" or "Full Acquisition" or "Mixed Life-Cycle investments,
i.e., selected one of these three choices in response to Question 6 in
Part I, Section A above.

Section A. Alternatives Analysis \[All Capital Assets\]

  -------------------------------------------------------- ------- ------
  1\. Did you conduct an alternatives analysis for this    Yes     
  project?                                                         

  -------------------------------------------------------- ------- ------

a\. If "yes," provide the date the analysis was completed.

> Analysis is performed annually ahead of each computer acquisition. The
> most recent analysis was conducted in April, 2006 in preparation for
> the FY 2006 procurement. Preliminary analyses are performed in the
> third quarter of each fiscal year in planning for the subsequent
> fiscal year, and more detailed analyses are done ahead of each
> procurement. Subsequent detailed analyses will be held in Fall 2006
> (prior to FY2007 procurement) and Fall 2007 (prior to FY 2008
> procurement).

b\. If "no," what is the anticipated date this analysis will be
completed.

c\. If no analysis is planned, please briefly explain why not.

2\. Use the results of your alternates analysis to complete the
following table:

  ----------------------------------------------------------------------------
  Alternatives   Description                    Lifecycle   Lifecycle Benefits
                                                Cost        Estimate
                                                Estimate    
  -------------- ------------------------------ ----------- ------------------
  Baseline       Status Quo (4.2 teraflop/s     \$2.2M      \$3.6M
                 sustained performance cluster              
                 in FY2008, 3.5 year operating              
                 lifetime)                                  

  Alternative 1  Expand supercomputer center    \$2.9M      \$3.6M
                 large clusters to have                     
                 additional \~4 teraflops                   
                 systained capacity                         

  Alternative 2  Procure IBM BlueGene/P         \$2.8M      \$3.6M
                 supercomputer, two racks                   
                 (estimate 4x boost over                    
                 BlueGene/L in performance at               
                 constant cost per rack)                    

  Alternative 3  Expand supercomputer center    \$3.1M      \$3.6M
                 BlueGene/P installations to                
                 provide 4.2 teraflop/s beyond              
                 what would otherwise have been             
                 used                                       
  ----------------------------------------------------------------------------

3\. Which alternative was selected by the Agency's Executive/Investment
Committee and why was it chosen?

The project Baseline alternative was selected because it is the lowest
cost alternative which achieves agency goals.

4\. What specific qualitative benefits will be realized?

This investment will provide quantitative benefits by directly impacting
the High Energy Physics (HEP) and Nuclear Physics (NP) experimental
physics programs. In 2006, 60% of the \$717M HEP program and 50% of the
\$367M NP program for a total of \$614M are impacted by these LQCD
calculations.\
\
There are three significant ways in which this project will yield a
return on investment: (1) reduced running time for given science goals,
(2) improved selection of experiments, (3) new discoveries.\
\
The scientific output of roughly 10% of these HEP and NP experiments is
influenced by the accuracy of the theoretical numbers that go into the
analysis of the experimental results. The proposed computing
capabilities (including those coming online in FY2007, and those being
added in FY2008) will improve the accuracy of these numbers by anywhere
from 10% to a factor of 1.5, depending upon the physical quantity. These
improvements will then improve the accuracy of the experimental results
by an amount roughly equivalent to increasing the running time of the
experiments by 5% (experiment dependent). In other words, this
investment increases the output of the \$614M investment in these
experiments by 5% of 10%, or 0.5%, a \$3.1M benefit. This benefit was
partially realized with the installation of the QCDOC in the fall of
2005. It will be fully realized beginning in FY 2007 and will continue
for the life of the project, and for an additional year and a half
beyond, albeit at a reduced rate as less new capacity is procured in the
final year of the project, and since machines will be de-commissioned.\
\
One of the LQCD investment impacts comes from the process used to select
the experiments in the HEP and NP experimental programs. In this process
individual experiments are proposed to Program Advisory Committees for
inclusion in the program. One of the key criteria for an experiment to
be approved is that it has an adequate theoretical foundation. Proposed
experiments that have justifications based on a solid theoretical
analysis have a significantly better chance of being approved. As a
result of expanding possible theoretical analysis to include the LQCD
calculations from this investment, better experiments will be performed,
optimizing the scientific output of investments made in the experimental
programs. Potential savings are expected to accrue from scientific
discoveries, which influence the design or selection of future major
facilities, thereby potentially significantly enhancing the scientific
output of very large investments. The dollar value of these impacts are
more difficult to assess. One approach is to say that the impact is most
significant for the lowest 10% of experiments ranked by priority or
scientific value. Improved selection of experiments might increase the
scientific output of these 10% of experiments by 5%, yielding an
additional \$3.1M benefit.

## Section B. Risk Management (All Capital Assets) 

All investments should perform a risk assessment during the early
planning and initial concept phase of the investment life-cycle.

  -------------------------------------------------------- ------- ------
  1\. Does the investment have a Risk Management Plan?     Yes     

  -------------------------------------------------------- ------- ------

a\. If "yes," what is the date of the plan?

Updated version, April 6, 2006

> b\. Has the Risk Management plan been significantly changed since last
> year's submission to OMB?
>
> No

c\. If there currently is no plan, will a plan be developed?

  -------------------------------------------------------- ------- ------
  2\. If there currently is no plan, will a plan be                
  developed                                                        

  -------------------------------------------------------- ------- ------

a\. If "yes," what is the planned completion date?

b\. If "no," what is the strategy for managing the risks?

3\. Briefly describe how investment risks are reflected in the life
cycle cost estimate and investment schedule.

> The investment plan is based upon a solid trend in industry in which
> computers gain in performance by 60% each year for fixed cost. There
> are minor fluctuations in this trend line, and over many years this is
> a conservative estimate for optimized clusters for Lattice QCD. There
> is a minor risk that this trend line could be missed slightly in any
> particular year, requiring a minor adjustment in milestones for that
> year (either in performance or time). There is no risk to project
> cost, as contracts are firm fixed price and include hardware
> warrantees.

## Section C: Cost and Schedule Performance (All Capital Assets)

Note: The DOE has determined that this investment does not meet the
criteria requiring EVM.

  -------------------------------------------------------- ------- ------
  1\. Does the earned value management system meet the             
  criteria in ANSI/EIA Standard 748?                               

  -------------------------------------------------------- ------- ------

N/A

  -----------------------------------------------------------------------
  2\. Answer the following questions about current cumulative cost and
  schedule performance. Indicate whether the information provided is
  contractor-only, or whether it includes both government and contractor
  costs.

  -----------------------------------------------------------------------

N/A

"As of" date: (mm/dd/yy)

a\. What is the Planned Value (PV)?

b\. What is the Earned Value (EV)?

c\. What is the actual cost of work performed (AC)?

d\. What is the calculated Schedule Performance Index (SPI=EV/PV)?

4\. What is the Schedule Variance (SV=EV-PV)?

N/A

5\. What is the calculated Cost Performance Index (CPI=EV/AC)?

N/A

6\. What is the Cost Variance (CV = EV - AC)?

N/A

  -------------------------------------------------------- ------- ------
  7\. Is the CV or SV greater than 10%?                            

  -------------------------------------------------------- ------- ------

N/A

a\. If "yes," was it the CV, SV, or both?

b\. If "yes," explain the variance

c\. If "yes," what corrective actions are being taken?

d\. What is the most current "Estimate at Completion"?

  -------------------------------------------------------- ------- ------
  8\. Is the department requesting a change in the         Yes     
  performance baseline?                                            

  -------------------------------------------------------- ------- ------

Based on operations during FY2006, additional manpower beyond that
proposed in the current approved baseline will be required to
competently operate the LQCD computing systems; note that the size of
the operated systems increases in each year of the investment, and that
the amount of SS spending in the current baseline increases in each year
of the project, with a corresponding decrease in the amount of D/M/E
spending. The requested change to the baseline shifts funding from the
D/M/E portion to the SS portion to support this increased manpower
requirement. The total project cost remains the same; however, the
quantities of deployed TFlops in FY07-FY09 and of delivered TFlops in
FY07-FY09 will likely decrease. These proposed changes will be presented
at the May 25/26, 2006, annual LQCD Technical Progress Review held by
the DOE. Afterwards the deployed and delivered Tflops numbers in the
table below will be adjusted.

9\. Complete the following table to compare actual cost performance
against the planned cost performance baseline. Milestones reported may
include specific individual scheduled preventative and predictable
corrective maintenance activities, or may be the total of planned annual
operation and maintenance activities. Indicate if the information
provided includes government and contractor costs.

a\. Government costs? No

b\. Contractor costs? Yes

  ----------------- -------------- ----------- ---------------- ----- ---------------- ---- --------------- ---- ----------
                    Comparison of                                                                                
                    Initial                                                                                      
                    Baseline and                                                                                 
                    Current                                                                                      
                    Approved                                                                                     
                    Baseline                                                                                     

                    Initial                    Current Baseline                             Current              
                    Baseline                                                                Baseline             
                                                                                            Variance             

  Description of    Planned        Total Cost  Completion Date        Total Cost (\$M)      Schedule/Cost        Percent
  Milestone         Completion     (\$M)       (mm/dd/yyyy)           Planned/Actual        (# days/\$M)         Complete
                    Date           Estimated   Planned/Actual                                                    
                    (mm/dd/yyyy)                                                                                 

  Procurement and   09/30/2006     \$1.850     09/30/2006             \$1.850                                    
  deployment of                                                                                                  
  systems totaling                                                                                               
  2.0 teraflops                                                                                                  

  6.2               09/30/2006     \$0.627     09/30/2006             \$0.627                                    
  Teraflops-years                                                                                                
  computing                                                                                                      
  delivered                                                                                                      

  Procurement and   06/30/2007     \$1.694     06/30/2007             \$1.616                                    
  deployment of                                                                                                  
  systems totaling                                                                                               
  3.1 teraflops                                                                                                  

  Additional 9      09/30/2007     \$0.782     09/30/2007             \$0.860                                    
  Teraflops-years                                                                                                
  computing                                                                                                      
  delivered                                                                                                      

  Procurement and   09/30/2008     \$1.598     06/30/2008             \$1.543                                    
  deployment of                                                                                                  
  systems totaling                                                                                               
  4.2 teraflops                                                                                                  

  Additional 12     09/30/2008     \$0.877     09/30/2008             \$0.932                                    
  Teraflops-years                                                                                                
  computing                                                                                                      
  delivered                                                                                                      

  Procurement and   09/30/2009     \$0.798     09/30/2009             \$0.668                                    
  deployment of                                                                                                  
  systems totaling                                                                                               
  3.0 teraflops                                                                                                  

  Additional 15     09/30/2009     \$0.902     09/30/2009             \$1.032                                    
  Teraflops-years                                                                                                
  computing                                                                                                      
  delivered                                                                                                      
  ----------------- -------------- ----------- ---------------- ----- ---------------- ---- --------------- ---- ----------