**"Lessons Learned from the ASAP Centers" Panel:**

The purpose of this session is to review some of the lessons we have
learned in the ASAP program that are useful to keep in mind. Some of the
lessons may seem obvious (and the panel will add concrete examples) and
some of the lessons took several years to learn.

-   Highlights included:

***Management and Leadership***

-   Creating a truly multidisciplinary Center in a university
    environment is often perceived as an "unnatural act". But,
    organizing a multidisciplinary Center in "stovepipes" will not work;
    the organizational emphasis must be on integration (of codes, theory
    [and]{.underline} experiment, and for PSAAP even more on V&V and
    UQ).

-   Defining a single, overarching problem is needed for conducting
    research across different university departments because managers
    can ask what has been done lately for the overarching problem.

-   Central resource allocation (versus distributing funds to individual
    faculty accounts) keeps attention focused on solving the overarching
    problem for the center. (The Golden Rule strongly applies.)

-   Open communication is essential. It is important to include all
    project personnel in the communication chain.

-   Having well defined goals and clear responsibility for their
    leadership is very important.

***Staffing***

-   There can be an inherent disconnect between university scientists
    who are theoretically oriented and the Center's need to develop
    full-system simulation code. This must be overcome if the Center is
    to succeed; annual strategic planning involving all key Center
    personnel is one way to address these different perspectives.

-   Be wary of involving "loner" faculty members (however brilliant they
    may be) because the ultimate success of the Center will depend on
    the success of the entire personnel team.

-   Post docs and grad students do not typically work on the time scale
    of a five-year or ten- year program. Therefore, it is important to
    have long term staffing so they can train the incoming students and
    post docs. This can be a difficult balancing act.

-   Over time some faculty, and/or subcontractors, may make less than
    expected contributions to the overarching problem. As a result,
    tough decisions will have to be made by the Center Director.

-   Full time staff such as Post docs and Research Assoc can be half or
    more of a Center's staff for the type of work required by ASAP and
    PSAAP.

***Balancing Science and Code Building***

-   Use V&V as a way to balance science and code building; V&V will help
    to identify what is critical to full-scale problem simulation.

-   A roadmap for the overarching system identifying end-to-end
    simulations that the center will run over the next six to twelve
    months is one key way to focus on integration.

-   Science and code development should take place under the same roof.
    The difference between PSAAP and other university centers is that a
    code has to be delivered and science has to be brought into the code
    and brought in according to the software engineering framework of
    the Center.

-   

***Integration***

-   It is important to put the full system (problem) put together (even
    with less details initially) so that Center personnel wrestle early
    on with the science of integration. Integration goals are critical
    from day one.

-   Weekly meetings of key Center personnel help to keep a focus on the
    next planned simulation and can serve as a catalyst for code and
    model integration.

-   Allowing scientists to build special "research codes" is OK (to a
    point) because it builds in a level of science that can be used as a
    check against the overarching application. But Center personnel
    should understand - from the outset - how such side efforts might be
    (or not) folded into the overarching application.

-   Algorithms are important when coupling diverse codes.

***Software Engineering***

-   The requirement for software quality assessment and engineering
    practices is non-negotiable.

-   Establishing someone on the Center staff as the code architect is
    highly recommended.

-   It is important to have a good software development process in place
    (and make it clear to those who want to work outside of it that it
    is not in their best interest).

***Other Areas***

-   Storage of Center calculation/simulation data is a critical issue
    (and often a surprise when calculations produce terabytes of data).

-   Because the Center will (hopefully) exist for five years the success
    of graduate students' work should not be tied to Center deliverables
    (but can be associated with Center research-oriented tasks that are
    not on the critical path).

-   Association with ASC Alliance Center programs led to many successful
    tenure track advancements for junior faculty during the ASAP phase
    of the program.

-   It can be useful to figure out who is doing work at the DP labs
    comparable to the Center research and communicate via workshops and
    visits.

-   It is important to see the interaction of the experimentalist with
    simulations -- this produces insights for the simulations and also
    for the experimentalists. (It also creates a "super" grad student
    who understands computation, software engineering, algorithms and
    the importance of experiments).

-   Very difficult to perform V&V with historical data.

-   Ties to outside communities for V&V can attract significant funding
    opportunities to extend core program.

-   Annual reviews are a lot of work but are invaluable because they
    lead to self-examination and improved planning.