**SAND2003-1899**

**Unlimited Release**

**Updated April 2008**

**Trilinos Developers Guide**

**Part II: ASC Software Quality Engineering Practices**

**Version 2.0**

# Michael A. Heroux, James M. Willenbring, Robert T. Heaphy

Scalable Algorithms Department

Sandia National Laboratories

P.O. Box 5800

Albuquerque, New Mexico 87185

# Abstract

The Trilinos Project is an effort to facilitate the design, development,
integration and ongoing support of numerical software libraries,
primarily focused on solvers. A new software capability is introduced
into Trilinos as a *package*. A Trilinos package is an integral unit
and, although there are exceptions such as utility packages, each
package is typically developed by a small team of experts in a
particular algorithms area such as algebraic preconditioners, nonlinear
solvers, etc.

The Trilinos Developers Guide Part II is a resource for Trilinos package
developers who are working under Advanced Simulation and Computing (ASC)
and are therefore subject to the ASC Software Quality Engineering
Practices as described in the Sandia National Laboratories Advanced
Simulation and Computing (ASC) Software Quality Plan Part 1: ASC
Software Quality Engineering Practices Version 1.0 document \[1\]. The
Trilinos Developers Guide \[2\] is a companion document to this second
part and contains much of the detailed information that is essential for
all Trilinos developers. The Trilinos Software Lifecycle Model \[3\]
defines the default lifecycle model for Trilinos packages and provides a
context for many of the practices listed in this document.

##  

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## Intentionally Left Blank 

## 1. Introduction

One objective of Advanced Simulation and Computing (ASC) is to develop
professional software quality engineering (SQE) practices that will
ensure the quality of software developed with ASC funding. To this end,
the Sandia National Laboratories Advanced Simulation and Computing (ASC)
Software Quality Plan Part 1: ASC Software Quality Engineering Practices
Version 1.0 document \[1\], lists 30 practices that should be addressed
by ASC software developers. Part II of the Trilinos Developers Guide
addresses each of the 30 practices, often referring to the main Trilinos
Developers Guide \[2\] except for issues that are unique to ASCI SQE
practices. Although all Trilinos developers are encouraged to adopt
these ASC practices, developers are only required to do so when working
on ASC funded capabilities. Each of the 30 practices is discussed in
terms of the responsibilities of the Trilinos Framework and each
individual Trilinos Package. As can be seen from the table in Section 3,
the Trilinos Framework provides a valuable service to the Trilinos
Packages, providing package developers with ready-made support for many
of the 30 practices, leaving to package developers only those practices
that should be under the direction of each package team. In fact, eleven
of the practices are the sole responsibility of the framework and the
framework provides significant support for the remaining nineteen
practices.

## 

## 2. Roles, Documents, Tools and Events

The primary content of this document is a large table in Section 3,
listing and discussing each of the 30 ASC SQE practices. Throughout the
discussion, a number of roles (people), documents, tools and events are
cited frequently. We list each of them here and assign acronyms where
appropriate:

**Roles:**

-   **ASC Program Management:** The ASC program (which includes ASC
    Algorithms) has an evolving set of processes for SQE. Many of the
    SQE processes in the Project Planning, Tracking, and Oversight, Risk
    Management, and Determination of Applicable Practices and level of
    Formality phases are driven by decisions made by ASC Program
    Management.

-   **ASC Algorithms Program Element Lead and PI:** The Trilinos Project
    receives a significant portion of its funding from the Algorithms
    portion of the ASC Program. As a result, the ASC Algorithms Program
    Element Lead and PI play an important role in the Project Planning,
    Tracking, and Oversight, Risk Management, and Determination of
    Applicable Practices and level of Formality, following the
    guidelines and requirements established by ASC Program Management.

-   **Trilinos Project Leader:** A fundamental management principal of
    the Trilinos Project is that packages should be as autonomous as
    possible, recognizing that local control with careful attention to
    interfaces is often the most effective approach to producing high
    quality software. At the same time there is a need for a single
    focal point in some situations. The Trilinos Project leader is the
    primary focal point for major project decisions. The duties of this
    person include:

1.  Arbitrating in cases where a consensus decision cannot be reached as
    part of inter-package decisions.

2.  Organizing Trilinos project events (see below).

3.  Managing and tracking progress on the Trilinos Framework
    Responsibilities as described in Section 3.

-   **Trilinos Release Manager:** The Trilinos Release Manager is
    responsible for tagging and branching the repository and generally
    coordinating the release process.

-   **Trilinos Capability Leaders:** Because of the broad scope of
    Trilinos, we have leaders assigned to the following capability
    areas:

1.  Framework, Tools & Interfaces.

2.  Discretizations.

3.  Geometry, Meshing & Load Balancing.

4.  Scalable Linear Algebra.

5.  Linear & Eigen Solvers.

6.  Nonlinear, Transient & Optimization Solvers.

> These leaders have responsibilities across packages and are
> responsible for strategic planning for Trilinos in their areas.

-   **Package Leader:** Each Trilinos package has one or more leaders.
    These leaders are responsible for managing and tracking package
    responsibilities as described in Section 3. These leaders are also
    responsible for attending the Monthly Trilinos Leaders Meetings,
    representing the package development team and disseminating
    information to the team. Only leaders of packages funded by ASC are
    responsible for all of the practices listed in Section 3.

-   **Package Developer:** Each package has an identifiable group of
    developers. Any given individual may be a member of multiple package
    development teams.

-   **Framework Developer:** The Trilinos Framework also has an
    identifiable group of developers. These individuals may also be
    members of package development teams.

**Documents:**

-   **Trilinos Developer Guide (TDG):** Primary development guide for
    Trilinos developers. Discusses software requirements that apply to
    all Trilinos developers and presents the suggested practices for
    each requirement. This guide also describes the services available
    to packages that are part of Trilinos \[2\].

-   **Trilinos Software Lifecycle Model:** Defines a 3-phase promotional
    lifecycle model that recognizes the changing requirements for
    Trilinos capabilities as work goes from proof-of-concept to
    production quality. The three phases are research, production
    growth, and production maintenance. To transition from one phase to
    the next, a promotional event must be completed. Most often,
    packages transition from phase to phase as a whole, but the model
    also allows for different capabilities within packages to be in
    different phases, provided certain conditions are met \[3\].

-   **Trilinos Strategic Plan (TSP):** Lists the strategic goals of the
    project and provides pointers to other important information, such
    as a list of ASC stakeholders and project capabilities \[4\].

-   **Trilinos Project Plan (TPP):** Each fiscal year the ASC Algorithms
    team gathers user and software requirements from ASC application
    teams and its own members. The exact form and number of documents
    generated has changed from year to year, as the ASC program itself
    defines its processes for software quality. Currently the TPP is
    called the *Objectives and Resource plan for ASC Algorithms R&D*
    \[5\]. In past years we have provided all analysis and documentation
    request by ASC program managers, and have developed additional
    documents that provide greater detail than what ASC requires. As ASC
    program practices evolve, we will continue to adapt our documents
    and processes to match, especially in the Requirements, Project
    Planning, Tracking and Oversight and Risk Management phases.
    Although the type and number of documents required by the ASC
    program has and will continue to change, for simplicity we refer to
    entire collection of these documents and related documents that
    provide greater detail as the Trilinos Project Plan. In the recent
    past, the primary documents have been the ASC Algorithms
    Implementation Plan and the ASC Algorithms Objectives and Resource
    Plan.

-   **ASC Algorithms Quarterly Report (AAQR):** At the end of each
    quarter, when requested by ASC program management, the ASC
    Algorithms team will generate a progress report and list adjusted
    milestones as needed.

-   **Package Specific Documentation (PSD):** In the Package
    Responsibilities column we discuss in detail how a package should
    satisfy a particular practice. However, it is always the case that a
    package may satisfy any given practice via its own process as long
    as the alternate process is documented in the appropriate Package
    Specific Document. For example, if Package X adopts a lifecycle that
    is different than The Trilinos Software Lifecycle Model, the
    alternative lifecycle model should be documented.

**Tools:**

-   **Concurrent Versions Systems (CVS):** All Trilinos source code and
    documents are maintained using CVS \[6\]. The primary CVS repository
    resides on the Trilinos Development platform "software.sandia.gov"
    on the Sandia Open Network (SON). Sensitive documents are retained
    under a separate repository on the Sandia Restricted Network (SRN).
    Use of CVS is documented in the TDG.

-   **Bugzilla:** All major features and software faults are reported
    using Bugzilla \[7\], a web-based issue-tracking package. Each
    Trilinos package, the Trilinos framework and Bugzilla itself are set
    up as Bugzilla products. Bugzilla is available at
    <http://software.sandia.gov/bugzilla>. Use of Bugzilla is documented
    in the TDG. Note that the processes for reporting bugs, tracking
    issues, and requesting enhancements are built directly into
    Bugzilla.

-   **Mailman:** Each Trilinos package has a set of Mailman \[8\] mail
    lists to support communication and archiving of important
    information and artifacts. List descriptions are documented in the
    TDG.

-   **Doxygen:** Many Trilinos packages use the documentation-generating
    package called Doxygen \[9\]. Doxygen processes source code comments
    producing detailed online and printed documentation of the processed
    source. Although it can be used with many languages and is highly
    configurable, the most common use of it within Trilinos is to
    process header files containing description of C++ classes and
    detailed documentation of the major user-callable methods in each
    class. Doxygen also extracts information about class interactions
    and dependencies. Most Trilinos packages presently use Doxygen to
    provide user reference documentation.

-   **Autoconf and Automake:** Trilinos configuration and building is
    facilitated by using Autoconf \[10\] and Automake \[11\],
    collectively referred to as Autotools \[12\]. These tools facilitate
    dynamic configuration and building of Trilinos across a broad set of
    computer platforms. Via runtime compilation and linking tests,
    queries to the operating system and user-specified parameters,
    Trilinos can be configured and built on almost any platform with
    minimal user knowledge of details such as location of system
    libraries and compilers. The autotools build system also supports
    installation of Trilinos for multiple users and automatic creation
    of distribution tar files.

-   **software.sandia.gov:** The primary Trilinos development platform
    is a Linux server called software.sandia.gov. This platform supports
    all of the tools listed above, contains the Trilinos CVS repository
    and all of the Mailman and Bugzilla archives. This platform is on
    the Sandia Open Network (SON), so it is accessible from any
    internet-connect machine. The file systems on this platform are
    backed up daily. Backup tapes are shipped offsite monthly.

-   **trilinos.sandia.gov:** The primary user-oriented website,
    containing most user documentation and download instructions.

-   **Microsoft Project:** The Trilinos project leader uses Microsoft
    Project \[13\], a project management and tracking tool, to track and
    communicate major Trilinos deliverables.

-   **Trilinos Process Checklists:** The Trilinos project team has a
    number of processes defined via process checklists. A list of
    current and past process checklists is available on the Trilinos
    developer website \[14\]. Completed checklists related to releases
    and CVS commits are stored in Bugzilla, either in the body of the
    bug, or as an attachment. Once a process is started, it can be
    completed using the version of the process checklist that was
    current at the time the process was initiated, unless otherwise
    requested by the Trilinos Project Leader \[14\].

-   **Trilinos Test Harness:** A suite of utilities that combine to
    automatically configure, compile, and test Trilinos packages on a
    range of important/interesting test machines. The results are then
    reported to a central database where they can be viewed online
    \[15\].

**Events:**

-   **Monthly Trilinos Leaders Meeting:** Package leaders for Trilinos
    packages, Trilinos management and other stakeholders participate in
    a monthly leadership meeting. Meeting minutes are sent to the
    <Trilinos-Leaders@software.sandia.gov> mail list for communication
    and archiving. Meeting topics include discussion of requirements,
    design, implementation, testing and documentation. We also conduct
    developer training as needed during these meetings. A request for
    agenda items and a meeting agenda is sent to the Trilinos-Leaders
    mail list prior to each meeting.

-   **Quarterly Trilinos Advisors Meeting:** Many Trilinos customers and
    a subset of the Trilinos Leaders group participate in a quarterly
    Trilinos Advisory Group meeting. These meetings give Trilinos
    developers the change to pass important information on to users and
    give users the opportunity to directly share concerns and
    suggestions with developers. Customer training can also take place
    at these meetings. Meeting minutes are sent to the
    <Trilinos-Advisors@software.sandia.gov> mail list for communication
    and archiving. A request for agenda items and a meeting agenda is
    sent to the Trilinos-Advisors mail list prior to each meeting.

-   **Quarterly Trilinos Capability Leaders Meeting:** The Trilinos
    Capability Leaders meet quarterly to discuss high level Trilinos
    issues including the general direction of the project. Meeting
    minutes are sent to the <Trilinos-Board@software.sandia.gov> mail
    list for communication and archiving. A request for agenda items and
    a meeting agenda is sent to the Trilinos-Board mail list prior to
    each meeting.

-   **Annual Trilinos User Group Meeting:** Approximately once a year we
    hold a meeting for Trilinos users. During this meeting we present an
    overview of Trilinos, and detailed presentations of Trilinos
    packages. The fourth annual meeting was held in November, 2006.

## 3. Trilinos Practices Table

The remainder of this document is a table that follows, item by item,
the 30 practices listed in the ASC Software Quality Engineering
Practices document. For each practice, the responsibilities of the
Trilinos Framework and each Trilinos Package are described using
present-tense phrasing. Please note that some of these responsibilities
are not fully addressed at this time, in which case this document serves
as a plan rather than a statement of practice.

+----------------+-------------------------+---------------------------+
| **Practice**   | **Trilinos Framework    | **Trilinos Package        |
|                | Responsibilities**      | Responsibilities**        |
+================+=========================+===========================+
| **Project      |                         |                           |
| Management**   |                         |                           |
+----------------+-------------------------+---------------------------+
| **1. Strategic |                         |                           |
| Planning**     |                         |                           |
+----------------+-------------------------+---------------------------+
| PR1. Document  | The Trilinos Strategic  | Provide input for the     |
| and maintain a | Plan \[4\] is           | Trilinos Strategic Plan   |
| strategic      | maintained at the       | as appropriate.           |
| plan.          | Trilinos project level  |                           |
|                | with input from         |                           |
|                | packages.               |                           |
+----------------+-------------------------+---------------------------+
| **2.           |                         |                           |
| Determination  |                         |                           |
| of Applicable  |                         |                           |
| Practices and  |                         |                           |
| Level of       |                         |                           |
| Formality**    |                         |                           |
+----------------+-------------------------+---------------------------+
| PR2. Perform a | Trilinos project level  | For packages in the       |
| risk-based     | risk identification and | Research phase the        |
| assessment,    | mitigation is included  | associated technical risk |
| determine      | in the TPP and AAQR,    | cannot be mitigated by a  |
| level of       | when and as requested   | high level of formality,  |
| formality and  | by ASC program          | so by default these       |
| applicable     | management. An ASC      | packages follow a low     |
| practices, and | risk-based assessment   | level of formality (LOF). |
| obtain         | has been completed and  | During the promotional    |
| approvals.     | approved.               | events defined by The     |
|                |                         | Trilinos Software         |
|                |                         | Lifecycle Model, risk     |
|                |                         | identification is         |
|                |                         | required. By default, the |
|                |                         | LOF at the Production     |
|                |                         | Growth phase is medium.   |
|                |                         | At the Production         |
|                |                         | Maintenance phase, the    |
|                |                         | default LOF is high.      |
+----------------+-------------------------+---------------------------+
| **3. Process   |                         |                           |
| Implementation |                         |                           |
| and            |                         |                           |
| Improvement**  |                         |                           |
+----------------+-------------------------+---------------------------+
| PR3. Document  | The Trilinos Software   | None.                     |
| lifecycle      | Lifecycle Model defines |                           |
| processes and  | the lifecycle for a     |                           |
| their          | Trilinos package.       |                           |
| inte           |                         |                           |
| rdependencies, |                         |                           |
| and obtain     |                         |                           |
| approvals.     |                         |                           |
+----------------+-------------------------+---------------------------+
| PR4. Define,   | Trilinos process        | Package level process     |
| collect, and   | checklists \[14\]       | checklists require that   |
| monitor        | require and recommend   | certain metrics are       |
| appropriate    | the collection of a     | gathered by package       |
| process        | number of project       | development teams in the  |
| metrics.       | metrics. Other metrics, | Production Growth and     |
|                | such as build and test  | Production Maintenance    |
|                | failures and coverage   | phases.                   |
|                | rates are automatically |                           |
|                | collected using the     |                           |
|                | Trilinos Test Harness   |                           |
|                | and its web interface   |                           |
|                | \[15\]. Certain         |                           |
|                | metrics, such as build  |                           |
|                | failures, are monitored |                           |
|                | frequently. There is an |                           |
|                | ongoing effort to       |                           |
|                | define new metrics for  |                           |
|                | the project.            |                           |
+----------------+-------------------------+---------------------------+
| PR5.           | Many process checklists | Package level process     |
| Periodically   | use Plan, Do, Check,    | checklists utilize        |
| evaluate       | Act, which provides     | process improvement in    |
| quality        | built in process        | the same way as Framework |
| problems and   | improvement. Other      | level process checklists. |
| implement      | checklists are reviewed | Metrics are made          |
| process        | periodically and        | available to package      |
| improvements.  | improved upon.          | development teams to      |
|                |                         | allow them to make        |
|                |                         | package level process     |
|                |                         | improvements.             |
+----------------+-------------------------+---------------------------+
| **4.           |                         |                           |
| Requirements   |                         |                           |
| Engineering**  |                         |                           |
+----------------+-------------------------+---------------------------+
| PR6. Identify  | The Trilinos Strategic  | None.                     |
| stakeholders   | Plan \[4\] identifies   |                           |
| and other      | Trilinos stakeholders.  |                           |
| requirements   |                         |                           |
| sources.       |                         |                           |
+----------------+-------------------------+---------------------------+
| PR7. Gather    | Primary user            | None.                     |
| and manage     | requirements are        |                           |
| stakeholders'  | gathered and documented |                           |
| expectations   | in the TPP. In          |                           |
| and            | addition, topics        |                           |
| requirements.  | discussed during the    |                           |
|                | monthly Trilinos        |                           |
|                | Leader's meeting        |                           |
|                | include Trilinos user   |                           |
|                | requirements. Meeting   |                           |
|                | minutes are archived on |                           |
|                | the                     |                           |
|                | <trilinos-leade         |                           |
|                | rs@software.sandia.gov> |                           |
|                | mail list.              |                           |
+----------------+-------------------------+---------------------------+
| PR8. Derive,   | The TPP covers the      | None.                     |
| negotiate,     | derivation,             |                           |
| manage, and    | negotiation,            |                           |
| trace          | management, and tracing |                           |
| requirements.  | of requirements. The    |                           |
|                | AAQR documents the      |                           |
|                | managing and tracing of |                           |
|                | and the success in      |                           |
|                | meeting requirements.   |                           |
+----------------+-------------------------+---------------------------+
| **5. Risk      |                         |                           |
| Management**   |                         |                           |
+----------------+-------------------------+---------------------------+
| PR9. Identify  | Risk identification and | During the promotional    |
| and analyze    | analysis is included in | events defined by The     |
| risk events.   | the TPP and AAQR, when  | Trilinos Software         |
|                | and as requested by ASC | Lifecycle Model, risk     |
|                | program management.     | identification is         |
|                |                         | required.                 |
+----------------+-------------------------+---------------------------+
| PR10. Define,  | Risk mitigation and     | Provide quarterly updates |
| monitor, and   | response is included in | to the ASC Algorithms PI. |
| implement the  | the TPP and AAQR, when  |                           |
| risk response. | and as requested by ASC |                           |
|                | program management.     |                           |
+----------------+-------------------------+---------------------------+
| **6. Project   |                         |                           |
| Planning,      |                         |                           |
| Tracking and   |                         |                           |
| Oversight**    |                         |                           |
+----------------+-------------------------+---------------------------+
| PR11. Create   | The TPP is maintained   | None.                     |
| and manage the | at the project level.   |                           |
| project plan.  |                         |                           |
+----------------+-------------------------+---------------------------+
| PR12. Track    | The AAQR is written and | None.                     |
| performance    | submitted quarterly,    |                           |
| versus project | when requested by ASC   |                           |
| plan and       | program management.     |                           |
| implement      |                         |                           |
| needed         |                         |                           |
| (corrective)   |                         |                           |
| actions.       |                         |                           |
+----------------+-------------------------+---------------------------+
| **Software     |                         |                           |
| Engineering**  |                         |                           |
+----------------+-------------------------+---------------------------+
| **7. Software  |                         |                           |
| Development**  |                         |                           |
+----------------+-------------------------+---------------------------+
| PR13.          | Doxygen \[9\], user     | This practice is          |
| Communicate    | mail lists and          | addressed by the          |
| and review     | developer mail lists    | lifecycle model. Briefly, |
| design.        | are provided on the     | design at research phase  |
|                | Trilinos development    | is typically captured in  |
|                | platform                | a notebook or on a mail   |
|                | software.sandia.gov.    | list. During the          |
|                | Trilinos Framework      | production growth phase,  |
|                | level design            | the design is often       |
|                | discussions occur at    | communicated and reviewed |
|                | Trilinos Framework      | during face to face or    |
|                | Developer meetings and  | phone meetings (and       |
|                | on the                  | minutes are sent to a     |
|                | trilinos-framework mail | mail list), or on a       |
|                | list. Meeting minutes   | developer mail list.      |
|                | are also sent to the    | During the production     |
|                | trilinos-framework mail | maintenance phase, formal |
|                | list. All mail list     | documentation and review  |
|                | traffic is subject to   | of design will occur.     |
|                | peer review.            |                           |
|                |                         | In all phases design is   |
|                |                         | also captured in the form |
|                |                         | of Doxygen documentation. |
|                |                         | In most cases, the        |
|                |                         | Doxygen documentation     |
|                |                         | represents a true design  |
|                |                         | as it is produced before  |
|                |                         | the associated code is    |
|                |                         | written.                  |
+----------------+-------------------------+---------------------------+
| PR14. Create   | A CVS repository is     | Packages follow the       |
| required       | provided for all code   | adopted lifecycle for     |
| software and   | and artifacts. Doxygen  | software development.     |
| product        | \[9\] is provided on    |                           |
| documentation. | the Trilinos            |                           |
|                | development platform    |                           |
|                | software.sandia.gov.    |                           |
+----------------+-------------------------+---------------------------+
| **8.           |                         |                           |
| Integration of |                         |                           |
| Third Party or |                         |                           |
| Other          |                         |                           |
| Software**     |                         |                           |
+----------------+-------------------------+---------------------------+
| PR15. Identify | Supported versions of   | Check new supported       |
| and track      | third-party software    | versions of 3PL's into    |
| third party    | can be kept in the      | the Trilinos3PL           |
| software       | Trilinos3PL CVS         | repository and/or         |
| products and   | repository. Third-party | document supported        |
| follow         | software that is widely | versions. Follow all      |
| applicable     | available and adheres   | applicable license        |
| agreements.    | to established          | agreements.               |
|                | standards such as BLAS, |                           |
|                | LAPACK, and MPI are not |                           |
|                | kept under version      |                           |
|                | control. Trilinos       |                           |
|                | supports a wide array   |                           |
|                | of versions and vendors |                           |
|                | of these common         |                           |
|                | libraries that adhere   |                           |
|                | to the appropriate      |                           |
|                | standard.               |                           |
+----------------+-------------------------+---------------------------+
| PR16.          | Trilinos packages do    | Follow all applicable     |
| Identify,      | not generally accept    | license agreements.       |
| accept         | ownership of            | Assimilated code should   |
| ownership, and | third-party software.   | be maintained like core   |
| manage         | If a package team       | package code.             |
| assimilation   | chooses to do so, they  |                           |
| of other       | have the option to use  |                           |
| software       | the Trilinos or         |                           |
| products.      | Trilinos3PL repository  |                           |
|                | to store the code.      |                           |
+----------------+-------------------------+---------------------------+
| **9.           |                         |                           |
| Configuration  |                         |                           |
| Management**   |                         |                           |
+----------------+-------------------------+---------------------------+
| PR17. Perform  | A CVS repository is     | Package developers        |
| version        | maintained for all      | utilize the Trilinos CVS  |
| control of     | Trilinos packages.      | repository.               |
| identified     | Package-checkins mail   |                           |
| software       | lists archive all       |                           |
| product        | product modifications.  |                           |
| artifacts.     | Trilinos release        |                           |
|                | versioning is handled   |                           |
|                | by release and release  |                           |
|                | update process          |                           |
|                | checklists \[14\].      |                           |
+----------------+-------------------------+---------------------------+
| PR18. Record   | A Bugzilla product is   | Package developers, or    |
| and track      | provided for each       | their customers, file     |
| issues         | Trilinos package. All   | issue reports using the   |
| associated     | issues are tracked via  | Trilinos Bugzilla site.   |
| with the       | Bugzilla and the        | This includes major       |
| software       | underlying MySQL \[16\] | feature requests and      |
| product.       | database.               | software problems. Issue  |
|                |                         | reports are kept          |
|                |                         | up-to-date.               |
+----------------+-------------------------+---------------------------+
| PR19. Ensure   | The Trilinos            | None.                     |
| backup and     | development platform,   |                           |
| disaster       | software.sandia.gov is  |                           |
| recovery of    | backed up regularly and |                           |
| software       | backup tapes are        |                           |
| product        | shipped offsite. All    |                           |
| artifacts.     | CVS, Mailman, and       |                           |
|                | Bugzilla data and       |                           |
|                | artifacts are retained  |                           |
|                | indefinitely.           |                           |
+----------------+-------------------------+---------------------------+
| **10. Release  |                         |                           |
| and            |                         |                           |
| Distribution   |                         |                           |
| Management**   |                         |                           |
+----------------+-------------------------+---------------------------+
| PR20. Plan and | Release requests are    | Part of the Trilinos      |
| generate the   | negotiated between the  | Package Level Release     |
| release.       | Trilinos Project Leader | Process Checklist         |
|                | and customers. A        | addresses release         |
|                | release cycle commences | generation. All packages  |
|                | when an announcement of | that are included in a    |
|                | a release target date   | major release are         |
|                | is sent to the          | required to complete this |
|                | Trilinos-developers     | checklist.                |
|                | mail list. This email   |                           |
|                | describes the release   |                           |
|                | plans, or the plan is   |                           |
|                | discussed as part of    |                           |
|                | the monthly Trilinos    |                           |
|                | Leaders Meeting. Part   |                           |
|                | of the Trilinos Level   |                           |
|                | Release Process         |                           |
|                | Checklist and Trilinos  |                           |
|                | Web Release Process     |                           |
|                | Checklist address       |                           |
|                | release generation.     |                           |
+----------------+-------------------------+---------------------------+
| PR21. Certify  | Part of the Trilinos    | Part of the Trilinos      |
| that the       | Level Release Process   | Package Level Release     |
| software       | Checklist and Trilinos  | Process Checklist         |
| product (code  | Web Release Process     | addresses release         |
| and its        | Checklist address       | certification.            |
| related        | release certification.  |                           |
| artifacts) is  | Part of the required    |                           |
| ready for      | certification is        |                           |
| release and    | provided by major       |                           |
| distribution.  | customers.              |                           |
+----------------+-------------------------+---------------------------+
| PR22.          | Part of the Trilinos    | None.                     |
| Distribute     | Level Release Process   |                           |
| release to     | Checklist, Release      |                           |
| customers.     | Update Checklist, and   |                           |
|                | Trilinos Web Release    |                           |
|                | Process Checklist       |                           |
|                | address release         |                           |
|                | distribution.           |                           |
+----------------+-------------------------+---------------------------+
| **11. Customer |                         |                           |
| Support**      |                         |                           |
+----------------+-------------------------+---------------------------+
| PR23. Define   | Customer support is     | Responsible for providing |
| and implement  | addressed in the TPP,   | customer support at the   |
| a customer     | and the Trilinos        | package level.            |
| support plan.  | Strategic Plan \[4\].   |                           |
|                | Frequent communication, |                           |
|                | the Trilinos User Group |                           |
|                | meeting, the Trilinos   |                           |
|                | User Guide, Trilinos    |                           |
|                | Tutorial, any existing  |                           |
|                | package user guides,    |                           |
|                | Bugzilla, the lifecycle |                           |
|                | level of formality, the |                           |
|                | trilinos-help and       |                           |
|                | trilinos-user mail      |                           |
|                | lists, and the list of  |                           |
|                | developer contacts      |                           |
|                | available online are    |                           |
|                | all important           |                           |
|                | components of Trilinos  |                           |
|                | customer support.       |                           |
+----------------+-------------------------+---------------------------+
| PR24.          | Organize the annual     | Create and maintain       |
| Implement the  | Trilinos User Group     | package documentation.    |
| training       | meeting. Maintain the   | Provide tutorials at the  |
| identified in  | Trilinos User Guide and | Trilinos User Group       |
| the customer   | Trilinos Tutorial.      | meeting as appropriate.   |
| support plan.  |                         |                           |
+----------------+-------------------------+---------------------------+
| PR25. Evaluate | Users provide feedback  | Respond promptly to user  |
| customer       | at Trilinos Advisory    | issues. Package specific  |
| feedback to    | Group meetings and via  | surveys can be conducted, |
| determine      | email, issue reports,   | when appropriate.         |
| customer       | and conversations       |                           |
| satisfaction.  | throughout the year.    |                           |
|                | They are also invited   |                           |
|                | to TUG where users are  |                           |
|                | given the chance to     |                           |
|                | provide feedback. In    |                           |
|                | addition, at least one  |                           |
|                | user is invited to give |                           |
|                | a presentation at TUG   |                           |
|                | that often includes     |                           |
|                | comments on their level |                           |
|                | of satisfaction and     |                           |
|                | opportunities for       |                           |
|                | improvement.            |                           |
|                | Occasionally, customer  |                           |
|                | surveys are conducted.  |                           |
+----------------+-------------------------+---------------------------+
| **Software     |                         |                           |
| Verification** |                         |                           |
+----------------+-------------------------+---------------------------+
| **12. Software |                         |                           |
| Verification** |                         |                           |
+----------------+-------------------------+---------------------------+
| PR26. Develop  | The Trilinos software   | None.                     |
| and maintain a | verification plan is    |                           |
| software       | included in the TPP.    |                           |
| verification   | The Trilinos Developer  |                           |
| plan.          | Guide and the Trilinos  |                           |
|                | website include         |                           |
|                | documentation for and a |                           |
|                | discussion of the       |                           |
|                | Trilinos Test Harness   |                           |
|                | to assist developers in |                           |
|                | setting up package      |                           |
|                | testing.                |                           |
+----------------+-------------------------+---------------------------+
| PR27. Conduct  | The results from        | Package developers are    |
| tests to       | nightly test cases are  | responsible for deciding  |
| demonstrate    | automatically mailed to | what tests need to be     |
| that           | the appropriate         | written for their         |
| acceptance     | <package-regressi       | packages. Part of the     |
| criteria are   | on@software.sandia.gov> | Trilinos Package Level    |
| met and to     | mail list. The results  | Release Process Checklist |
| ensure that    | are archived. The       | addresses testing that    |
| previously     | completion of           | must be completed prior   |
| tested         | acceptance tests that   | to a release.             |
| capabilities   | are run by users before |                           |
| continue to    | a release are also      |                           |
| perform as     | archived via the        |                           |
| expected.      | <trilinos-framewo       |                           |
|                | rk@software.sandia.gov> |                           |
|                | and/or the              |                           |
|                | <trilinos-develope      |                           |
|                | rs@software.sandia.gov> |                           |
|                | mail list. Release      |                           |
|                | testing is addressed by |                           |
|                | part of the Trilinos    |                           |
|                | Level Release Process   |                           |
|                | Checklist.              |                           |
+----------------+-------------------------+---------------------------+
| PR28. Conduct  | Framework list          | The formality of          |
| independent    | discussions and peer    | technical reviews depends |
| technical      | reviews occur as        | on which phase of the     |
| reviews to     | appropriate. Feedback   | lifecycle model package   |
| evaluate       | on the adequacy of      | is in. During the         |
| adequacy with  | existing and new        | research phase, this      |
| respect to     | framework components is | often means publishing    |
| requirements.  | gathered at TUG and     | results. During the       |
|                | Trilinos Monthly        | production maintenance    |
|                | Leaders meetings.       | phase, formal inspections |
|                |                         | occur as appropriate.     |
|                |                         | During the production     |
|                |                         | growth phase publications |
|                |                         | are produced and/or       |
|                |                         | inspections occur as      |
|                |                         | appropriate.              |
+----------------+-------------------------+---------------------------+
| **Training**   |                         |                           |
+----------------+-------------------------+---------------------------+
| **13.          |                         |                           |
| Training**     |                         |                           |
+----------------+-------------------------+---------------------------+
| PR29.          | Training is performed   | None.                     |
| Determine      | as needed during the    |                           |
| project team   | monthly Trilinos        |                           |
| training       | Leaders Meeting.        |                           |
| needed to      | Training events are     |                           |
| fulfill        | announced on the        |                           |
| assigned roles | meeting agenda that is  |                           |
| and            | sent to the             |                           |
| res            | trilinos-leaders mail   |                           |
| ponsibilities. | list prior to the       |                           |
|                | meeting. Package        |                           |
|                | leaders or designated   |                           |
|                | representatives are     |                           |
|                | expected to attend.     |                           |
|                | Other training events,  |                           |
|                | such as the Software    |                           |
|                | Engineering Seminar     |                           |
|                | Series, are announced   |                           |
|                | on the                  |                           |
|                | trilinos-developers     |                           |
|                | mail list. Training,    |                           |
|                | based on the needs      |                           |
|                | determined by the       |                           |
|                | Trilinos Project        |                           |
|                | Leader, is also         |                           |
|                | frequently provided     |                           |
|                | during developer day at |                           |
|                | the Trilinos User Group |                           |
|                | meeting.                |                           |
+----------------+-------------------------+---------------------------+
| PR30. Track    | Training is tracked by  | None.                     |
| training       | checking attendance     |                           |
| undertaken by  | lists into the          |                           |
| project team.  | TrilinosSQE CVS         |                           |
|                | repository, or via      |                           |
|                | meeting minutes sent to |                           |
|                | the appropriate mail    |                           |
|                | list.                   |                           |
+----------------+-------------------------+---------------------------+

#### References

\[1\] Edward A. Boucheron, Richard R. Drake, H. Carter Edwards, Molly A.
Ellis,

Christi A. Forsythe, Robert Heaphy, Ann L. Hodges, Constantine Pavlakos,

Joseph R. Schofield, Judy E. Sturtevant and C. Michael Williamson,
*Sandia National Laboratories Advanced Simulation and Computing (ASC)
Software Quality Plan Part 1: ASC Software Quality Engineering
Practices, Version 1.0*, Sandia National Laboratories, SAND2004-6602,
January 2005.

\[2\] M. Heroux, J. Willenbring and R. Heaphy, *The Trilinos Developers
Guide, Version 1.0*, Sandia National Laboratories, SAND2003-1898, May
2003.

\[3\] J. Willenbring, R. Heaphy, M. Heroux and M. Phenow, *The Trilinos
Software Lifecycle Model*, Sandia National Laboratories, SAND2006-6929,
November 2006.

\[4\] M. Heroux, J. Willenbring and R. Heaphy, *Trilinos Project
Strategic Plan*, Sandia National Laboratories, SAND2008-XXXX, January
2008.

\[5\] S. Collis, *Objectives and Resource plan for ASC Algorithms R&D,
Version 2.1,* 10/05/2006.

\[6\] Gnu CVS Home Page: <http://www.gnu.org/software/cvs>.

\[7\] Mozilla Bugzilla Home Page:
<http://www.mozilla.org/projects/bugzilla>.

\[8\] Mailman Home Page: <http://www.gnu.org/software/mailman>.

\[9\] Doxygen Home Page:
[http://www.doxygen.org](http://www.doxygen.org/).

\[10\] Autoconf Home Page: <http://www.gnu.org/software/autoconf>.

\[11\] Automake Home Page: <http://www.gnu.org/software/automake>.

\[12\] G. Vaughan, B. Elliston, T. Tromey, and I. Taylor. *GNU Autoconf,
Automake and Libtool.*New Riders, 2000.

\[13\] Microsoft Project Home Page:
<http://www.microsoft.com/office/project>.

\[14\] Trilinos Process Checklists:
<http://software.sandia.gov/trilinos/developer/sqp/checklists/index.html>.

\[15\] Trilinos Test Harness Results:
<http://software.sandia.gov/trilinos/developer/test_harness/results/index.html>.

\[16\] MySQL Home Page: [http://www.mysql.com](http://www.mysql.com/).