# Presentation: 290931

## Software Engineering in the Trilinos Project: Then, Now and To Come Michael A. HerouxSandia National Laboratories

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

**Notes:**
Title:  Software Engineering in the Trilinos Project:  Then, 
> Now and to Come
> Speaker:  Michael A. Heroux
> Date:  Wednesday October 3rd, 2007
> Location & Time:  CSRI/90 @ 11:30am
> 
> Brief Abstract:
> From the very beginning of the Trilinos project software 
> engineering processes have been a core focus area.  At the 
> same time, we have taken a pragmatic, organic approach to 
> improving software practices and tools.  In this presentation 
> we provide an overview of the state of software engineering 
> in the Trilinos project, its history and some ideas for where 
> we can grow in the future.

## Trilinos Contributors (past 3 years)

**Chris Baker**
- Developer of Anasazi, RBGen
**Ross Bartlett **
- Lead Developer of MOOCHO, Stratimikos, RTOp,
- Thyra
- Developer of Rythmos
**Pavel Bochev**
- Lead Developer Intrepid
**Paul Boggs **
- Developer of Thyra
**Erik Boman**
- Lead Developer Isorropia
- Developer Zoltan
**Todd Coffey**
- Lead Developer of Rythmos
**Karen Devine**
- Lead Developer Zoltan
**Clark Dohrmann**
- Lead Developer of CLAPS
**Carter Edwards**
- Lead Developer phdMesh
**Michael Gee **
- Developer of ML, Moertel, NOX
**Bob Heaphy **
- Lead developer of Trilinos SQA
- Developer Zoltan

**Mike Heroux **
- Trilinos Project Leader
- Lead Developer of Epetra, AztecOO,
- Kokkos, IFPACK, Tpetra
- Developer of Amesos, Belos, EpetraExt
**Ulrich Hetmaniuk**
- Developer of Anasazi
**Robert Hoekstra **
- Lead Developer of EpetraExt
- Developer of Epetra
**Russell Hooper**
- Developer of NOX
**Vicki Howle **
- Lead Developer of Meros
**Jonathan Hu **
- Developer of ML
**Joe Kotulski**
- Lead Developer of Pliris
**Rich Lehoucq **
- Developer of Anasazi and Belos
**Kevin Long **
- Developer of Thyra
**Roger Pawlowski **
- Lead Developer of NOX
**Michael Phenow **
- Trilinos Webmaster** **
- Developer WebTrilinos
**Eric Phipps**
- Lead developer Sacado
- Developer of LOCA, NOX

**Dennis Ridzal**
- Lead Developer of Aristos, Intrepid
**Marzio Sala**
- Lead Developer of Didasko, Galeri, IFPACK, WebTrilinos
- Developer of ML, Amesos
**Andrew Salinger **
- Lead Developer of LOCA, Capo
**Paul Sexton **
- Developer of Epetra and Tpetra
**Bob Shuttleworth**
- Developer of Meros.
**Chris Siefert**
- Developer of ML
**Bill Spotz**
- Lead Developer of PyTrilinos
- Developer of Epetra, New_Package
**Ken Stanley **
- Lead Developer of Amesos and New_Package
**Heidi Thornquist **
- Lead Developer of Anasazi, Belos, RBGen  and Teuchos
**Ray Tuminaro **
- Lead Developer of ML  and Meros
**Jim Willenbring **
- Developer of Epetra and New_Package.
- Trilinos library manager** **
**Alan Williams **
- Lead Developer Isorropia, FEI
- Developer of Epetra, EpetraExt, AztecOO, Tpetra

## Take Home Messages

- We (the Trilinos team) are not experts, but committed to improvement.
- Quality matters.
- New practices and tools can help us.

## Background/Motivation

## Target Problems: PDES and more...

**PDES**

**Circuits**

**Inhomogeneous**
**Fluids**

**And More...**

## Target Platforms: Any and All(Now and in the Future)

- Desktop: Development and more...
- Capability machines:
- Redstorm (XT3), Clusters
- Roadrunner (Cell-based).
- Large-count multicore nodes.
- Parallel software environments:
- MPI of course.
- UPC, CAF, threads, vectors,...
- Combinations of the above.
- User “skins”:
- C++/C, Python
- Fortran.
- Web, CCA.

## Motivation For Trilinos

- Sandia does LOTS of solver work.
- When I started at Sandia in May 1998:
- Aztec was a mature package.  Used in many codes.
- FETI, PETSc, DSCPack, Spooles, ARPACK, DASPK, and many other codes were (and are) in use.
- New projects were underway or planned in multi-level preconditioners, eigensolvers, non-linear solvers, etc...
- The challenges:
- Little or no coordination was in place to:
- Efficiently reuse existing solver technology.
- Leverage new development across various projects.
- Support solver software processes.
- Provide consistent solver APIs for applications.
- ASCI (now ASC) was forming software quality assurance/engineering (SQA/SQE) requirements:
- Daunting requirements for any single solver effort to address alone.

## Evolving Trilinos Solution

**Evolving Trilinos Solution**

- Trilinos1 is an evolving framework to address these challenges:
- Fundamental atomic unit is a *package*.
- Includes core set of vector, graph and matrix classes (Epetra/Tpetra packages).
- Provides a common abstract solver API (Thyra package).
- Provides a ready-made package infrastructure (new_package package):
- Source code management (cvs, bonsai).
- Build tools (autotools).
- Automated regression testing (queue directories within repository).
- Communication tools (mailman mail lists).
- Specifies requirements and suggested practices for package SQA.
- In general allows us to categorize efforts:
- Efforts best done at the Trilinos level (useful to most or all packages).
- Efforts best done at a package level (peculiar or important to a package).
**Allows package developers to focus only on things that are unique to ****their package.**

- 1. Trilinos loose translation: “A string of pearls”

## Evolving Trilinos Solution

**Evolving Trilinos Solution**

- Beyond a “solvers” framework
- Natural expansion of capabilities to satisfy application and research needs

- Discretization methods, AD, Mortar methods, ...

## Trilinos Package Concepts

***Package: The Atomic Unit***

## Trilinos Packages

- Trilinos is a collection of *Packages*.
- Each package is:
- Focused on important, state-of-the-art algorithms in its problem regime.
- Developed by a small team of domain experts.
- Self-contained: No explicit dependencies on any other software packages (with some special exceptions).
- Configurable/buildable/documented on its own.
- Sample packages: NOX, AztecOO, ML, IFPACK, Meros.
- Special package collections:
- Petra (Epetra, Tpetra, Jpetra): Concrete Data Objects
- Thyra: Abstract Conceptual Interfaces
- Teuchos: Common Tools.
- New_package: Jumpstart prototype.

## Trilinos Package Summary

|  | Objective | Package(s) |
| --- | --- | --- |
| Discretizations | Meshing & Spatial Discretizations | phdMesh, Intrepid |
|  | Time Integration | Rythmos |
| Methods | Automatic Differentiation | Sacado |
|  | Mortar Methods | Moertel |
| Core | Linear algebra objects | Epetra, Jpetra, Tpetra |
|  | Abstract interfaces | Thyra, Stratimikos, RTOp |
|  | Load Balancing | Zoltan, Isorropia |
|  | “Skins” | PyTrilinos, WebTrilinos, Star-P, ForTrilinos |
|  | C++ utilities, (some) I/O | Teuchos, EpetraExt, Kokkos, Triutils |
| Solvers | Iterative (Krylov) linear solvers | AztecOO, Belos, Komplex |
|  | Direct sparse linear solvers | Amesos |
|  | Direct dense linear solvers | Epetra, Teuchos, Pliris |
|  | Iterative eigenvalue solvers | Anasazi |
|  | ILU-type preconditioners | AztecOO, IFPACK |
|  | Multilevel preconditioners | ML, CLAPS |
|  | Block preconditioners | Meros |
|  | Nonlinear system solvers | NOX, LOCA |
|  | Optimization (SAND) | MOOCHO, Aristos |

## Characterizing the Trilinos “Project”

- Not a “project” but an infrastructure to support inter-related projects: A project of projects.
- Package participation is voluntary:
- Framework must be attractive (and continue to be).
- Requirements are few, opportunities are many.
- Package team decides what and when.
- Opt-out is always an option.
- Package autonomy is carefully guarded:
- Even if redundant development occurs.
- Decision-making pushed to lowest (best) level.
- Participation is attractive:
- Increasing infrastructure capabilities.
- Access to many other packages.

## Implications

- Package teams must see value of SE tools and processes.
- Won’t adopt something because SE theory says it’s good.
- Might adopt something “imposed” on them:
- But use will be temporary.
- Ultimately a negative impact.
- New tools and processes adopted incrementally:
- Package teams busy with algorithm development.
- Open to learning SE as time permits.
- Prefer established tools to bleeding edge:
- Package teams busy with algorithm development.

## Trilinos Strategic Goals

**Scalable Computations:** As problem size and processor counts increase, the cost of the computation will remain nearly fixed.
**Hardened Computations:** Never fail unless problem essentially intractable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error.
**Full Vertical Coverage:** Provide leading edge enabling technologies through the entire technical application software stack: from problem construction, solution, analysis and optimization.
**Grand Universal Interoperability:** All Trilinos packages will be interoperable, so that any combination of packages that makes sense algorithmically will be possible within Trilinos and with compatible external software.
**Universal Accessibility:** All Trilinos capabilities will be available to users of major computing environments: C++, Fortran, Python and the Web, and from the desktop to the latest scalable systems.
**Universal Solver RAS:** Trilinos will be:
- Integrated into every major application at Sandia (**Availability**).
- The leading edge hardened, efficient, scalable solution for each of these applications (**Reliability**).
- Easy to maintain and upgrade within the application environment (**Serviceability**).

- Algorithmic
- Goals

- Software
- Goals

## Then

***When Trilinos was unknown***

## Important Early SE Decisions2000-2002 

- Package Autonomy:
- Carefully monitored by Tammy Kolda.
- Meeting NOX needs kept us on track.
- Self-contained OO Data Model:
- Critical to decoupling packages.
- Participation in Equation Solver Interface (ESI) Forum.
- Petra heavily influenced by Alan Williams, Robert Clay OO design.
- Public visibility:
- Public repository: Improves SW quality (*ad hoc* code reviews)*.*
- Use of mail lists to (over) communicate.
- Regular phone meetings.
- Tammy Kolda and Paul Sery critical here.
- Focus on working with customer:
- Problem-driven, close interaction.
- Roger Pawlowski integrating NOX into many codes.

## Starting Points

- uid=26437(mheroux) ...
- CVSROOT loginfo,1.2,1.3
- Fri Dec 14 15:26:06 PST 2001
- Update of /usr/local/CVS/CVSROOT
- In directory iterative:/tmp/cvs-serv5952
- Modified Files:
-         loginfo
- Log Message:
- Fixed mail alias on loginfo

- Epetra-Checkins Archives
- Starting: Sat Sep 28 07:19:50 2002
- Ending: Sat Sep 28 08:38:37 2002
- Messages: 4
- [Epetra-Checkins] Trilinos/packages/epetra/src classicMakefile   mheroux@skip.sandia.gov
- [Epetra-Checkins] Trilinos/packages/epetra/src classicMakefile   mheroux@skip.sandia.gov
- [Epetra-Checkins] Trilinos/packages/epetra/src classicMakefile   mheroux@skip.sandia.gov
- [Epetra-Checkins] Trilinos/packages/epetra/src classicMakefile   mheroux@skip.sandia.gov
- Last message date: Sat Sep 28 08:38:37 2002
- Archived on: Sat Sep 28 01:39:00 2002

## Process Improvement

- Process Improvement: Tradition for Trilinos.
- Yearly rollout of new goals.
- Discussed and decided during Trilinos User Group meeting (Nov 6-8 this year).
- Here are some past years.

## Major Framework Themes: FY05/06

- Trilinos Package Architecture:
- Continue refinement of *new_package.*
- Explicitly define Trilinos compatibility.
- Resolve the abstract interface issue.
- Software Quality:
- Expand use and ease-of-use of test harness.
- Identify metrics and automate capture and display.
- Establish a life-cycle model (hybrid agile/unified process?).
- Customize the ASC SQP to our environment.
- Packages:
- Foster new package development.
- Manage the growth.  Issue: complexity of package coupling.
- Harden our mature packages.
- Transition to post-delivery maintenance:
- Organizational issue: Tough to solve.

## Major Themes for FY06/07Framework

- Take Steps toward dynamic package addition.
- In light of more outward focus:(SciDAC, Boeing, ???)
- Trilinos-compatibility definition or something similar.
- Define SW Lifecycle(s) and begin formalized efforts.
- Need something for external audits.
- Agile vs. UP vs. hybrid.

## FY07/08 Themes: Framework

- Trilinos Level II Milestone.
- Demonstrate Use of Full Vertical capabilities in Charon.
- Joint licensing and copyright of development with other organizations.
- Other DOE Labs, international orgs, private companies.
- Package Autonomy (reacting to rapid growth):
- 18 to 27  to ??
- Guarding against incidental coupling.
- Revisiting the location of “skins”.
- Stratimikos: Uniform access to many packages.
- Access from Fortran.
- DOE Science Users, DOD Users.
- Stratimikos focus.
- Split of User vs Developer tools.
- _[http:](http://software.sandia.gov/trilinos/rd100_2004.html)__[//trilinos](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[sandia](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[gov](http://software.sandia.gov/trilinos/rd100_2004.html)_
- http://software.sandia.gov

## Remainder of “Then”

- Use of serious configuration tools.
- Active, well used website.
- Responsive support of framework.
- Much more...
- Attitude of Trilinos contributors:
- True desire to work together.
- Focused on problem solving.
- The primary reason for success.

## Now: Current Tools & Practices

## Software Goals

- Quality:
- Everybody wants it, but expensive in time and money.
- No longer sufficient to say “Trust us, we do good work.”
- Modularity/Interoperability:
- Tight cohesion within a package.
- Loose coupling across packages.
- Scalability:
- Make the cost of adding/supporting a new package as low as possible.
- Shield the user from this complexity as much as possible.
- Efficient use of Experts’ Time:
- Provide as much support to package teams as possible.
- Provide defaults with ‘opt-out’ clauses.
- Accessibility and Support: Get the software out there.

## Software Principles

- Package Orthogonality:
- Trilinos “string-of-pearls”.
- Asynchronous, flexible package maturation process.
- Global Services:
- Framework:  Established tools and processes.
- NewPackage package.
- Tight collaboration:
- Environment that fosters interaction:
- Across package teams.
- With Users and Clients.
- Tools for automated communication of events.
- Iterative Development: User access to multiple SW branches.
- Process Improvement:
- Record what we do and don’t do but want to do.
- Track progress.

## Software Tools and Practices

- Source Management: 	CVS & Bonsai.
- Communication channels: 	Email lists, regular phone meetings.
- Documentation: 	Custom docs, tech reports, journals, Doxygen.
- Configuration management: 	Autoconf, Automake.
- Information distribution: 	Website.
- Testing: 	Test Harness.
- Release process: 	Formal, checklist driven.

## Development Practices and Tools

*What We Do and Use*

## Source Management

- Source management is a first order activity:
- Required for any other process.
- Useful for personal files too!
- CVS:
- Tried and True.
- No User Training Required.  Free. Ubiquitous.
- Not the fullest in features anymore.
- Subversion (svn) clearly a good replacement at some point.
- Already used by related projects.
- Not yet for Trilinos: Bonsai.
- Other Options: GIT, Mercurial, ...

## Bonsai “blame mode”

## Mailman: Extensive use for many types of communication

## Issue Tracking

- Bugzilla:
- Normal everyday issues.
- Release Process “metabugs”.
- Performance testing records.
- Completed release checklists.
- 2458: Metabug blocked by all normal bugs to be fixed before release.
- 2447: Trilinos level release bug blocked by Epetra release bug.

## Sometimes we have a little fun!

**Sometimes we have a little fun!**

## Reference Documentation

- Automatically generate twice a day from repository.
- Avid fans of Doxygen:
- Dimitr van Heesch
- http://www.stack.nl/~dimitri/doxygen/

## Building Trilinos

**Building Trilinos**

## Simple Sample Scripts

- ../configure

- ../configure --prefix=/home/mheroux/Trilinos/EPETRA_OPT_SERIAL \
-          CXXFLAGS="-O3“ --disable-default-packages --enable-epetra

- ../configure --enable-mpi --with-mpi=/usr/MPICH/SDK.gcc \
-              --prefix=/home/mheroux/Trilinos/EPETRA_MPI \
-              --with-mpi-cxx=g++ --with-mpi-cc=gcc --with-mpi-f77=g77 \
-              --with-mpi-libs=-lmpich \
-             --disable-default-packages --enable-epetra

-  Builds serial version of default packages.
-  All 3rd party libraries (BLAS, LAPACK) must be in lib search path.

-  Builds serial version of Epetra only with some optimization.
-  All 3rd party libraries (BLAS, LAPACK) must be in lib search path.
-  “make install” will put /include/*.h and /lib/*.a in specified directory.

-  Builds MPI version of Epetra only with specified MPI environment.
-  All 3rd party libraries (BLAS, LAPACK) must be in lib search path.
-  “make install” will put /include/*.h and /lib/*.a in specified directory.

## More Complex Sample ScriptsMost found in Trilinos/sampleScripts

- ../configure --host=powerpc-ibm-aix5.1.0.0 \
-   --enable-teuchos --enable-teuchos-extended --disable-new-package \
-   --enable-aztecoo-azlu \
-   --enable-anasazi \
-   --enable-epetraext --enable-epetraext-transform \
-   --enable-epetraext-transform-tests \
-   --enable-amesos \
-   --with-ml_teuchos --with-ml_anasazi \
-   --with-ml_external_mpi_functions \
-   --disable-examples \
-   --with-blas="-lessl -L/sierra/Release/lapack/3.0/lib/dbg_dp_ibm -lblas" \
-   --with-lapack="-lessl -L/sierra/Release/lapack/3.0/lib/dbg_dp_ibm -llapack" \
-   --enable-mpi \
-   --with-ldflags="-L/sierra/Release/y12m/1.00/lib/dbg_dp_ibm -ly12m" \
-   --prefix=/sierra/Release/Trilinos/3.1.1/install_ibm \
-   --with-mpi-incdir=. \
-   --with-mpi-libdir=. \
-   --with-mpi-libs="-binitfini:poe_remote_main -lmpi_r -lvtd_r" \
-   --with-ar="ar -X64 csrv" \
-   CXX="mpCC -q64" CC="mpcc -b64 -q64" F77="mpxlf -b64 -q64" \
-   CXXFLAGS="-O3 -w -qnofullpath -qlanglvl=ansi" \
-   CCFLAGS="-O3 -w" \
-   FFLAGS="-O3 -w" \
-   CPPFLAGS=  LDFLAGS= LIBS=-lm \
-   FLIBS="-lxlf90 -lxlopt -lxlf -lxlomp_ser"

-  Configure/make on some platforms is   complex.
-  We maintain a directory of sample   scripts to guide installers.

## Website

- http:trilinos.sandia.gov Developer content on software.sandia.gov.
- Always looking to improve layout, content.
- Site was recently redesigned.

## Test Harness

- Tests detected in repository.
- Presence of test definition file triggers harness run.
- Form of def file:
- 	## This group defines # CATEGORY-1 and # CATEGORY-2 tests. # (CATEGORY-1, CATEGORY-2) {      NAME-A = VALUE-A;      NAME-B = VALUE-B;      NAME-C = VALUE-C; } # # This group defines# CATEGORY-3 tests. # (CATEGORY-3) {      NAME-A = VALUE-A;      NAME-B = VALUE-B;      NAME-C = VALUE-C; }

## Test Harness Example

## Release Process

- Checklists Drive the Release Process.
- Quality and Stability of the Release has improve dramatically with each additional list.
- Let’s Look at one.

## Package-level Release Process Checklist

## Four Parts: Plan, Do, Check, Act

- Plan: 	Determine what you will do.
- Do: 		Do it.
- Check: 	Confirm results.
- Act: 	Take action to complete process.Also known as the Deming Cycle (W. Edwards Deming)

## Part 1: Plan

## Part 2: Do

## Part 3: Check

## Part 4: Act

## History of a Package Maturation ProcessExample: ML

***Use of Tools******Asynchronicity***

## History of ML Maturation Process

| Step | Example |
| --- | --- |
| Package added to CVS: Import existing code or start with new_package. | ML CVS repository migrated into Trilinos (July 2002). |
| Mail lists, Bugzilla Product, Bonsai database created. | ml-announce, ml-users, ml-developers, ml-checkins, ml-regression @software.sandia.gov created, linked to CVS (July 2002). |
| Package builds with configure/make, Trilinos-compatible | ML adopts Autoconf, Automake starting from new_package (June 2003). |
| Epetra objects recognized by package. | ML accepts user data as Epetra matrices and vectors (October 2002). |
| Package accessible via Thyra interfaces. | ML adaptors written for TSFCore_LinOp (Thyra) interface (May 2003). |
| Package uses Epetra for internal data. | ML able to generate Epetra matrices.  Allows use of AztecOO, Amesos, Ifpack, etc. as smoothers and coarse grid solvers (Feb-June 2004). |
| Package parameters settable via Teuchos ParameterList | ML gets manager class, driven via ParameterLists (June 2004). |
| Package usable from Python  (PyTrilinos) | ML Python wrappers written using new_package template (April 2005). |

- Startup Steps

- Maturation Steps

## Day 1 of Package Life

**CVS:** Each package is self-contained in Trilinos/package/ directory.
**Bugzilla:** Each package has its own Bugzilla product.
**Bonsai:** Each package is browsable via Bonsai interface.
**Mailman:** Each Trilinos package, including Trilinos itself, has four mail lists:
- package-checkins@software.sandia.gov
- CVS commit emails. “Finger on the pulse” list.
- package-developers@software.sandia.gov
- Mailing list for developers.
- package-users@software.sandia.gov
- Issues for package users.
- package-announce@software.sandia.gov
- Releases and other announcements specific to the package.
**New_package** (optional): Customizable boilerplate for
- Autoconf/Automake/Doxygen/Python/Thyra/Epetra/TestHarness/Website

## Maturation Jumpstart:  NewPackage

**Maturation Jumpstart:  NewPackage**

- NewPackage provides jump start to develop/integrate a new package
- NewPackage is a “Hello World” program and website:
- Simple but it does work with autotools.
- Compiles and builds.
- NewPackage directory contains:
- Commonly used directory structure: src, test, doc, example, config.
- Working Autoconf/Automake files.
- Documentation templates (doxygen).
- Working regression test setup.
- Working Python and Thyra adaptors.
- Substantially cuts down on:
- Time to integrate new package.
- Variation in package integration details.
- Development of website.

- NOTE: NewPackage can be used independent from Trilinos

## Software Quality

## SQA/SQE

- Software Quality Assurance/Engineering is important.
- Not sufficient to say, “We do a good job.”
- Trilinos facilitates SQA/SQE development/processes for packages:
- 10 of 30 ASC SQE practices are directly handled by Trilinos (no requirements on packages).
- Trilinos provides infrastructure support for the remaining 20.
- Trilinos Dev Guide Part II: Specific to ASC requirements.
- Trilinos software engineering policies provide a ready-made infrastructure for new packages.
- Trilinos philosophy: Few *requirements*.  Instead mostly *suggested practices.  *Provides package with option to provide alternate process.

## Slide 54

| Trilinos Service | SQE Practices Impact |
| --- | --- |
| Yearly Trilinos User Group Meeting (TUG) and Developer Forum:
Once a year gathering for tutorials, package feature updates, user/developer requirements discussion and developer  training. | — All Requirements steps: gathering,    derivation, documentation, feasibility,etc.— User  and Developer training. |
| Monthly Trilinos leaders meetings:
Trilinos leaders, including package development leaders, key managers, funding sources and other stakeholders participate in monthly phone meetings to discuss any timely issues related to the Trilinos Project. | —Developer Training.
—Design reviews.
—Policy decisions across all development    phases. |
| Trilinos and package mail lists:
Trilinos lists for leaders, announcements, developers, users, checkins and similar lists at the package level support a variety of communication. All lists are archived, providing critical artifacts for assessments and audits. | —Developer/user/client communication.
—Requirements/design/testing artifacts.
—Announcement/documenting of releases.
 |
| Trilinos and Trilinos3PL source repositories: 
All source code, development and user documentation is retained and tracked. In addition, reference versions of all external software, including BLAS, LAPACK, Umfpack, etc. are retained in Trilinos3PL. | —Source management.
—Versioning.
—Third-party software management. |
| Bugzilla Products: 
Each package has its own Bugzilla Product with standard components. | —Requirements/faults capturing and     tracking. |
| Trilinos configure script and M4 macros:
 The Trilinos configure script and related macros support portable installation of Trilinos and its packages | —Portability.
—Software release.
 |
| Trilinos test harness: 
Trilinos provides a base testing plan and automated testing across multiple platforms, plus creation of testing artifacts. Test harness results are used to derive a variety of metrics for SQE. | —Pre-checkin and regression testing.
—Software metrics.
 |

## Software Lifecycles

## (Typical) Project Lifecycle

- Consider this lifecycle

## Scientific Research and Life Cycle Models

- Life Cycle Models are generally developed from the point of view of business software.
- Little consideration is given to algorithmic development.
- Traditional business execution environment is traditional mainframe or desktop, not parallel computers.
- Traditional development “techniques” are assumed.

## Research Software needs a different model

- Research should be “informal”:
- Allow external collaborators, students, post-docs, etc.
- Allow changes of direction without seeking permission
- Should use modern software development paradigms
- i.e. Lean/Agile methods
- Must be verified more than validated
- Production code must:
- Have formality appropriate to risks,
- Be Complete (documentation, testing, ...),
- Be “user proofed”,
- Be platform independent (as necessary),
- Be validated not just verified.

## “Promotional” Model

**Phase *****k***

**Phase *****k+1***

**Promotional**
**Event**

- Lower formality
- Fewer Artifacts
- Lean/Agile

- Higher formality
- Sufficient Artifacts
- Bullet proof
- Maintainable

## Trilinos Software Lifecycle Model

## Lifecycle Models

- A major component of any software project.
- Exists, whether formal or *ad *hoc.
- Trilinos model:
- Really a meta-model.
- Attempts to captures the reality of our software engineering environment.

## Trilinos Software Environment

- Many formal software lifecycle models exist.
- Trilinos environment seems somewhat unique:
- (When compared to commercial environments, or not?)
-  On one hand: Tasked to develop algorithms and software that are leading-edge, with the goal of solving problems that were previously intractable.
- On the other: Required to deliver software that can eventually be used to certify critically important engineering systems.

## Further Complexities

- Trilinos composed of *packages*:
- Self-contained pieces of software developed by semi-independent small teams.
- Each package matures at its own pace:
- Typically evolving from small algorithms study.
- Becoming a widely-used piece of software.
- Embedded in multiple applications.
- Requirements for rigor change as a particular Trilinos package matures.

## Trilinos Lifecycle Phases

- Three phases:
- Research.
- Production Growth.
- Production Maintenance.
- Each phase contains its own lifecycle model.
- Promotional events:
- Required for transition from one phase to next.
- Signify change in behaviors and attitude.
- Phase assigned individually to each package.

## Lifecycle Phase 1: Research

- Conducting research is the primary goal.
- Producing software is potentially incidental to research.
- Any software that is produced is typically a “proof of concept” or prototype.
- Software that is in this phase may only be released to selected internal customers to support their research or development and should not be treated as production quality code.

## Phase 1 Required Practices

- The research proposal is the project plan.
- Software is placed under configuration control as needed to prevent loss due to disaster.
- Peer reviewed published papers are primary verification and validation.
- The focus of testing is a proof of correctness, not software.
- Periodic status reports should be produced, presentation is sufficient.
- A lab notebook, project notebook, or equivalent is the primary artifact.

## Phase 1 Remarks

- Phase 1 practices are common to efficient research in general.
- Research phase software:
- Need not be written in the “target” language.
- Nor support all target machines.
- Usually has limited error checking and recovery.
- (Software) risk is low (primarily technical, not mitigated formality processes.
- Level of formality is low.

## Phase 2: Production Growth

- Goals:
- Elevate package to releasable product.
- Satisfy the Advanced Scientific Computing (ASC) Software Quality Plan, at a minimum.
- Make software product suitable for use by highly skilled users.

## Phase 1 2 Promotion Event

- Risk Assessment:
- What are the package’s primary technical and project management risks?
- How can these risks be mitigated?
- Gap Analysis:
- Which practices and processes must be added or improved to get the package into a releasable state?
- What special actions or training will be required?
- What is the target date for complying with the level of practices and processes required for release?
- Promotion Decision:
- Considering the results of the risk assessment, gap analysis, and other data, will the package be promoted to Phase 2?
- What is the target date for releasing the package?

## Phase 2 Required Practices(Most Important)

- Agile methods (with associated lifecycles) are encouraged, for example the practices and processes promoted by Extreme Programming .
- All essential ASC SQE practices performed at an appropriate level (predetermined during promotion event from the research phase).
- Artifacts should naturally “fall out” from SQE practices and periodic status reviews and management reports.
- Process improvement and metrics are appropriate.

## Phase 2 Remarks

- Phase may be cyclic (spiral, etc.) as new algorithms become incorporated.
- Software may not yet support all intended missions or platforms.
- Risk level is medium:
- Technical risks are reduced.
- Total risk is more project management oriented such as schedule, budget, staffing, etc.).
- Default level of formality is medium.

## Phase 3: Production Maintenance

- Goal: Robust software suitable for typical end uses.
- At this point:
- Requirements and prototype software foundation are stable.
- However, Agile methods no long sufficient:
- Product maintenance during the coming decades of software use where typically only adaptive maintenance
- Response to computer system changes.
- More complete set of artifact is required.
- Software itself will require changes to improve maintainability.
- In extreme case: May make sense to rewrite large portions.

## Phase 2  3 Promotional Event

- Risk Assessment:
- What are the package’s primary technical and project management risks?
- How can these risks be mitigated?
- Gap Analysis:
- Which practices and processes must be added or improved to get the package into a maintenance ready state?
- What is the target date for complying with the required level of practices and processes?

## Phase 2  3 Promotional Event(cont)

- Promotion Decision:
- What is the medium to long-term funding outlook for the package?
- Who is going to provide long-term maintenance services for the project?  (One or more of the original developers, or a different group?)
- If funding is not likely to be available for future maintenance, current customers should be notified so they have a chance to offer continued funding if it is in their best interest.  A list of these customers should be produced.
- If the customer base of the package is small or the package has been replaced with another code, the development team may consider retiring the code rather than moving to the third development phase.  Any such decision should be approved by customers and management.
- Considering the answers to the above questions, and other available data, will the package be promoted to Phase 3?
- What is the target date (if any) for turning the package over to the long-term maintenance team?

## Phase 3: Required Practices

- After achieving maintenance ready status, the package may (as determined during the promotion event) be handed over to another party during this phase for continued support and development.
- If the code is transferred to a different party, the ownership of the design is not necessarily transferred. The design owner must attend meetings concerning requirements changes and potential design changes.
- A widely-accepted lifecycle model such as the Waterfall or Unified Process  methods is used.
- End of life planning is a key component during this phase.  In particular, the software must have good compliance with SQE practices and internal documentation must be formal (UML is suggested).
- SQE practice compliance and solid documentation will help to ensure a successful transfer of the code, and must be completed whether a transfer is currently planned or not.

## Phase 3 Remarks

- The risk level is low (almost totally project management risks, which can be mitigated by appropriate process formality). The default level of formality is high (particularly if the project may be handed off to another party).
- This phase is untested so far.
- Moving to this phase is expensive.

## Exceptional Cases

- Isolated Lower Phase:
- Multiple techniques:
- Subdirectories: All lower phase software is contained in specially-designated subdirectories and is only activated by special compilation procedures.
- Interface adapters: Lower phase software is self-contained in new class files and accessed via polymorphism of abstract interface mechanisms or similar techniques that are not necessary for basic operation.
- Conditional compilation directives: discouraged in general.
- Externally-developed Packages:
- By default in Phase 1: Must go through same process.
- BLAS/LAPACK different: Certified as part of dependent-package process.

## Usage Status

- Lifecycle Model newly define: Gives us a target.
- 25% of package firmly in Phase 1.
- 75% of packages on the way to Phase 2.
- None are in Phase 3.
- Phase 3 is thus just speculation at this point.

## To Come

*Opportunities and Challenges*

## Themes for FY08/09

- Redefinition of Trilinos scope beyond solvers.
- Next steps in packaging and distribution.
- Continued outreach to other communities
- Rethinking source management.
- Formalizing App-Trilinos relationship.
- Post-delivery maintenance improvements.

## Scope of Trilinos

- Addition of Sacado, Zoltan, FEI, Intrepid, phdMesh: Not solvers.
- Framework support natural.
- Rephrasing of project goals, descriptions underway.
- Grouping of packages into meta-packages: At least conceptually.

## Packaging and Distribution

- Mac and Windows are ever more popular development environments.
- Goal: Provide click-install capabilities for Mac OS, MS Visual Studio, Linux COE.

## Outreach

- Trilinos packages part of SciDAC:
- ITAPS, CSCAPES, TOPS-2.
- Opportunity to serve broader DOE community.
- Trilinos popular in universities:
- Single largest sector of users.
- Trilinos part of several industrial efforts.
- Improves capabilities.
- Amortizes costs over broader funding sources.
- Elevates certain activities:
- Fortran accessibility.
- Packaging & distribution.

## Source Management

- Think of repository as a database.
- Logical collections gathered dynamically.
- Consider use of multiple source management tools:
- Local vs. global management.
- Fully distributed.
- Certainly svn is option, but looking at all options.

## App-Trilinos Relationship

- How should apps to interact with Trilinos?
- Many facets:
- Interfaces: concrete vs. abstract.
- Testing: How frequently, who does it.
- Distribution: Trilinos separate or integrated.
- ...
- Goal: Any two applications should differ only in:
- Essential needs due to formulation.
- Specific uses of enabling technologies.

## Post-delivery Maintenance and Support

- Long-term support for software that is:
- Heavily used but,
- Not actively developed,
-  is a chronic issue across the DOE complex.
- BLAS, LAPACK, Scalapack, Aztec...
- Organizational structures do not reflect overall cost of software:
- 60%-70% in post-delivery maintenance.
- We are research focused.
- Working on plan to use matrixed org.

## Take Home Messages

- We (the Trilinos team) are not experts, but committed to improvement.
- Regular assessment of opportunities helpful.
- Team sees past adoption successful, willing to try more.
- Quality matters.
- We want our software to be trusted and reliable.
- New practices and tools can help us.
- Open Source tool sets are truly impressive.
- More opportunities are possible.

## Trilinos Availability/Information

- Trilinos and related packages are available via LGPL.
- Current release (8.0) is “click release”.  Unlimited availability.
- Trilinos Awards:
-  2004 R&D 100 Award.
- SC2004 HPC Software Challenge Award.
- Sandia Team Employee Recognition Award.
- Lockheed-Martin Nova Award Nominee.
- More information:
- _[http:](http://software.sandia.gov/trilinos/rd100_2004.html)__[//trilinos](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[sandia](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[gov](http://software.sandia.gov/trilinos/rd100_2004.html)_
- _[http://software.](http://software.sandia.gov/trilinos/rd100_2004.html)__[sandia](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[gov](http://software.sandia.gov/trilinos/rd100_2004.html)_
- Additional documentation at my website:_[http://www.](http://software.sandia.gov/trilinos/rd100_2004.html)__[cs](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[sandia](http://software.sandia.gov/trilinos/rd100_2004.html)__[.](http://software.sandia.gov/trilinos/rd100_2004.html)__[gov/~mheroux](http://software.sandia.gov/trilinos/rd100_2004.html)_.
- 5th Annual Trilinos User Group Meeting: November 6-8, 2007 at 	CSRI/90 (This room)	Sandia National Laboratories, Albuquerque, NM, USA.