# Presentation: 083316

## To bug or not to bug:  Lessons learned developingbug-compatible software

- 22 June 2005
- Eric Keiter
- Dept. 9237: Electrical and Microsystems Modeling
- Sandia National Laboratories
- Albuquerque, NM, USA

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

## Outline

- Context
- What is circuit simulation?
- What is Xyce?
- What do you mean, “bug-compatible”?
- Lessons learned
- Development tools:  Bugzilla, Bonsai, CVS, autotools, etc.
- The release process

## Analog Circuit Simulation (SPICE or Xyce)

- System of Coupled Differential Algebraic Equations (DAEs)
- Kirchoff’s Laws, Arbitrary Network
- Formulation:  Modified Nodal Analysis. (modified KCL)
- Most equations are Kirchoff Current Law (KCL) equations.
- Most solution variables are nodal voltages.
- Most currents are obtained via an Ohm’s law relationship.
- Nonlinear system of equations:
- Implicit time integration
- Newton’s method for the nonlinear solve at each time step.
- Linear system of equations (Ax=b) solved at each Newton iteration.

**Kirchoff’s Current Law (KCL)**

**Kirchoff’s Voltage Law (KVL)**

## Simulation Hierarchy

- Analog circuit simulation is just one of many types of simulation used by electrical designers.
- Tradeoff between fidelity and speed/problem size.
- Digital (VHDL) simulation:  **very fast, but low fidelity**
- TCAD Device simulation:  **very slow, but very high fidelity.**
- Circuit simulation:  **in-between.**

- Speed

- Fidelity

**Digital (VHDL)**

- Analog
- (like SPICE)
***Xyce***

- TCAD Device
***Charon***

- Board and
- Device Parasitics

## History of analog circuit simulation (SPICE)

- SPICE = _S_imulation _P_rogram with _I_ntegrated _C_ircuit _E_mphasis.
- Developed at UC-Berkeley.
- 1969:  Class project, called CANCER = _C_omputer _A_nalysis of _N_onlinear _C_ircuits, _E_xcluding _R_adiation.  6000 lines of Fortran.
- 1971:  SPICE 1.  Released into public domain.
- 1975:  SPICE 2.  PhD thesis of Larry Nagal.  8000 lines of Fortran.
- 1989:  SPICE 3.  PhD thesis of Tom Quarles.  135,000 lines of C.
-  Xyce, excluding solver libraries, is ~400,000 lines of C++.  Progress!
- SPICE was popular because it was public domain, and it was quickly adopted by universities, and by industry.
- SPICE became an industry standard for analog circuit simulation.
- Around 1990, most circuit simulation innovation became commercialized, mostly with codes built on the SPICE3 engine.
- Pspice, Hspice, SmartSpice.  Sporty Spice, Scary Spice, Posh Spice
- More recently, EDA companies have been buying each other, and retiring a lot of SPICE-related products.

## More Miscellaneous Context

- 10 years ago, conventional wisdom: *iterative matrix solvers don’t work for circuits.  *wrong!
- In 2000, there were over 500 PSpice licenses at Sandia.
- Many SPICE models are terrible; part of the standard.
- Simplifications based on 1960’s computers.
- Discontinuities, bad derivatives.
- As SPICE is a design tool, a bad model can be OK.
- Sometimes, a designer just wants a generic behavior, rather than a precise one.
- “bug compatible” is expected.

## One Perspective on SPICE

- Q: If you could go back over your career and change something within your field, what would you change?
- Pease:  “I’d shoot the guys who were going to invent SPICE.”
- Robert A. Pease, ad insert from EDN “Movers and Shakers” 2003 supplement.

-  *Taken from a talk by Larry Nagal, the inventor of SPICE*

## Xyce Requirements

- Xyce is an ASC project, started in 1999.  Requirements:
- SPICE-compatible
- Massively parallel
- Include special Sandia physics models (ie radiation)
- Useful to unsophisticated users
- Another way to put it:
- “Be the same as SPICE, but also be much better”
*“**I never want to see the “Time step too small” error again**”* – Ken Marx, Sandia Electrical Analyst

## Xyce Status, circa 2005

- We’ve *mostly* succeeded:
- SPICE-compatible (sort of)
- Radiation models
- Demonstrated Parallel scalability on “real” problems (see chart)
- Real  customers using Xyce on real DSW problems.
*We still have a lot to do!*

**Photograph of Permafrost ASIC (PA)**

**PA Scaling study (~800,00 devices)**

## Largest Analog Circuit Calculation Ever Performed !

- XyceTM massively parallel circuit analysis code
- Nonlinear transmission line problem
- 14 million electrical devices modeled
- 6 million unknowns
- Over factor of 500 speedup using 1024 processors of LLNL’s ASCI White supercomputer
- 14,000 electrical devices per processor

## How did we do it?  Lessons learned

- Hire good people
- Get everyone on the same page.
- Difficult, but REALLY important!
- Open mind about legacy (ie SPICE) codes
- Support the low end as well as the high end
- Most Xyce simulations are not massive.
- Use development tools (within reason)
- Bugzilla, Bonsai, UML, etc.
- Regression testing:  nightly, weekly, monthy
- Rigorous requirement-tracking, issue-tracking, and bug-tracking.
- Formal release process

## Lesson:  Hire good people

- “Hire the best people you can and then get out of their way!” – Bill Camp, circa 1993.
- This may seem obvious, but in practice is difficult.
- Look for:
- Lots of programming experience, even if not SPICE (or whatever)
- Programs for fun, has done so for years. (maybe as a kid)
- Background in numerical programming
- Handholding a bad developer is (eventually) a waste of time.
- One great developer is worth 5 lousy developers.

## Lesson: Get everyone on the same page

- Note:  Make sure you *have *a page.
- Good people often disagree
- Good people, when they are new, are going to do “dumb” things.  This includes everyone, including the best people.  Don’t get upset - plan for it!
- Have an established code-development “business model”
- Interact with new developers a LOT.
- Correct misconceptions early.
- Have documentation ready:  developer guide, theory guide, etc.
- This is the most important issue.

## The Same Page: Coding standard/guidelines

- This is the first issue for Xyce where getting all the developers to agree was _difficult_.
- C++ or C-style comments?
- Philosophy of pointers and/or references?
- Curly bracket usage?
- Standard class and function comment headers.
- Should we use STL?  (ASC Red didn’t support it)
- Directory structure?
- The important thing was not so much WHAT we chose, but that we chose SOMETHING and STUCK TO IT.
- Many of these things are a matter of taste.
- In general, to achieve consensus, final choices were:
- Acceptably Imperfect
- Something everyone was willing to live with

## The Same Page:Plan for new developers in the code.

- Some parts of the code benefit a LOT from a very sophisticated structure
- Especially modules with just one developer (Xyce topology)
- However, be careful:
*Some parts of the code will be touched by many developers, with varying skill levels*
- Xyce Device module (>10 developers since 2000)
- Continual refactoring and abstracting in the multi-developer sections can be problematic
- In my initial development of the Xyce device package, part of that development included getting other developers working in it, giving me feedback.
- This included (among others) 2 undergrad student interns.

## Lesson:  Open mind about legacy code

- Initially, we thought all of SPICE was lousy
- Slamming it was popular (by users and developers)
- “I’d shoot the people who were going to invent SPICE”
- Much of it looked strange, “out of the mainstream”
- The last “free” version of Spice came out in the late 1980’s.
- Reality:  some of it *was* lousy, some of it very clever.
- Kundert Sparse direct solver
- Voltage limiting
- As the project has evolved, we’ve referred to the Spice3 source a lot.
- It is the standard we are attempting to emulate, after all.
- To an extent, we have to be “bug-compatible”!

## Prototype

- Make your initial mistakes in a code that you plan to throw away.
- Summer, 2000 - Xyce prototype was written.
- Some of the structure was retained for Xyce
- Some was thrown away.

## Developer tools

- Rational Rose - UML
- CVS/ Bonsai
- Bugzilla
- Regression testing
- Certification testing

## Xyce UML Diagrams

- We used Rational Rose for initial code generation.
- For a larger project, round-trip engineering may have been worth doing - for us, it wasn’t.
- One danger - spending months developing a UML diagram before attempting code generation - result:  un-compilable code.

## CVS/Bonsai

- Like with most codes, configuration management is very important.
- We use CVS/Bonsai.
- Hyperlinked to bugzilla.

## Bugzilla

- We use Bugzilla *extensively.*
- Over 700 issues since 2000.
- We use it for tracking all sorts of project planning - it is not just for user reported bugs.
- Most bugzilla issues are entered by developers
- A typical usage:
- In FY05, we’ve used it to track all the development neede for our Advanced Development (AD) project.
- One “blocker” issue for AD, that sits on top of a large number of specific issues. (like “develop the SOI device model”)
- Bugzilla plays a major, major role in our release process.

## Testing:  Regression and Certification

- Regression testing - last chance “safety net”
- Nightly tests
- Weekly tests
- 49 different builds
- Certification testing - used for release process
- Each test tied to a specific bugzilla issue
- Many certification tests ultimately become regression tests later, after a release.

## The release process

## The release process

- Planning:  identify features, and bugzilla issues for this release.
- QA: Every fixed issue *must* have a certification test to start QA, demonstrating the issue.
- The code in “QA” until every certification test, for every support platform, passes.
- If *any* test fails, we drop out of QA, fix code, *and restart QA, from the beginning*.
- Production builds: created if Xyce gets through QA w/o failure, *for all platforms*.
- Certification:  Signed by a manager.
- Regression:  Certification tests are migrated to regression tests, post-release.

## The release process: Planning

- The Planning phase happens first.
- Developer meeting.
- Decisions made:
- Which bugzilla issues will be fixed (addressed) for the release, and which ones will be deferred?
- Which issues are “deal killers”, and which ones are expendable?
- Who is assigned to each issue? (we usually know already, but sometimes assignments change)
- What date do we freeze/branch the code?
- For each issue, developers commit to produce a “certification test” that proves the issue is fixed and/or addressed.
- These certification tests are used in QA.
- Note:  Of course, some  issues can’t easily be tested - those issues simply get documented. (see Dilbert cartoon)

## The release process: Branching

- The Branching the code is crucial.
- On the freeze date, tag and branch to create a “release branch”.
- From that point onward:
- New features are only developed in the “development (main) branch”
- The release branch will be changed minimally, for small bugfixes.
- Later in the release process, we sometimes branch the code again, to plan for patch releases.
- After the release process is over, the branches are merged back together.

## The release process: QA

- The QA phase dominates the release process, in terms of time.
- Typically, we plan for 3 rounds of QA.
- In our last release, we needed 6 (too many!)
- We typically plan for each QA round to take 2 weeks.
- Once the code is in a round of QA, it should be *completely* frozen.
- If the code needs to be fixed (certification test fails):
- the current round of QA stops.
- the release branch code is fixed, and QA restarts, *from the beginning*.
- QA is complete only when the code can pass every test, without any code modifications.
- In general, the release branch code should change minimally over all the QA cycles.

## The release process: Post-QA

- Once we get through a round of QA with no certification test failures, we can move to the next phase(s).
- Code is branched *again*, to plan for possible “patch” release.
- Production builds
- Documentation:  Users/Reference guides, Release notes.
- Certification:  This includes setting up and filing paperwork, which requires getting signatures from:
- Manager
- HPEMS PI
- Technical lead
- Testing team lead
- Announcement/Distribution
- Release branch is merged back into the development branch.
- If the release process has taken a long time (months), this is a big job.
- Certification tests are migrated to regression tests, within reason.

## The End