From: marick@m.cs.uiuc.edu (Brian Marick)
Subject: Re: Testing Complex Systems
Organization: University of Illinois, Dept. of Comp. Sci., Urbana, IL
Date: Tue, 1 Sep 1992 19:59:25 GMT


If you want to test for locking bugs (deadlocks, races, etc.) in an
existing system, here is an effective procedure that's reasonably
cheap.  Other people's comments about attacking the problem at the
design level of course also apply.  Especially important is adding
testing support to the system.

1.  Enlist the active support of a senior developer, someone who knows
the system well and helped develop it.  Have them tell you the areas
of the system that are likely to be buggy.  If you have a bug history,
use that too.  You are looking for chunks of code to target.

2.  You will need a stress driver, one that can run many copies of a
particular test in parallel.  Spend some effort designing a prototype.
Don't get so wrapped up in that that you never get around to writing
tests.  Your testing experience will certainly cause major revisions
to the driver, no matter how good a design you think you have.

3.  Find a workload that roughly approximates "normal use" of the
system.  It doesn't have to match normal use exactly - even if such a
thing were possible - but it should exercise all the major parts of
the system.  For UNIX, MUSBUS/KENBUS aren't bad, though they have some
holes.

4.  Take your list of shakiest system components.  Pick the top few.
Test them in order of severity:  in which ones will a crash or hang do
the most harm to a customer?

5.  If you test your system through use (easy to do with the UNIX
kernel, harder with embedded systems), winnow the list by measuring
what normal use does (see below).  Don't write special tests that
present the same system load as 15 programmers writing code.

6.  For each component, write a test that exercises it.  You are most
concerned with the main paths through the component.  If you are not
experienced in the use and internals of the system, the senior
developer will be invaluable in helping you write sensible tests (that
exercise the component well).  

7.  Run N copies of the test alongside the background workload.
The tests target the component you chose; the background workload will
help find bugs that are interactions between that component and the
rest of the system.   Be careful not to make N so large that the
system thrashes; this concentrates the stress in too-small an area,
often not the area you're concerned with, and tends to find
"unrealistic" bugs (though not so much as you'd think).

8.  Measure what your tests do.  For this type of testing, a good
approximate measure is how often routines in the component of interest
are entered while another processor/thread is already in them.  (See
my .signature for a tool that measures this.)  Don't be surprised when
your tests don't stress what you think they're stressing, when they
get bottlenecked in some other part of the system.  You can also use
coverage to tune the resources you devote to a test.

Brian Marick, marick@cs.uiuc.edu, uiucdcs!marick, marick@testing.com (pending)
Testing Foundations:  Consulting, Training, Tools.
Freeware test coverage tool:  see cs.uiuc.edu:pub/testing/GCT.README
Newsgroups: comp.software-eng