From: ron@vicorp.com (Ron Peterson)
Subject: Automated testing
Organization: V. I. Corporation, Northampton, Massachusetts
Date: Mon, 31 Aug 1992 19:37:40 GMT

Here's an addendum to Vonn Marsch's summary of responses about
automated testing of X applications.  I asked a similar question and
received somewhat different replies.  In particular, it was quite
interesting to learn a little about how others go about testing.
(Some use college students, some use regression, some develop
their own tools, some purchase tools.)
If you have experience testing medium to large graphic applications
please send me a note describing what you worked on and how you
approached it.  If I get enough of them I'll post another summary.

Here's a description of my experience to get things started:

   VI Corp. markets a graphic interface prototyper called DataViews.
DataViews is built around 'views' which are 2D collections of graphical
objects (such as lines, rectangles, graphs, input widgets, images, etc.)
The product consists primarily of DVdraw (a layout and drawing editor which
also lets you define dynamics for graphical objects and define programming
rules for the way objects and views change and behave) and DVtools (a
library of graphics routines which users can use to control and modify
views, objects, event-handlers, etc.)  Other related products are
starting to come on line also such as DVXDesigner---an X widget layout
editor and code generator that allows DataViews views to display in
a drawing area widget.
   I've been with the company for about three years in the testing
department and am the primary designer and coder of new tests.
When I first arrived here the testing philosophy was "the developers
are done with it, now you test it before we ship it in a few weeks."
As the company has grown things have changed to where we currently
view testing as part of the design process for the products.  Hence, 
I am investigating trying to automate some of our testing.  Towards 
that end I've tried writing a few things in-house such as an event 
recorder/player and a view tester.  The event-recorder/player was a 
failure because it took longer to record the tests than to do them 
manually and since the user interface kept changing I had to keep 
starting over to record the tests.  Eventually we ran out of time 
and reverted to manual testing.  The view tester is currently being 
used and with a little work looks like it will be a valuable tool 
for finding memory leaks, run-time errors, and crashes.  My current 
thinking is that an X based event recorder/playback tool combined 
with a bitmap comparer (that uses compressed bitmaps) might be made 
independent of the user interface as well as being easy to port to 
the 30+ Unix and VMS machines we support.  By using X and being 
careful to control window size and fonts, a single automated test 
suite might be devised that could run on most or all platforms.
  Currently all of our testing tools have been developed in-house and
many rely on the prototype building ability of DataViews which allows
us to create interactive tests without writing code.  Some code has
been written to test drivers on a function by function basis, which has
proved valuable, but there has never been enough time to write test code 
for every function in the library (and I see less benefit in such tests
than I used to.)  Demo and example code also form a critical part of
our testing process since they are designed to be similar to applications 
our customers would write.  Still, much of testing involves sitting
and staring at the screen while clicking and moving a mouse.  This
is the most time consuming and dull part of testing and is the part
we'd like most to automate.

ron@vicorp.com or uunet!vicorp!ron


-----------------------------------------------------------------------------
>From: uunet!cd.amdahl.com!wdc50 (Winthrop D Chan)
>For PD, take a look at TCL, GCT, and EXPECT.The last two are based on TCL. I've
>heard good things they are all supposed to be able to handle keyboard & mouse
>event recording/playback on multiple windows.

I downloaded a copy of TCL.  So far I haven't seen anything about it being
able to handle keyboard and mouse events.  If it does handle them, then
I get the impression you have to link some TCL code into your executable;
sounds a lot like REXX.  GCT has nothing to do with TCL (an extensive
note from the creator of GCT is included below.)
Here's an excerpt from the TCL README:

>Tcl
>
>by John Ousterhout
>University of California at Berkeley
>
>This directory contains the sources for Tcl, an embeddable tool command
>language.  For an introduction to the facilities provided by Tcl, see
>the paper ``Tcl:  An Embeddable Command Language'', in the Proceedings
>of the 1990 Winter USENIX Conference.


I also found a copy of EXPECT which is indeed based on TCL:


>This is the README file from "expect", a program that performs
>programmed dialogue with other interactive programs.  It is briefly
>described by its man page, expect(1).  More examples and further
>discussion about implementation, philosophy, and design are in
>"expect: Curing Those Uncontrollable Fits of Interaction" by Don
>Libes, Proceedings of the Summer 1990 USENIX Conference, Anaheim,
>California, June 11-15, 1990.
>
>expect was designed and written by Don Libes, January - April, 1990.

>NAME
>     expect - programmed dialogue with interactive programs
>
>SYNOPSIS
>     expect [ -d ] [ -c cmds ] [[ -f ] cmdfile ] [ args ]
>
>INTRODUCTION
>     expect is a program that "talks" to other  interactive  pro-
>     grams  according  to a script.  Following the script, expect
>     knows what can be expected  from  a  program  and  what  the
>     correct  response  should  be.  An interpreted language pro-
>     vides branching and high-level control structures to  direct
>     the  dialogue.   In  addition, the user can take control and
>     interact directly when desired, afterward returning  control
>     to the script.
>
>     The  name  "expect"  comes  from  the  idea  of  send/expect
>     sequences  popularized  by uucp, kermit and other modem con-
>     trol programs.  However unlike uucp, expect  is  generalized
>     so  that it can be run as a user-level command with any pro-
>     gram and task in mind.  (expect can actually talk to several
>     programs at the same time.)
>
>     For example, here are some things expect can do:
>
>          +   Cause your computer to dial you back, so  that  you
>              can login without paying for the call.
>
>          +   Start a game (e.g., rogue) and if the optimal  con-
>              figuration  doesn't  appear,  restart it (again and
>              again) until it does, then  hand  over  control  to
>              you.
>
>          +   Run fsck, and in response to its questions,  answer
>              "yes",  "no"  or give control back to you, based on
>              predetermined criteria.
>
>          +   Connect to another network or BBS (e.g., MCI  Mail,
>              CompuServe) and automatically retrieve your mail so
>              that it appears as if it  was  originally  sent  to
>              your local system.

-----------------------------------------------------------------------------
From: uunet!euclid.JPL.NASA.GOV!pjs (Peter J. Scott)

If you have Motif 1.2 then it has a QA scripting language for
this purpose.  Even compares windows.  Only Motif apps, though.
-----------------------------------------------------------------------------
From: uunet!cbnea.att.com!tr (Aaron L Hoffmeyer)

I encourage you to contact Dr. Bill Wolters at w.j.wolters@att.com

Bill is working on a front-end to a commercial application called CAPBAK,
which in raw form, is, to a limited degree, the kind of tool you are
looking for.  However, as Bill will be able to tell you, one must spend
a great deal of effort in making it a solid, useable tool.  Bill can
give you specs, but, considering that he has worked with this tool for a
year and it will probably take another year to make it the end-all,
be-all tool that he would like, I feel I must warn you that purchasing
such a tool and putting it to use, are not one and the same.

Bill's really leading edge with this technology, he has presented at
some conference, maybe USENIX, quite recently, so he knows where the
market is.
[I investigated CAPBAK about two years ago.  We decided not to use it
because it was more than we could afford at the time (being a small
company) and it wasn't portable to the 20+ Unix and VMS platforms
our product ran on.  It is a tool for capturing mouse and keyboard
events and then replaying them.  It was designed to be used in
conjunction with some other testing tools written by the same
company which, I think, is Software Research, Inc.]
-----------------------------------------------------------------------------
From: GAMBLE <uunet!heifetz!image!dos!gamble>

 We have looked here for automated tools, and I can pass on what
 little we have learned.

 * We do not like the 'total solution' packages. Instead we use a
 varity of small tools and utilities.
 * Software Management News magazine is a good source for tools.
     SMN (415) 969-5522
 * We use tools from Software Research (Unix) - expensive but ok. has
 graphic compare capabilities
 * C-Cover for code coverage analysis - not good for huge programs
 * Microsoft Test for Windows applications - nice easy to use

 We have an extensive list of testcases and usually hire college
 co-ops to run the tests. It provides them with valuable experience,
 and we get the testing done.

 For automation, I have had better luck using keyboard cursor capture
 rather than mouse capture. Our graphic screens change resolution from
 different graphic boards so mouse movement is not always accurate. We
 have so far been able to save our graphic images as files, and use a
 file compare program with more success. We use ExDiff from Software
 Research that allows us to use byte offsets to eliminate time and
 date tags, or version numbers imbedded in our graphic files.

 Mostly be cautious of software that says they are the total
 solution. Use many small useful utilities, and keep track of things
 by using a bugs database. (We wrote our own using DBase IV for Unix)
-----------------------------------------------------------------------------
From: Jim Curran <uunet!cis.ohio-state.edu!jim>

XTest and XTrap are extensions to the X server.  Extensions provide additional
capabilities over and above what is provided by a "vanilla" server.  It is
unfortunate, but in order to use these extensions (and others) you must
rebuild the X server.  This may not be a hassle for your X developer but
it does take a little extra time and patience.  Once the extension is installed(you can list your extensions by running xdpyinfo) you can use calls to the
extension library (libXext.a) and it will talk to the server's extensions
for you.

XTest and XTrap are (as I recall) copyrighted by HP and Dec, respectively.
However, the copyright is one of those "It belongs to us but you can
do anything you want with it" types of copyrights.  In other words, you
can have it.

As I mentioned, XTrap is on export.lcs.mit.edu.  Here's the confusing part
to me:  XTest seems to have come out around the time that R4 appeared and seemsto have been part of the official MIT dist (I could be wrong).  I have not
encountered the XTest extension by itself EXCEPT when I snagged the package
called xrecplay from export.  It contains the extension and some special
stuff for Suns.  I don't really feel like grabbing the R4 dist and
looking around for XTest but my guess is that it is there.  If it is not,
then you can get it from xrecplay and beat it into usefulness.

I've spent the last few weeks beating XTest into submission for my purposes
(a program which does demos - moves the mouse, keyboard hits, voice etc...)
and I can say that it more or less works.  I looked at XTrap and it was very
elaborate and came with a better manual but it added *lots* to our X server
which was something I couldn't have. (XTest ~20K, XTrap ~200K).
-----------------------------------------------------------------------------
From: Brian Marick <uunet!cs.uiuc.edu!marick>

Capture/playback tools and graphics image "differencers": there are
several out there.  Two I have in my tools file are from Software
Research (800-942-7638) and Mercury (408) 982-0100.  I haven't used
either.  DEC also has one, which may be of interest for VMS.
fullerton@clt.enet.dec.com may be the person to talk to.

You may have problems with portability, not to mention price, if
you're wanting to use these on several machines.

I don't believe any of the commercial tools do anything more than
bitmap comparisons, which will cause problems with spurious
differences and storing huge bitmaps (Mercury compresses the bitmaps,
I think, but that's still a lot of data per test).

---
Automatic test generation does not work well, except in specialized
circumstances.

If you have a large testing problem like this, don't worry about path
coverage.  Worry about writing good tests at the system or subsystem
level, based on what the program is supposed to do, what the users are
likely to do, and potential implementation errors.  Measure coverage
later, to point out problems with your testing.  (I distribute a free
coverage tool that runs on most of the machines you mentioned, but not
on VMS - details below.)  I can send you a paper on "three ways to
improve your testing", either postscript or hardcopy.  It talks about
some sensible tactics in situations like yours.

If you want lots of detail about testing, I teach a course in testing
subsystems and I make my living consulting in testing.  (217)351-7228
--

One upcoming conference is the Pacific Northwest Software Quality
Conference, Oct 19-21, in Portland.  Call Pacific Agenda (503)
223-8633 for details.  It was pretty good last year.

IEEE Software has had some good practitioner-oriented articles about
testing in recent years.

I run a mailing list on testing research
(testing-research-request@cs.uiuc.edu).  It's pretty dead, unfortunately.

Brian Marick, marick@cs.uiuc.edu, uiucdcs!marick, marick@testing.com (pending)
Testing Foundations:  Consulting, Training, Tools.
Freeware test coverage tool:  see cs.uiuc.edu:pub/testing/GCT.README

====
GCT is a freely-distributable coverage tool for C programs, based on
the GNU C compiler.  Coverage tools measure how thoroughly a test
suite exercises a program.  I tested without a coverage tool for
years, but I've used GCT and its predecessor for about three years now.
I'll never go back to testing without coverage.

GCT provides coverage measures of interest to both practitioners and
testing researchers:

(1) branch coverage.  Do the tests force every branch in both
directions?  Has every case in a switch been taken?

(2)  multiple-condition coverage.  In a statement like
        if (A ||  B)
has A been both true and false?  Has B?  Multiple-condition coverage
is stronger than branch coverage.

(3) loop coverage.  Has every loop been skipped?  Has every loop been
iterated exactly once?  More than once?  Tests to satisfy loop
coverage can force the detection of some bugs that branch coverage
might miss.

(4) relational operator coverage.  Do tests probe for off-by-one
errors?

(5) routine and call coverage.  Has every routine been entered?  Has
every function call in a routine been exercised?  These two measures
are weaker than the above, but they are useful for getting a general
picture of the coverage of system testing.

(6) race coverage.  This special-purpose measure is useful for
evaluating stress tests of multi-threaded systems, tests designed to
find locking and concurrency problems.

(7) weak mutation coverage.  This is a type of coverage of most
interest to researchers.

GCT has been used in large C development projects.  It is
"industrial-strength".

Platforms

GCT runs on many versions of UNIX; porting it to new ones is usually
simple.  An earlier version ran on HP/Apollo's Aegis, so the current
version should be easy to port.

Mechanism

GCT takes C source code and adds instrumentation.  This new C source
is then compiled as usual.  This allows you to use your favorite C
compiler. GCT is designed to work with existing makefiles (or other
system build mechanisms).  For example, instrumenting the entire
System V Release 4 UNIX kernel requires changes to only the top-level
makefile.

As the compiled program runs, it updates an internal log.  The log is
written on exit (or, in the case of an unending program like the UNIX
kernel, extracted from the running program).  Other tools produce
coverage reports (detailed or summary) from the log.

Instrumentation is time-efficient, a few instructions per
instrumentation point.

Retrieving GCT

GCT is available via anonymous FTP from cs.uiuc.edu.  Log in as user
"anonymous", and give your email address as the password.  The file
/pub/testing/GCT.README describes the GCT distribution.

GCT and its documentation are also available on 8mm and QIC tapes for
$150.

After retrieving and installing GCT, be sure to run the tutorial.  See
the document 'A Tutorial Introduction to GCT'.

Support and Training

While GCT is free, the services you expect for a commercial product
are available for a price.  Contact me for information about support
and training, or for information about other services I offer.
Informal GCT support is available through the GCT mailing list; send
mail to gct-request@ernie.cs.uiuc.edu to join it.

Brian Marick
Testing Foundations
809 Balboa, Champaign, IL  61820
(217) 351-7228
marick@cs.uiuc.edu
----------------------------------------------------------------------------

From: Brian Marick <uunet!cs.uiuc.edu!marick>

This may also be of interest.  It describes what I do when I test my
own software, a smallish addition - maybe 10,000 lines - embedded
inside a much larger C compiler.

                   A CASE STUDY IN COVERAGE TESTING
                             Brian Marick
                         Testing Foundations

Abstract

I used a C coverage tool to measure the quality of its own test suite.
I wrote new tests to improve the coverage of a 2600-line segment of
the tool.  I then reused and extended those tests for the next
release, which included a complete rewrite of that segment.  The
experience reinforced my beliefs about coverage-based testing:

1.  A thorough test suite should achieve nearly 100% feasible coverage.
2.  Adding tests for additional coverage can be cheap and effective.
3.  To be effective, testing should not be a blind attempt to achieve
    coverage.  Instead, use coverage as a signal that points to weakly-tested
    parts of the specification.
4.  In addition to suggesting new tests, coverage also tells you when existing
    tests aren't doing what you think, a common problem.
5.  Coverage beyond branch coverage is worthwhile.
6.  Even with thorough testing, expect documentation, directed
    inspections, beta testing, and customers to find bugs, especially design
    and specification bugs.

The Generic Coverage Tool

GCT is a freeware coverage tool for C programs, based on the GNU C
compiler.   It measures these kinds of coverage:
- branch coverage (every branch must be taken in both directions)
- multi-condition coverage (in 'if (a && b)', both subexpressions must
  evaluate to true and false).
- loop coverage (require loop not to be taken, to be traversed exactly once,
  and traversed more than once)
- relational coverage (require tests for off-by-one errors)
- routine entry and call point coverage.
- race coverage (extension to routine coverage for multiprocessing)
- weak mutation coverage (a research technique)

(For more, see [Marick92].)

The tool comes with a large regression test suite, developed in
parallel with the code, using a "design a little, test a little, code
a little" approach, much like that described in [Rettig91].  About
half the original development time was spent in test construction (with, I
believe, a corresponding reduction in the amount of frantic debugging
when problems were found by users - though of course there was some of
that).  Most of the tests are targetted to particular subsystems, but
they are not unit tests.  That is, the tests invoke GCT and deduce
subsystem correctness by examining GCT's output.  Only a few routines
are tested in isolation using stubs - that's usually too expensive.  When
needed, test support code was built into GCT to expose its internal
state.

In early releases, I had not measured the coverage of GCT's own test
suite.  However, in planning the 1.3 release, I decided to replace the
instrumentation module with two parallel versions.  The original
module was to be retained for researchers; commercial users would use
a different module that wouldn't provide weak mutation coverage but
would be superior in other ways.  Before redoing the implementation, I
wanted the test suite to be solid, because I knew a good test suite
would save implementation time.

Measuring Coverage

I used branch, loop, multi-condition, and relational coverage.  I'm
not convinced weak mutation coverage is cost-effective.  Here were my
initial results for the 2617 lines of code I planned to replace.
(The count excludes comments, blank lines, and lines including only
braces.)

BINARY BRANCH INSTRUMENTATION (402 conditions total)
47 (11.69%) not satisfied.
355 (88.31%) fully satisfied.

SWITCH INSTRUMENTATION (90 conditions total)
14 (15.56%) not satisfied.
76 (84.44%) fully satisfied.

LOOP INSTRUMENTATION (24 conditions total)
5 (20.83%) not satisfied.
19 (79.17%) fully satisfied.

MULTIPLE CONDITION INSTRUMENTATION (390 conditions total)
56 (14.36%) not satisfied.
334 (85.64%) fully satisfied.

OPERATOR INSTRUMENTATION (45 conditions total)  ;; This is relational coverage
7 (15.56%) not satisfied.
38 (84.44%) fully satisfied.

SUMMARY OF ALL CONDITION TYPES (951 total)
129 (13.56%) not satisfied.
822 (86.44%) fully satisfied.
These coverage numbers are consistent with what I've seen using black
box unit testing combined with judicious peeks into the code.  (See
[Marick91].)  I do not target coverage in my test design; it's more
important to concentrate on the specification, since many important
faults will be due to omitted code [Glass81].

When the uncovered conditions were examined more closely (which took
less than an hour), it was clear that the tests were more thorough
than appears from the above.  The 129 uncovered conditions broke down
as follows:

28 were impossible to satisfy (sanity checks, loops with fixed bounds
can't be executed 0 times, and so on).

46 were support code for a feature that was never implemented (because
it turned out not to be worthwhile); these were also impossible to
exercise.

17 were from temporary code, inserted to work around excessive stack
growth on embedded systems.  It was always expected to be removed, so
was not tested.

24 were due to a major feature, added late, that had never had
regression tests written for it.

14 conditions corresponded to 10 untested minor features.

All in all, the test suite had been pleasingly thorough.

New Tests Prior to the Rewrite

I spent 4 hours adding tests for the untested major feature.  I was
careful not to concentrate on merely achieving coverage, but rather on
designing tests based on what the program was supposed to do.  Coverage
is seductive - like all metrics, it is only an approximation of what's
important.  When "making the numbers" becomes the prime focus,
they're often achieved at the expense of what they're supposed to measure.

This strategy paid off.  I found a bug in handling switches within
macros.  A test designed solely to achieve coverage would likely have
missed the bug.  (That is, the uncovered conditions could have
been satisfied by an easy - but inadequate - test.)

There was another benefit.  Experience writing these tests clarified
design changes I'd planned to make anyway.  Writing tests often has
this effect.  That's why it's good to design tests (and write user
documentation) as early as possible.

I spent two more hours testing the minor features.  I did not write
tests for features that were only relevant to weak mutation.

Branch coverage discovered one pseudo-bug: dead code.  A particular
special case check was incorrect.  It was testing a variable against
the wrong constant.  This check could never be true, so the special
case code was never executed.  However, the special case code turned
out to have the same effect as the normal code, so it was removed.
(This fault was most likely introduced during earlier maintenance.)

At this point, tests written because of multi-condition, loop, and relational
coverage revealed no bugs.  My intuitive feel was that the tests
were not useless - they checked situations that might well have led to
failures, but didn't.

I reran the complete test suite overnight and rechecked coverage the
next day.  One test error was discovered; a typo caused the test to
miss checking what it was designed to test.  Rechecking took 1/2 hour.

Reusing the Test Suite

The rewrite of the instrumentation module was primarily a
re-implementation of the same specification.  All of the test suite
could be reused, and there were few new features that required new
tests.  (I did go ahead and write tests for the weak mutation features
I'd earlier ignored.)  About 20% of the development time was spent on
the test suite (including new tests, revisions to existing tests, and
a major reorganization of the suite's directory structure and
controlling scripts).

The regression test suite found minor coding errors; they always do,
in a major revision like this.  It found no design flaws.  Rewriting
the internal documentation (code headers) did.  (After I finish code,
I go back and revise all the internal documentation.  The shift in
focus from producing code to explaining it to an imaginary audience
invariably suggests improvements, usually simplifications.  Since I'm
a one-man company, I don't have the luxury of team code reads.)

The revised test suite achieved 96.47% coverage.  Of 37 unsatisfied
conditions:

27 were impossible to satisfy.
2 were impossible to test portably (GNU C extensions).
2 were real (though minor) omissions.
1 was due to a test that had been misplaced in the reorganization.
5 were IF tests that had been made redundant by the rewrite. They were removed.

It took an hour to analyse the coverage results and write the needed
tests.  They found no bugs.  Measuring the coverage for the augmented
test suite revealed that I'd neglected to add one test file to the
test suite's controlling script.

Other Tests

The 1.3 release also had other features, which were duly tested.  For
one feature, relational operator coverage forced the discovery of a
bug.  A coverage condition was impossible to satisfy because the code
was wrong.  I've found that loop, multi-condition, and relational
operator coverage are cheap to satisfy, once you've satisfied branch
coverage.  This bug was severe enough that it alone justified the time
I spent on coverage beyond branch.

Impossible conditions due to bugs happen often enough that I believe
goals like "85% coverage" are a mistake.  The problem with such goals
is that you don't look at the remaining 15%, deciding, without
evidence, that they're either impossible or not worth satisfying. It's
better - and not much more expensive - to decide each case on its
merits.

What Testing Missed

Three bugs were discovered during beta testing, one after release (so
far).  I'll go into some detail, because they nicely illustrate the
types of bugs that testing tends to miss.

The first bug was a high level design omission.  No testing technique
would force its discovery. ("Make sure you instrument routines with a
variable number of arguments, compile them with the GNU C
compiler, and do that on a machine where GCC uses its own copy of
<varargs.h>.")  This is exactly the sort of bug that beta testing is
all about.

Fixing the bug required moderately extensive changes and additions,
always dangerous just before a release.  Sure enough, the fix
contained two bugs of its own (perhaps because I rushed to meet a
self-imposed deadline).

- The first was a minor design omission.  Some helpful code was added
to warn GCC users iff they need to worry about <varargs.h>. This code
made an assumption that was violated in one case.  Coverage would not
force a test to detect this bug, which is of the sort that's fixed by
changing

        if (A && B)

to

        if (A && B && C)

It would have been nice if GCT would have told me that "condition C,
which you should have but don't, was never false", but this is more
than a coverage tool can reasonably be expected to do.  I found the
bug by augmenting the acceptance test suite, which consists of
instrumenting and running several large "real" programs.  (GCT's test
suite contains mostly small programs.)  Instrumenting a new real
program did the trick.

- As part of the original fix, a particular manifest constant had to
be replaced by another in some cases.  I missed one of the cases.  The
result was that a few too few bytes of memory were allocated for a
buffer and later code could write past the end.  Existing tests did
indeed force the bytes to be written past the end; however, this
didn't cause a failure on my development machine (because the memory
allocator rounds up).  It did cause a failure on a different machine.
Memory allocation bugs, like this one and the next, often slip past
testing.

The final bug was a classic: freeing memory that was not supposed to
be freed.  None of the tests caused the memory to be reused after
freeing, but a real program did.  I can envision an implementable type
of coverage that would force detection of this bug, but it seems as
though a code-read checklist ought to be better.  I use such a
checklist, but I still missed the bug.


References

[Rettig91]      Marc Rettig, "Practical Programmer:  Testing Made
        Palatable", CACM, May, 1991.

[Marick91]      Brian Marick, "Experience with the Cost of Different
        Coverage Goals for Testing", Pacific Northwest Software Quality
        Conference, 1991.

[Marick92]      Brian Marick, "A Tutorial Introduction to GCT",
        "Generic Coverage Tool (GCT) User's Guide", "GCT Troubleshooting",
        "Using Race Coverage with GCT", and "Using Weak Mutation Coverage with G
CT",
        Testing Foundations, 1992.

[Glass81]       Robert L. Glass. "Persistent Software Errors", Transactions
        on Software Engineering", vol. SE-7, No. 2, pp. 162-168,
        March, 1981.

Brian Marick, marick@cs.uiuc.edu, uiucdcs!marick
Testing Foundations:  Consulting, Training, Tools.

-------------------------------------------------------------------------
From: uunet!veritas.com!joshua (Joshua Levy)

You may be interested in ViSTA, a product made by my company.
You can send email to our sales people at vsales@veritas.com, or
call our east coast sales person, Jerry DeBaun, at 508-624-7758.

ViSTA is a tool for gathering coverage metrics on your code as you test it.
It can tell you how much of your code is being tested by your test suite.
This can be valuable in determining how much testing still needs to be done.
ViSTA also can tell you exactly which pieces of code have been tested, so
you can write code to specifically test those parts which are not currently
tested.

We have found that using ViSTA significantly reduces the amount of time
it takes to test code.  Without coverage data, most test suites repeatedly
test the same code over and over again, wasting most of their time (both
development and execution).  The data ViSTA provides can direct testing
in a way which greatly reduces this waste.

ViSTA works on C code (Application level or Kernel level) on these platforms:
SunOS 4.1.x, AIX 3.x, HP 700s and 800s.  We also support several "custom"
ports including SVR3, SVR4, SCO UNIX, Pyramid, etc.  Availability for
these ports varies.

Please give me a call or send email if you have any questions, or if
I have not clearly explained what ViSTA does.  Thanks.

Joshua Levy  (joshua@veritas.com)   408-727-1222x253 or 800-if-vista

-------------------------------------------------------------------------
From: uunet!veritas.com!paula (Paula Phelan)

Thanks for your interest in ViSTA.  I will send along marketing literature
this afternoon.  If you like what you see give Jerry DeBaun a call
at 508-624-7758 he is the Regional Sales Manager in your area.

Paula




HIGHLIGHTS
        Code coverage testing tool for developers and testers
        Static and Dynamic Analysis of new or existing code
        McCabe and Halstead Metrics
        Error Seeding (to test testcase quality)
        Selective Instrumentation
        Works on multi-threaded code
        Multi-processor development support
        Application simulation of: OS calls, I/O errors, hardware states
        Kernel simulation of: Process credentials, DKI/DDI, I/O errors
        Ability to customize simulations

OVERVIEW DESCRIPTION of VERITAS ViSTA

VERITAS Vista provides metric-based testing tools for application and
kernel code.  The tools inform developers, testers, and managers about test
coverage and code maintainability.

ViSTA reports on source code static metrics; estimates the number of test cases
needed, determines the complexity of the code, and estimates the number of
errors. ViSTA dynamic metrics identifies code segments missed by existing test
cases and highlights redundant coverage.  All metric data can be displayed
graphically for at-a-glance comprehension.

Error seeding is available to test the quality of test cases by verifying if
they detect the errors they're intended to find.

ViSTA utilities can simulate event-driven conditions to improve code coverage.
This technology allows for testing hard to reach code or where replicating
the physical I/O resource would be prohibitively expensive.


DESCRIPTION of VERITAS ViSTA Modules

VistaTEST
The foundation of VERITAS ViSTA is VistaTEST.  This facility provides the
basic capabilities for code coverage, complexity metrics, error seeding and
report generation.

VistaTEST supplies development teams with the basic tools to immediately
improve the detection and elimination of latent defects, increases the
insight and understanding of existing code, and reduces the cost of software
testing.

VistaTEST defines what needs to be tested, estimates the cost of testing and
the level of effort required.  An advanced preprocessor that works with
ANSI C or K&R C is provided, and is platform independent.  VistaTEST also
provides coverage measurements by determining what sections of the code are
actually exercised by testing programs.  It provides coverage of function
entry/exits, segments, conditionals, basic blocks, paths and interface
behaviors.  Metrics are collected to determine the complexity of the code,
level of coverage with existing tests and the number of tests needed for
adequate coverage.  Run-time monitors support coverage statistics for library,
application, daemon, and kernel code.

VistaSIM
VistaSIM offers state of the art software validation for application and kerneldevelopers, by providing error handling logic, simulating behaviors at
interface points and creating a more complete test scenario.  It simulates all
external behaviors at function interface proints including function return
values, argument return values, and global variable values.  The UNIX
operating system calls are included and other simulation libraries may be
extended by the user.

ORDERING INFORMATION
Contact VERITAS sales at 408-727-1222 or vsales@veritas.com, VistaTEST list
price $1800.  Quantity pricing is available on request.

TRADEMARK ATTRIBUTES
VERITAS ViSTA, VistaTEST, VistaSIM and VistaKERNEL are trademarks of VERITAS
Software Corporation.

--
Paula Phelan                                                    VERITAS Software
4800 Great America Parkway #420   Santa Clara, CA 95054     +1 408-727-1222 x241
paula@veritas.com                               ...{apple,pyramid}!veritas!paula

---------------------------------------------------------------------------

Newsgroups: comp.windows.x.apps