From: ghm@sserve.cc.adfa.oz.au (Geoff Miller)
Subject: Re: Testing (Was: Do Software Engineers Actualy Do So Badly?)
Organization: Australian Defence Force Academy, Canberra, Australia
Date: Tue, 13 Oct 1992 02:58:49 GMT


First, a general comment  -  one thing which Tom's article does
highlight is how our attitudes and approaches are influenced by the 
kind of work we do.  My own backgound is largely in the development
of database applications (at least, most recently) so it is perhaps
not surprising that I see some things differently.

wicklund@intellistor.com (Tom Wicklund) writes:

>Let me throw out a few thoughts on testing.

>Of course, a purist might say that software needs to be proved, or
>with proper design test isn't important, but the fact is that in most
>environments testing is the most effective way to discover problems,
>even when setup in the usual rather haphazard fashion.

Purists work in Computer Science Departments with minimal connection
with the real world.  (Not everyone in CS departments fits this
description, of course:-)

>One of the differences between software and other disciplines is how
>testing is done.

>If you build something physical, each component is generally tested.
>Either on receipt, or by the manufacturer in some manner.  In
>electronic products early on all parts may be tested, then those with
>proven reliability may be spot tested or one may buy pre-tested parts
>from the manufacturer.  It's the exception to put together a mainframe
>computer (or larger) then turn it on without any intermediate test.

But surely this is the advantage of re-usable code, software libraries
and so on.  Not only do they speed up development, but they enable you
to construct a system largely from tested components whose behaviour is
known.  You can then concentrate on testing how those components interact.

This, of course, is not trivial.  People talk about "top-down
development" and "top-down testing", using dummy routines at
the lower levels to test the functioning of the higher levels,
but this often ignores the way in which that functioning depends
on the values returned from subroutines.  By the time you have
developed a dummy which returns the full range of possible values
you could often have written the real thing.

>Yet this is how software is often developed.

>I see two general reasons for this lack of test (other than the fact
>that the developer finds test cases boring).  Note I'm talking about
>component test right now, not final product test (which should have a
>separate test suite).

>1.  It's hard to test individual components.  I work with software
>attached to physical devices (disk drives, host computers).  It's very
>hard to test a single unit of code because the device needs too much
>of the outside world attached.  And it's very hard to simulate this
>because the timing can be extremely complex.  Physical devices also
>like to do things which you never would have expected (neither do the
>developers of those devices in many cases).

This sort of behaviour is not restricted to physical devices!!!  One 
of the most difficult tasks I have to do is to break young programmers
of the idea that users will use systems in the way that they should.
Users will return null values when they shouldn't, will try to key
ahead, will try to select non-existent menu options, etc., etc., etc.
Certainly testing systems in my environment is somewhat easier, in that
I can simulate the things that a user might do to the system without
needing to break a physical device, but the principle is the same  -
in any situation requiring input, the system must cope with any possible
input.

(I'm reminded of a test which a lecturer once gave to his class, to 
write a program to return the solutions of a quadratic equation
               y = a*x**2 + b*x + c
upon input of a, b, c.  Every program failed for the case a = b = c = 0.)

Of course, some of this can be handled by insisting on certain design
rules, such as "Every CASE statement must have an explicit trap at
the end, even if it only executes a null statement."  At least this
forces the programmer to think about this case, even if it _should_
never arise.  However, many other situations are far less clear, and you
can't just rely on design rules.

>.... The test tools I've seen advertized tend to be designed to test the
>finished product, especially tests which replay keystrokes (since
>interactive programs tend to fit in category #1 above).  This is nice
>but when a problem occurs you still have to find the component in
>error.

>If such products exist I'd be interested in hearing about them.  I've
>got something hacked together to provide fairly painless C++ class
>testing as part of a normal make, but it's just been spare time
>playing around.

I guess I'm unconvinced that testing tools will ever be useful outside
specialised areas, although probably very useful within such.  IMHO the
best test is a nasty-minded person, and _never_ the one who wrote the system.

Geoff Miller  (g-miller@adfa.oz.au)
Computer Centre, Australian Defence Force Academy

Newsgroups: comp.software-eng