From: paulb@syacus.acus.oz.au (Paul Bandler)
Subject: Value of Code Coverage Analysis Metrics - Summary
Organization: ACUS Australian Centre for Unisys Software, Sydney
Date: Mon, 21 Dec 1992 06:15:09 GMT


Organization: Motorola Land Mobile Products Sector

In article <1992Dec14.072812.13689@syacus.acus.oz.au>, paulb@syacus.acus.oz.au (Paul Bandler) writes:

<preamble deleted>
 
|> I believe we have a tool to measure the %BFA 'Branch Flow Analysis' but
|> of course the engineers are responsible for producing the test cases to 
|> exercise the code.
|> 
|> I have 3 questions:-
|> 
|> 1)  Do people think that this is a valuable metric?
|> 2)  Is it a cost effective excersize to get engineers to achieve a particular
|>     %BFA as a completion criteria?
|> 3)  What is a realistic %BFA to aim for?
|> Paul Bandler

BFA is a valuable tool for testing. %BFA, as with all unitless numbers, is
of a more dubious nature.

%BFA does not tell the person looking at it which paths were not tested
and why. This is important because engineers could, subconsciously or
otherwise, use their quota of untested branches to avoid testing the more
complex/niggly areas of code.

The nature of the testing is also important - only if you are programming
in an interpreted language where syntax must be checked by execution,
is executing every line of code useful by itself. (You can
run through every branch of a square root function but it won't
test for what happens with a negative parameter. There is always a
temptation to say that a line i++ 'passes because it increments i' without
checking when it shouldn't.)

As to cost effectiveness this is so dependant upon the available tools,
maintainance costs, whether or not testability was built into the code etc.
that any global statement would be rash.

A realistic %BFA target is again dependant upon specific circumstances.
I don't know if anyone has done any work to find out the %age of
defects found, against testing coverage and time taken but I would
be interested to find out. The benefits of higher percentages are not
linear and certainly peaks.

Setting %BFA as a part of completion criteria is probably worthwhile but
getting dogmatic about it is probably not. BFA is very useful for helping
engineers see where more testing is required it is not that useful as an
exam mark for the testing.

-- David

========= All opinions are mine and not necessarily Motorola's ============
=  @mail : David Alexander, Channel Tunnel Software, Motorola, Lyon Way,  =
=                           Camberley (ZUK20), Surrey GU15 3QG, U.K.      =
= Email : (Internet) davidal@comm.mot.com  Motorola X400-gateway : CDA004 =
= Telephone : (office) +44 (0)276-413340   (home)+44 (0)276-24249         =
===========================================================================

Subject: >>>  Value of High Code Coverage Metrics in Testing - Request for Opinion
Date: Mon, 14 Dec 1992 12:28:01 GMT
Organization: Engineering, CONVEX Computer Corp., Richardson, Tx., USA
X-Disclaimer: This message was written by a user at CONVEX Computer
              Corp. The opinions expressed are those of the user and
              not necessarily those of CONVEX.


Performing test coverage analysis on code is more of a good attitude than
it is a silver bullet for removing bugs.  There is an old mgmt saying,
"What is inspected is what gets done". 

> 1)  Do people think that this is a valuable metric?

Again, it improves the code but it does not remove all bugs. Design flaws
about cases not being handled cannot be found.  Also, any code that uses data
driven tables (e.g. finite state parsers) cannot be measured effectively. 

> 2)  Is it a cost effective excersize to get engineers to achieve a particular
>    %BFA as a completion criteria?

Sure... if you have the time in the schedule and effective tools to do it. 
Usually, it is best to have people do coverage analysis on their own code
and have people who don't know the code test it for unanticipated conditions.

> 3)  What is a realistic %BFA to aim for?

Fairly low percentage of 50% is usually the maximum possible value.  Much 
code is assertion testing.  However, every condition should be accounted. 

Thank you. 


						Steve Simmons

From comp.software-eng Wed Dec 16 08:41:18 1992
Newsgroups: comp.software-eng