From: hammons@csl.dl.nec.com (Bud Hammons)
Subject: Re: Separation of Software testing from System Testing
Organization: NEC America, C & C Software Development Laboratory
Date: Tue, 7 Apr 1992 13:06:42 GMT

To: rms@Miles.COM (Rob Schultz)
Subject: Re: Separation of Software testing from System Testing


Dear Mr. Schultz:

In article <1992Apr6.210631.3132@Miles.COM>, you write:

|>
|>
|> The two sides of the issue are summarized as follows:
|> 
|> Software System Testing should be a part of System Testing for the
|> instrument, since there is so much overlap between the two.  Many of
|> the software functions cannot be adequately tested without some hardware
|> intervention (primarily since the System Specification never made a clear
|> distinction between the hardware operations and the software operations -
|> there is no clear interface).  Therefore, let's just skip the Software
|> System Testing since it will all be handled during the Instrument System
|> Testing anyway.
|> 
|> The other side is that Software System Testing should NOT be a part of the
|> System Testing, since System Testing targets a much broader type of errors
|> than Software System Testing.  The fear here is that if the Software 
|> System Testing phase is skipped, then an entire class of errors may be 
|> missed or masked out by the hardware or systems issues.
|> 
|> The response to this is that it really doesn't matter, since the system
|> testing will prove out the instrument's functionality.  After all, we are
|> building a system that consists of Mechanical, Optical, Chemical, Electrical,
|> and Software Components.  The end result is more than just a software
|> product.
|> 
|> Both sides in this debate have good points.  My problem is to determine which
|> approach to take for this project.  Can anyone provide some additional 
|> insights for me?
|> 

I'm puzzled, but do appreciate your situation.  I've been through this on a
few projects before, and I could not have survived without doing some level of
separate software testing.  As you are apparently building a tightly coupled system
in the sense that software and hardware are heavily independent, it is worth
going back to the beginning:

   [1] was there an explicit allocation of requirements to software/hardware?

   [2] if so, in those areas where the hardware and software must interact to
       meet a requirement (say, some noise threshold or other figure of merit
       for a control system) was there an explicit partitioning of the 
       requirements between software and hardware components?

If you cannot answer these questions in the affirmative, the ability to separate
the software and hardware testing becomes quite problematic _for those
requirements_.  In any case, separate unit and subsystem tests for the software 
is the absolute minimum I would accept as a workaround.  I would also try to 
re-open the requirements specifications to address any allocation issues, since
you may have a hard time convincing yourself or a customer that you have a 
roadmap to reliably produce the physical components of the system, besides the
assertion that "if we build them just like this breadboard that seems to work
we will be o.k."  That argument is not a strong position in the case of a system
that has a long lifecycle and may undergo significant component and/or subsystem
revisions.

Next, one way to deal with your situation is to use the _requirements_ as a basis
to construct a computer simulation of the physical platform and its environment,
which can serve as a platform for system engineering tradeoff studies as well as 
software testing.  This gives you three levers to address your project risks:

   [1] the simulation provides a medium in which to understand the details of
       the requirements allocation between hardware and software, and, depending
       on the degree of fidelity that you model the operational environment, this
       simulation provides you with a tool to predict the performance of your
       system.  This DOES NOT obviate the need for a feedback loop composed 
       of empirical data from actual tests. 

   [2] since your system is 'tightly coupled' the simulation provides you with
       a means to study effects of what must be done if some of the electronics
       or mechanical components of the system cannot be provided as originally
       specified, which can occur due to one or more of cost, schedule, size,
       size/weight/power constraints, or the intrusion of physical reality or
       current technology limitations.  Unless everything on the hardware side
       of the system is strictly off-the-shelf with no unknowns in integrating 
       the physical system, this kind of study is almost inevitable.         

   [3] your development has a better chance of successfully running in parallel
       with electronic, mechanical, and computer software design actually
       integrating in the end, since the simulation provides early sanity
       checks of what was initially planned on paper.


Good luck,

-- Bud Hammons
-- 
****************************************************************************
* Bud Hammons            Internet:   hammons@csl.dl.nec.com                *
* NEC America Inc.       CompuServe: 71160,3220                            *
* 1525 Walnut Hill Lane  voice:      (214)518-3488     CCSL S/W Dev.Lab.   *
* Irving, Texas 75038    FAX:        (214)518-3552     S/W Eng. Group      *
****************************************************************************  

Newsgroups: comp.software-eng