From: barrett@snoopy.cs.umass.edu (Daniel Barrett)
Newsgroups: comp.sys.amiga.emulations
Subject: Testing is hard (was Re: Emplant)
Date: 23 Aug 92 22:33:29 GMT
Reply-To: barrett@gleep.cs.umass.edu (Daniel Barrett)
Organization: University of Massachusetts, Amherst

>In article <52302@dime.cs.umass.edu> barrett@gleep.cs.umass.edu (Daniel Barrett) writes:
[Claiming that Emplant can't be accurately reviewed in 1 day]
>>	What if this "1 day wonder review" said that the product works
>>as advertised.  Would you believe it?  I would be very skeptical; how
>>much could the reviewer really do or know in that amount of time?

In article <1992Aug23.185744.23796@news.iastate.edu> barrett@iastate.edu (Marc N Barrett) writes:
> How much time should it take to check out a product such as this?  Emplant
>is conceptually simple.  I know if I had access to Emplant, and it did work 
>as advertized, I could put it through a thorough test in an hour or two.

	Marc, I don't mean to be condescending here; but if you truly believe
that, then you know very little about evaluation and testing.  What would
you do -- run a few applications, notice they don't crash, and say it
works?  Sigh.  If testing is so easy, then why is it still a major open
research question in computer science?  Why do companies spend so much time
and money on it?  Why is such a large percentage of the software/hardware
lifecycle devoted to it?

	When evaluating a product (Emplant, or whatever), people have to
work in the area between theory and practice.  In theory, it is impossible
to decide whether Emplant and a Mac II work identically or even compatibly.
(CS people will recognize this as "The Halting Problem" in disguise.)  In
practice, we have to know whether Emplant "works" or not, and within a short
(or even finite) amount of time.  So what should be done?

	This is a hard problem.  Some people will run "MacPaint" and "Word"
for 15-20 minutes each, notice no problems, and declare the emulator a
success.  Of course, this would totally miss the fact that XYZ MIDI program
crashes instantly.  Or that Word crashes when you use the spell checker on a
large document, something that the reviewer didn't bother testing.  [Note!
I am just speculating -- this is all hypothetical!!]

	Obviously, we cannot test all possible Mac programs on Emplant in
all possible configurations.  So how much testing is "enough"??  I'll give
you a hint:  very few engineers would agree that "an hour or two" is enough.

Marc sez:
>...it would not take more than an hour to try out various programs to 
>make sure they run correctly with color.

	Thank you, Marc, for illustrating exactly the reviewer mentality
that I was criticizing in my previous post.

	Everybody, please take a moment and remember when you read a
positive review of a product, so you ran out and bought it.  But when you
used the product yourself, you noticed some shortcomings not mentioned in
the review.  After a few weeks of use, you looked at the magazine again and
wondered how the hell this lousy product got a favorable review.

	Has this happened to you?  Probably!!  Did you buy an early
Psygnosis game because a reviewer raved about its amazing graphics, only to
find that the game is totally unplayable?  How about Origin's "Times of
Lore", reviewed with high marks in .INFO magazine, which can make you lose an
entire journey's worth of progress if you happen to click the joystick ONCE
incorrectly?  How about that early Hypercard-like program (I have forgotten
the name; the precursor of "Foundation") that actually was endorsed by one
of the original Amiga creators, reviewed favorably, and turned out to have
more bugs than a beehive?  And how about programs that worked on the
reviewer's machine but crashed on your hardware configuration?  Or didn't
run under your version of the OS?

	Testing is HARD, folks.  Evaluating a product is HARD.  Writing a
careful, accurate review takes time.  If someone tells you otherwise, then
that person is wrong.  This isn't just my opinion, either.  Pick up any book
on software engineering and read the chapter on Testing, and you'll see the
same ideas.

	Alternatively, pick up a magazine whose reviewers really give the
product a workout.  For example, look at KEYBOARD MAGAZINE's main reviews of
computer music hardware and software.  Some of their reviewers are quite
thorough, torture-testing the stuff until it bleeds.  Of course, they don't
catch everything (nobody can).  But they try, and it shows.

>>I sincerely hope that anybody who publishes an Emplant review has
>>used the board extensively for at least a month.

>Overkill.  Emplant makes an Amiga appear just like a real Mac II.  Therefore
>it should not take much longer to give Emplant a thorough test, than it
>would take to ensure that a brand-new real Mac II system works well after
>being purchased.  

	Sorry Marc, but a month is actually "underkill"; but since we live
in the real world, we have to stop sometime.  On the theoretical side of
testing, the whole problem of deciding whether a product works as advertised
is "undecidable" -- that means there is literally no way to do it with 100%
accuracy.  In any case, the more testing, the better.

	But let's be practical.  Your assertion that testing Emplant is
analogous to testing a new Mac II is untrue.  The Mac II *defines* its own
standard.  Of course it works just like a Mac II.  So running a few
diagnostics or programs, and seeing no problems, probably means that the
machine is working with high probability.

	But Emplant is a new product with (perhaps) who knows how many
subtle bugs that could make it slightly or radically different from a Mac
II.  You're going to find these problems, if they exist, by running a few
programs?  No way.  I don't think you could exercise "MacPaint" (that is,
force it to run through every logical path in its source code at least once)
in an hour.  How many 3rd-party hardware products can you hook up to Emplant
and test in 1 hour?  And so on.

	To put it bluntly -- if testing Emplant for its Mac II emulation
were easy, then Jim Drew could have done it in "an hour or two" just like you
said you could.  But I believe most of us feel intuitively that Mr. Drew
*has* to do more than an hour of testing to convince us, and himself, that
Emplant works.  And I assume he did.

	To conclude:  Marc, sometimes you're right and sometimes you're
wrong.  In this case, you spoke too quickly, and you're now over your
head.  I advise you to bail out. :-)

	I'm not sure whether discussion of software/hardware testing is
appropriate for c.s.a.emulations.  If people want, I'll move it to another
newsgroup.  But for now, since it started here, I'll continue discussing it
here.

                                                        Dan

 //////////////////////////////////////\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
| Dan Barrett -- Dept of Computer Science, Lederle Graduate Research Center |
| University of Massachusetts, Amherst, MA  01003  --  barrett@cs.umass.edu |
 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\/////////////////////////////////////