From: alan@auntbea.austin.ibm.com (Alan R. Weiss)
Newsgroups: comp.software-eng
Subject: Re: X-runner
Date: 6 Oct 92 18:06:31 GMT
Organization: IBM AWD Austin, TX

In article <1992Oct6.080355.27824@cbfsb.cb.att.com> nh@cbnewsg.cb.att.com (nicholas.hounsome) writes:
>From article <1992Oct5.224401.29918@saifr00.cfsat.honeywell.com>, by dickey@saifr00.cfsat.honeywell.com (Bill Dickey):
>> 
>> Do anyone know anything of a tool called X-Runner?  I think it is a capture
>> and playback testing tool for X-window applications.  Any information anyone
>> could provide on how the tools works, support, etc. woulde be appreciated.
>> 
>> Thanks in advance.
>> 
>> Bill
>> dickey@saifr00.cfsat.honeywell.com
>
>Apparently it is for recording/playback of mouse and keyboard events with 
>what sounds like very good synchronization capability.
>
>Mercury Interactive
>98 Rue Alberts Calmette
>7850 Jouy en Josas
>France
>tel: +33 1 34 65 72 40
>
>price from $20000

You should also check out Software Research's "Capback/X" product,
which is about $7500 for a seven host license (and they seem to
be able to negotiate :-)

At the last place I was at I evaluated their product, and found
that their usage of time synchronization instead of event
synchronization was problemmatic; scripts recorded on one
system in one environment (performance characteristics composed
of memory size, swap size, load, etc. etc. ad nauseum) tended
to screw-up when played back on a different system with different
characteristics.  They have since fixed this with an event synchronization
version.


>I just read about it in a VERY intersting article on computer aided
>software testing (CAST) in "Computing" (24th September) which you 
>should read - To summarize it says that although CAST tools are 
>often very clever they are almost useless for GUIs unless you have
>a stable product and are using them for soak testing or regression testing.
>This is mainly because it is impossible for a program to know
>what is relevant and what is not in a graphic display and because
>differences must be either a list of unintelligible X packets or
>massive screen dumps.
>
>Nick Hounsome

I believe that this is just dumbness.  :-)  Nothing personal,
Nick, but here's my logic:

1.  GUI test tools work like this:  you record X events (keyboard,
mouse and screen).  Every so often, you take screen snapshots, and
store these in files.  At some point, you diff (compare) your standard
reference screens against your test run screens.  If they are different,
you fail the test (which *really* means mark it for later analysis,
whether human or by some other post-processing).

2.  It is true that if your GUI design changes, then (some portion)
of your test script will become invalid.  HOWEVER, understand this:
the only OTHER way of testing GUI's is to have a human being sit
down, generate scenarios, and run through them by clicking open
icons/menus/etc. and generally exercising the product from the GUI.

Whether you record this session or not, a person still must do this
work.  Why not record the session?  That way, if the GUI DOESN'T
change, the marginal cost of your NEXT test run is greatly reduced.

3.  How to minimize the basic problem:  split up your test assertions
into VERY small units (essentially make the test cases atomic).  That
way, if something changes, it affects a smaller number of tests.
You can also mix and match your cases better to derive logical system
test scenarios (or, more accurately, cross-functionality tests).
This is clearly imperfect, but it helps.  A little.

4.  The program doesn't have to know what is relevant or not -- it
is doing dumb diffs.  You can TRY and create a virtual layer between
X and the test tool to analyze X packets for relevant changes, but
this is Very Hard To Do  :-)   The fact is, screen dumps do take up a lot
of disk space, so use compress or something similar.  Analysis doesn't
have to be real time, either.

5.  My last point:  GUI test tools are clearly imperfect, but they
*are* tools in the Test Engineer's arsenal.  A wrench makes a terrible
hammer, but a fine wrench.

6.  Wouldn't it be great if, every time, say, Customer Support ran
through a problem recreation session, they could capture the session
and deliver a file to Development and QA/Test?

BTW, the reason my last firm decided against Capback/X was threefold:

a.  The GUI was, theoretically, going to undergo massive changes (this
was only semi-true), necessitating constant re-doing of the scripts.
But, note my logic above.

b.  Politics -- the boss asked the wrong person for his opinion, and this
person was just burned out from GUI testing and didn't know how to solve
some of the technical problems with automated GUI testing.  The lack of
event synchronization was used as an excuse for the next item:

c.  Money -- IMHO the *real* reason.  Some firms are running VERY lean,
and $7500 comes off the bottom line.  


-- 
   _  Alan R. Weiss   IBM AWD Austin AIX Kernel   11501 Burnet Rd, Bld 905
 _| ~-. auntbea.austin.ibm.com!alan@ibmpa.awdpa.ibm.com | Austin, TX 78758
 \, *_} Inside IBM:  alan@auntbea.austin.ibm.com        | 512-838-8715
   \(   DISCLAIMER:  Working at, not speaking for, IBM  |