From: glasser@PACorp.com (Scott Glasser)
Subject: Re: X Test tools (was: Re: X-runner)
Date: Wed, 14 Oct 1992 17:45:49 GMT
Organization: Performance Awareness Corp


I've been reading with interest the recent discussion on automated
testing of GUIs.  My company provides such a tool [preVue-X(tm)], 
so I felt compelled to respond to a few of the issues raised here.

Nick Hounsome writes:

>I just read about it in a VERY intersting article on computer aided
>software testing (CAST) in "Computing" (24th September) which you 
>should read - To summarize it says that although CAST tools are 
>often very clever they are almost useless for GUIs unless you have
>a stable product and are using them for soak testing or regression testing.
>This is mainly because it is impossible for a program to know
>what is relevant and what is not in a graphic display and because
>differences must be either a list of unintelligible X packets or
>massive screen dumps.

I agree with most points made by Alan Weiss in his response, and 
would like to clarify it with a few additional points.

Alan Weiss writes:

>4.  The program doesn't have to know what is relevant or not -- it
>is doing dumb diffs.

True. In addition, good GUI test tools let you dump selected windows
and exclude areas that may be irrelevant to the test at hand.  They are
also very good at highlighting differences which can easily go unnoticed
by humans.

>You can TRY and create a virtual layer between
>X and the test tool to analyze X packets for relevant changes, but
>this is Very Hard To Do  :-).

preVue-X does in fact analyze the X packets on a per-client basis.  
This information is used for synchronization and verification.
(You're right, X protocol analysis is no picnic! :-)

>The fact is, screen dumps do take up a lot
>of disk space, so use compress or something similar.

We address this problem by dynamically compressing the image as it is
written to disk.  Although the compression factors varies based on image
contents, we typically experience compression factors in the range of
15 - 20 for graphical user interfaces to commercial applications.  By
combining this approach with the user's judicious selection of when and
what windows to dump, we find that window image storage is usually 
manageable.
 
>5.  My last point:  GUI test tools are clearly imperfect, but they
>*are* tools in the Test Engineer's arsenal.  A wrench makes a terrible
>hammer, but a fine wrench.

I agree that GUI test tools are imperfect when they are used early in the
development cycle, but can be useful nonetheless.  Where they shine are;
1) during the system test phase where the application's user interface has
   stabilized, and regression tests are critical to insure that bug fixes
   do not break other things;
2) during ports to new platforms, where they can automate the repetitious
   manual testing, providing significant savings on each port;
3) for stress testing and performance studies where it is desirable to
   maintain a reproducible workload of multiple X user sessions (with
   multi-user GUI test tools, such as preVue-X).

>6.  Wouldn't it be great if, every time, say, Customer Support ran
>through a problem recreation session, they could capture the session
>and deliver a file to Development and QA/Test?

Yes, especially if that file can ALSO be played back without running the
live application(s).  We've developed technology that allows replaying
the "canned" X protocol requests of the original session.  This allows
guaranteed reproduction of the original problem, and avoids complicating
issues such as program versions, user X environment customization, and
other user environment differences.  A number of vendors are trying to
come together to provide a standard solution.  If you want more
information on this, give me a call.

+-------------------------------------------------------------------+
Scott Glasser			Phone:  (919) 870-8800
Performance Awareness Corp.	inet:	glasser@PACorp.com
Raleigh, N.C.			uucp: 	uunet!prevue!glasser
-- 

Newsgroups: comp.software-eng