From: vmarsch@adobe.com (Vonn Marsch)
Subject: SUMMARY: Automated Testing of X Applications
Organization: Adobe Systems Incorporated
Date: Thu, 27 Aug 1992 21:25:04 GMT

Dear Friends, 

Here is a summary of responses I've received through e-mail and net postings.
At the end are my impressions and a list of systems, company names, addresses,
and phone numbers.

Thanks, again, for all of the responses.  -VM

------------------------------------------------------------------------------

From: pstephan@test490.pac.sc.ti.com (Patty Stephan)
Date: Thu, 20 Aug 92 05:06:47 CDT

We are currently looking at the same issue, how to do automated regression
testing on our OpenWindows application.  I am in the process of getting an
evaluation copy of Software TestWorks from Software Research, Inc.  I also
will be looking into getting the same for XRunner.  Basically, as far as
I've heard, TestWorks provides good basic functionality of regression testing
for X applications and seems to work pretty good.  I've also heard that 
XRunner works better, but is alot more expensive.  I've never heard of Ferret.

For more info on Software TestWorks, contact Luigi Favero at:

Software Research, Inc.
625 Third Street
San Francisco, CA 94107-1997
Tel: (415)957-1441
FAX: (415)957-0730
E-Mail: favero@soft.com

Patty Stephan
Software Design Engineer / Software Process Engineer

-----------------------------------------------------------------------------

From: Spiros Triantafyllopoulos <c23st@kocrsv01.delcoelect.com>
Date: Thu, 20 Aug 92 10:42:24 EST

We had the XRunner folks come here for a demo. The price of the
software is spectacular. 50,000 dollars for 2 test developers and
3 test users licence, 100,000 for 5 dev / 9 user. The software looked
OK though.

What bothered me about them is the price of the software is way
out of line; 50,000 dollars buys me, say, N copies of interleaf,
or a few full blown SAS copies. Both products represent years
if not DECADES of development by large corporations, while XRunner
is (self admittedly) just a very clever, simple program. (their words).

Top that off with the fact that they've been calling us ever since,
faxing, etc....

The software IS GOOD, though. Don't know if it's good enough for 50 grand
but it's good.

The funny part is I know it's not hard as I implemented single
window event capture/playback in a few days myself. The hard 
part is window manager transactions.

-- 
Spiros Triantafyllopoulos                    c23st@kocrsv01.delcoelect.com
Software Technology, Delco Electronics       striantafyll@kosds1.gm.hac.com
GM Hughes Electronics, Kokomo, IN 46904      [A Different Kind of Disclaimer]

------------------------------------------------------------------------------

From: newkerk@zso.dec.com

I haven't used either one, but I know that XRunner at least depends on doing
comparisons of saved bit maps to verify the operation of your application.  
This can tend to be VERY slow.  There is another company you might want to 
get some information from called Segue Softare Inc.  The have a GUI test
tool that they are just releasing that supports X, Motif, SUN, and MS Windows. 

The basic idea is that rather than recording and saving bitmaps, they have a
script language and a driver that inserts keyboard and mouse events into your
application's event queue throught a library you link into your app.

Then the script can 'drive' your program and compare the internal state of
the widgets (screen location, contents of text widgets, etc) instead of 
bitmaps of screen images.  They also support using the same test script to
drive the app on different window systems.

Segue Software Inc.
25  Shirley Street.
West Newton Mass  02165
(617) 964-7565

------------------------------------------------------------------------------

From plp@uel.com Thu Aug 20 16:33:57 1992
Date: Thu, 20 Aug 92 13:12:56 BST

Several weeks ago, I too was looking for this information.
Because of the specifics of my task (Intel machine, limited
test requirements, deadlines!) I decided to write the tool myself. 
Below is a summary of information I gathered - hope some of it is 
relevant:-

There appear to be two main problems: 1) many of the tools require
some sort of extension that your server may or may not support 2) few
tools have an Intel version (most are for Suns). XRunner has, or at
least was known to have, both these problems.  Apart from XRunner, 
there are two products that are of interest: gTSIMX and preVue-X.  

gTSIMX is an (all singing and all dancing) 386-based product from 
SNI. Its major drawback is that the documentation is in German.  
PreVue-X from Performance Awareness looks like it may be a useful 
product.

Here are the numbers at each of the companies:

   gTSIMX from SNI:

	   +49-89-636-45295

   preVueX from PAC:

	   (703) 749-7738

   XRunner (this you probably know already):

	Mercury Interactive Corporation
	3333 Octavius Drive
	Santa Clara, CA 95054
	(408) 987-0100 (voice)
	(408) 982-0149 (FAX)

Predrag Popovic

Unix System Laboratories Europe Ltd.
Ealing Broadway, London.

-------------------------------------------------------------------------------

From royce@mdcbbs.com Fri Aug 21 19:48:56 1992

( royce writes: )
>> X applications need to build in automated testing tools to get the most bang
>> for the buck.

( vmarsch writes: )
>Is this necessary for either FERRET or XRunner?

No, this is the biggest strength of these tools.  But you can't adapt easily to
changes in the user interface without careful programming and limiting the
kinds of changes involved.  Development will want to use the tool to debug
defects, so the tool has to work effectively with the debugger.  These issues
can be better dealt with by a tool built into the product.

-- Royce
royce@mdcbbs.com

------------------------------------------------------------------------------

From: jim@bat.cis.ohio-state.edu (Jim Curran)
Date: 24 Aug 92 21:28:23 GMT

You could check into the XTrap or XTest extensions.  Both record user input and
allow playback.  There is a package on export.lcs.mit.edu called xrecplay
which contains some applications which use XTest to do application testing
which covers lots of what you want.

Jim

------------------------------------------------------------------------------

From: akerson@deimos.srg.af.mil (Pete Akerson - akerson%srg@uunet.uucp)
Date: Wed, 26 Aug 92 13:46:33 GMT

 We've seen with XRunner, we're getting a copy of Preview X, also there
is something called Cap Back X, dunno 'bout that one. 
FYI, Ferret uses propriatary hardware, and gave us lots of problems in
this mostly SCO-unix shop.

Your mileage may vary.
--
Usually Reply-to: akerson%srg@uunet.uucp
Internet: akerson@srg.af.mil@uunet.uucp
Uunet: uunet!srg!akerson
Milnet: akerson@srg.srg.af.mil     maybe ...

------------------------------------------------------------------------------

Here is a list of systems, where you can get them, and my own opinions based
upon what literature I could get my hands on.:

-----

XRunner

Mercurey Interactive Corp.
3333 Octavius Drive
Santa Clara, CA  95054
Tel: 408.987.0100
Fax: 408.982.0149

The only literature I have is the User's Guide, which is not the most stellar
piece of documentation I've encountered.  Like most aspects of XRunner, it's
pretty good on its own, but not given the outragous price that is being charged
for the product.

I have two impressions from the User's Guide, either or both of which could
be mistaken: First, XRunner has the advantage of being originally developed
for X; and, second, that have to be fairly well along in the development
process, or have a stable X version up and running, to use XRunner.

-----

Software TestWorks
STW/REG Regression Analysis Tools
STW/COV Coverage Test Tools

Software Research, Inc.
625 3rd Street
San Francisco, CA  94107-1997
Tel: 415.957.1441
TFN: 800.942.SOFT
Fax: 415.957.0730
E-Mail: info@soft.com

This looks like a relatively mature product with enough different modules and
sub-systems to keep you and your co-workers speaking in acronyms 'til you're
blue in the face.  If you buy and use it all, you'll have an entire automated
QA system (for better or worse).  It's also competitively priced.

-----

FERRET

Tiburon Systems, Inc.
1290 Parkmoor Ave.
San Jose, CA  95126
Tel: 408.293.4400
Fax: 408.293.9090

Similar to STW, but a turnkey system.  You plug this 486 PC into your
workstation, and everything is done through the PC.  This means you have to 
learn to use some MicoSoft programs, though.

It looks like, for mid-scale QA departments, it will cost more than STW, but
far less than XRunner.

-----

preVue-X

Marcus Rogerson
Sales Manager
Performance Awareness
2010 Corporate Ridge
Suite 700
McLean, VA  22102
Tel: (703)749-7738
Fax: (703)893-9340

This software system doesn't appear to do as much as STW, but it doesn't
cost nearly as much, either.  There is a unique pricing scheme where you pay
based on the number of users that are simultaniously emulated (emulated 
employees must be the way to do things in DC, these days).

-----

QA Partner

Segue Software
25 Shirley St.
West Newton, MA 02165
Tel: 617.494.1830

The only literature I have is a paper by the systems creator, Lawrence Kepple,
that was originally published in X Resource Journal.  The paper outlines 
what I think is a wonderful paradigm for an automated test system.  You end
up with T (based on C) language test scripts that can be used on any platform
supported by QA Partner.  Unfortunately, the X version is only in beta now.
I highly recommend calling these people up and asking for a copy of the paper;
and maybe they will sign you up for their beta program, too.

---

Vonn Marsch
Adobe Systems, Inc.
vmarsch@adobe.com
Newsgroups: comp.windows.x.apps