From: brian@jaguar.cs.utah.edu (Brian Sturgill)
Subject: OS/2 vs NT benchmarking (was: Re: Can we get the details of ...)
Organization: University of Utah CS Dept
Date: Tue, 15 Dec 92 23:09:48 GMT

In article <JACOBS.92Dec15071114@cells.cs.utah.edu> jacobs@cs.utah.edu (Steven R Jacobs) writes:
>In article <1992Dec15.091108.25482@fcom.cc.utah.edu> brian@jaguar.cs.utah.edu (Brian Sturgill) writes:
>>In article <1992Dec15.062001.14820@midway.uchicago.edu> sip1@midway.uchicago.edu writes:
>>>In article <1992Dec15.051348.22687@fcom.cc.utah.edu> brian@jaguar.cs.utah.edu (Brian Sturgill) writes:
>>   ...
>>>The fact is that neither represents code that will be delivered at any
>>>date that is cast in stone.  IBM has announced no date whatsoever on
>>>that code, and Microsoft has said, "First half, 1993" -- or
>>>thereabouts.
>>   ...
>>
>> Sorry, Tim, but IBM has set a date.
>>
>> Here is an excerpt from "ibmplans.txt" on ftp-os2.nmsu.edu.  It
>> is a part of an IBM document detailing IBMs near-future plans for OS/2
>> (and a little MS bashing as well).
>>
>>>  1992 OS/2 PRODUCT PLANS
>>> 
>>>       There will be a lot happening in the OS/2 product line through
>>>  the remainder of 1992.
>>> 
>>>  ENHANCEMENTS TO THE BASE PRODUCT
>>>       In the fall:   32-bit graphics engine; Windows 3.1; device
>>>  drivers to support XGA, SVGA, and 8514 displays in seamless mode.
>
>Funny, I don't see any announced *date* in there.
>
>> Now I believe the claim by OS/2 fans has been that if IBM gives a date
>> that they mean that date, so I can only assume that the release
>> of this beta should occur on or before Dec. 15.
>
>Come off it, Brian.   If someone else took a clip from a file talking
>about MS plans (especially a source as "official" as a non-MS FTP site)
>and tried to attribute an actual DATE from general terms such as "fall"
>and "end of year", you'd be jumping all over them.  Try to be reasonable.

Where have you been?  That's all the sorts of dates that the OS/2 fans
have been holding MS too.  For example, until a bit over a month ago, MS
had said they were targeting the end of the year with NT.  They never
said December 31, 1992, but we all know what was meant.  The point here
is that IBM "never" does this (at least according to OS/2 fans).

If the above IBM press release is not enough for you, then about Tim Sipples:

** Newsgroups: comp.os.os2.misc,comp.os.os2.advocacy
** From: sip1@ellis.uchicago.edu (Timothy F. Sipples)
** Subject: OS/2 Future Trends and Directions
** Message-ID: <1992Sep26.012811.6658@midway.uchicago.edu>
** Date: Sat, 26 Sep 1992 01:28:11 GMT
**  ...
** Today IBM Chicago held its monthly OS/2 User's Group meeting.  This
** month's meeting was devoted, in large part, to a presentation by two
** OS/2 experts from Austin (mainly delivered by Don Crabtree).
** ...
** Now, where is OS/2 headed?
** 
** By year end the following enhancements will be in place [meaning some
** will obviously be delivered well before December 31]:
** 
** ** LAN Server 3.0.  It will contain multiprocessing (asymmetric) and
** peer-to-peer features, among other things.
** ** 32-bit graphics engine. (*)
** ** SuperVGA drivers ("...representing 80 to 90 percent of the
** marketplace...") (*)
** ** Reduced memory requirements.  A standalone OS/2 2.0 station which
** now requires 4.5 MB will be trimmed by one megabyte to 3.5 MB, i.e.
** OS/2 2.0 will really fit in 4 MB. (*)
** ** Improved performance (in addition to the RAM diet). (*)
** ** Windows enhancements (i.e. Win-OS/2 3.1 with TrueType, multimedia
** extensions, bitmaps, and applets). (*)
** ...


And if that is not enough how about this quote from a slide shown at
the OS/2 developer conference.
	OS/2: 1992 ENHANCEMENTS

	* 32 bit multimedia extensions
	* Improved performance
	* 32-Bit Graphics Engine
	* 32-Bit Seamless Hi-Res Display Drivers
	* Windows 3.1 Support
	* Additional Printer Support
	* Laptop/Notebook Support
	* Service


Any way you look at it IBM promised that the release Tim was time would
come out this month (further Tim, by his own words, knew about this
at one time, though I assume he forgot).

The point here is that Tim was comparing the _first_ beta of a brand
new operating system, with what is likely a _final_ beta of a major
bug fix release.  It is questionable whether benchmarking these two
releases has much merit.

Certainly one should warning readers explicitly of the fact.
Tim had sufficient warnings for the knowledgable in his post, but posted
it widely where many would not be so knowledgable.

My understanding is that the PC Mag benchmark tests a wide variety
of graphics capability, poor showings in just a few areas could greatly
skew the results.  Tim's post claims that NT's 16-bit compatibiltiy
performance is 40% of Windows 3.1's.   Yet I use a 16-bit Windows terminal
emulator under NT,  I would die in such a program if this 40% figure
were generally valid.  I also occasionally play the 16-bit Windows game
Alien Force under NT... normally it works just fine, but when you shoot
a ship, the explosion draws _VERY_ slowly.  The point here is that
a general purpose benchmark would test the drawing mode used by the
explosion.  Clearly it is not indicative of the normal performance
of NT.  Tim says in his report that Windows 3.1 was clearly faster
than NT's emulation of 16-bit Windows. To this I agree, it is about
the speed of Windows 3.0 or a bit slower.  In other words faster than
OS/2 2.0 GA when using seamless Windows.

I've been yelled at for criticizing Tim's "fair" benchmark, but the
thrust of his post is clearly that of showing OS/2 somehow superior to
NT, but given the relative states of the two beta distributions such a
conclusion simply cannot be drawn.  Given the fact that he attempts
to draw such conclusions, how can this benchmark on these systems in
such different stages of development, be called "fair"?

Further he hand waves over the fact that the test is NT seamless
Windows against OS/2 non-seamless ones.  At least currently OS/2, is
much slower when running Windows seamlessly.  I think a word here is in
order to Windows 3.1 users about what the difference is.

Under OS/2 Windows programs can be run in two modes, seamless and
full-screen.  In seamless mode, things run like you would expect: DOS
windows, OS/2 command-line windows, PM apps, and Windows apps exist
side by side on your screen, you click on the window you want to be
active, and then type to it.

In full-screen mode, Windows runs as a separate Desktop.  If you start one
this way, the screen clears, a Windows desktop paints, and then your
Windows app appears.  If you then want to access a PM app, or a command-line,
you toggle back to the OS/2 desktop (double click a special ICON or Alt-ESC).
The screen clears and the OS/2 desktop is redrawn.  The point here is
that not only can you not see data at the same time, but that switching
is reasonably slow.  This is a very inferior way to use Windows programs,
yet this is the method used in Tim's benchmark.

The reason that full-screen mode runs faster is that in effect, one can
let the underlying Windows routines take over, thus it is not surprising that
it can approach the speed of Windows 3.1.  It is quite unlikely that
"seamless" Windows mode will be that fast as it cannot use the underlying
Windows functionality to do the drawing, but must emulate the drawing
inside OS/2's drawing engine.  This is the major reason why, once it has
been tuned, NT is likely to be faster than OS/2 at seamless Windows.
NT of course is based on the Win32 API, and Win32 is much closer to
the Windows 3.X API that what OS/2 is.  OS/2 therefore has to do more
work in performing the Windows 3.X emulation.

In conclusion, please take any benchmark results Tim posts against the
NT _first_ beta with a large grain of salt.  If Windows fans had done
this to the early OS/2 betas, OS/2 would have looked even worse.
When the March/April time frame rolls around, the current beta then should
be closer, and then perhaps some results worth looking at can be obtained.
At the moment, the current attempts can only be viewed as FUD.  There is
no rational basis for comparing untuned code with tuned code.

Brian
-- 
C. Brian Sturgill                          NeXTStep 486 is for _YOU_!!!!
University of Utah                    Microsoft needs competition, but I want
Center for Software Science          to be one of the many running Windows NT!
brian@cs.utah.edu; CIS: 70363,1373    :-) :-) :-)  NT SDK: $69; (800) 227-4679
Xref: sparky comp.os.os2.advocacy:10662 comp.os.ms-windows.advocacy:3326