From: rlosee@unlinfo.unl.edu (Robert Losee)
Newsgroups: comp.os.os2.advocacy,comp.os.ms-windows.advocacy
Subject: Re: OS/2 vs NT benchmarking (was: Re: Can we get the details of ...)
Date: 16 Dec 1992 14:52:36 GMT
Organization: University of Nebraska--Lincoln	
Distribution: world
NNTP-Posting-Host: unlinfo.unl.edu

[...]
>The point here is that Tim was comparing the _first_ beta of a brand
>new operating system, with what is likely a _final_ beta of a major
>bug fix release.  It is questionable whether benchmarking these two
>releases has much merit.

As a person who must support clients and their decisions to purchase 
products I'd like to know what fairer method exists.  Mr. Sipples 
essentially took the most current available version of OS/2 and NT and
compared them.  If he or anyone must take wait for the next great 
release of either product then no evaluations would ever be done.  
While it may be true that NT in six to nine months will be a much 
improved product the same benefit of the doubts must given to OS/2. 
In short we have two moving targets for operating systems, the best I
as a purchases/recommender can expect is the real, the current
systems, in a comparison; then I can draw my own conclusions about 
where they be on the race track as they round the bend in a year.  
Please don't compare imagined systems as that introduces biases based 
on imagination and prejudice.

>Certainly one should warning readers explicitly of the fact.
>Tim had sufficient warnings for the knowledgable in his post, but posted
>it widely where many would not be so knowledgable.

Oh please don't add any more warnings.  If we must get a full digest 
of every possible thing effecting the test in order to meet the lowest
common denominator of reader, these posts would literally fill
volumes.  They'd but chuck full of data and offer no information. 
Let's assume readers have some modicum of intelligence, even that
they are intelligent enough to know when and how to come up to speed
in understanding these things.
[...]
>I've been yelled at for criticizing Tim's "fair" benchmark, but the
>thrust of his post is clearly that of showing OS/2 somehow superior to
>NT, but given the relative states of the two beta distributions such a
>conclusion simply cannot be drawn.  Given the fact that he attempts
>to draw such conclusions, how can this benchmark on these systems in
>such different stages of development, be called "fair"?

As a judge of systems I'd expect you and others to offer counter 
benchmarks.  Until then I'd say the current version is superior, 
although I'd fully expect the differences to lessen a lot by release 
time of NT.  It's up to me to guess how much.
[...]
>In conclusion, please take any benchmark results Tim posts against the
>NT _first_ beta with a large grain of salt.  If Windows fans had done

Of course, the buyer must beware.

>this to the early OS/2 betas, OS/2 would have looked even worse.
>When the March/April time frame rolls around, the current beta then should
>be closer, and then perhaps some results worth looking at can be obtained.
>At the moment, the current attempts can only be viewed as FUD.  There is
>no rational basis for comparing untuned code with tuned code.
--
*-------------------------------------------------------------------------*
Robert D. Losee                 BITNET: ucas402@unlvm
University of Nebraska          Internet: rlosee@unlinfo.unl.edu
Lincoln NE  68588-0496          Phone: (402) 472-7663
Newsgroups: comp.os.os2.advocacy