From: bruce@utafll.uta.edu (Bruce Samuelson)
Subject: for Claus Gittinger
Organization: University of Texas at Arlington
Date: Tue, 26 Jan 1993 19:14:46 GMT

Claus, here is email I tried to send you but my mailer returned with
host unknown.

Date: Tue, 26 Jan 93 13:00:18 CST
From: bruce (Bruce Samuelson)
To: gitting@slsvftt.us-es.sel.de
Subject: your comments on slopstones and smopstones

Thanks for your comments, which are all valid. I have received similar
comments privately and publicly, and I agree that Slopstones in particular has
shortcomings. I tried developing several earlier versions prior to the one I
published. All the earlier ones were either flawed worse, or made ST/V-DOS
crash when I ran them.

Slopstones is intended to measure strictly low level operations and will not
necessarily reflect performance in an actual application. I wanted to measure
seven common operations and get the fastest time I could for each. I agree
that a compiler could theoretically compile away some of them, and in fact, if
you had seen a posting a couple weeks ago by Urs Hoelzle, Self did in fact do
just that. You are correct, though, that theoretically they cannot be
optimized away, because one could always redefine integer addition, for
example, to not only add two integers, but to do something else such as keep
track of how many times integer addition has been invoked.

Fractonaccis: yes, fractions in themselves are not that interesting, but the
machinery needed to add/subtract them does invoke quite a lot of integer
arithmetic, which *is* interesting. However, I did not choose fractions
because they were interesting, but because fibonaccis do not produce usable
runs.  Either the numbers grow larger than 16-bit integer limits on ST/V-DOS
(16363) or the run time is too small to be reliably measured. Regarding
breaking the fractonacci test into recursion plus something else, that would
have been a possibility. I'll comment on this more later.

[] value: I agree that passing arguments and referencing method variables
would have also been interesting tests. I limited myself severely as to the
number of tests I did because ST/V-DOS was too crash-prone whenever I got too
ambitious. If I had not been constrained to 16-bit integers and ST/V-DOS
compatibility, I could have done more that was interesting. The reason I broke
the tests up into a 'setup' method and 'execute' method was that the ST/V-DOS
compiler could not handle it when it was combined into one large method.

Your simple recursion test: this was almost identical to the one posted by
Marten Feldtmann a few weeks ago. It turned out that ST/V-PM outperformed
ST80-Win on his benchmark. I wrote Slopstone and Smopstone to see whether it
would also beat ST80 on a wider range of tests. In general, I don't think ST80
is very well optimized for recursive calls.

Your sieve test: I wish I had this when I wrote Smopstones. I probably would
have included it.

Conclusion: Thank you for your constructive criticisms. I would incorporate
some of your suggestions into a revised form of Slopstone and Smopstone, but I
don't have the time. They are already set in concrete, and quite a few results
have already been tabulated for them. However, perhaps in the future, you or I
or someone else may offer a new set of benchmarks that are more theoretically
sound and well balanced. Thanks again for your comments.

I plan to write a short article for The Smalltalk Report on benchmarking, and
will cite your concerns there, along with similar concerns from others. I'll
keep your name anonymous unless you tell me otherwise.
-- 
**********************************************************
* Bruce Samuelson	Department of Linguistics	 *
* bruce@ling.uta.edu	University of Texas at Arlington *
**********************************************************
Newsgroups: comp.lang.smalltalk