From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver)
Subject: Re: Free NFS Response Time Measurement Software
Summary: you never get more than what you pay for
Organization: Silicon Graphics, Inc.  Mountain View, CA
Date: Wed, 19 Aug 1992 14:41:01 GMT

In article <nxfyh!-@rpi.edu>, fitzgb@mml0.meche.rpi.edu (Brian Fitzgerald) writes:
> NFS RTM Distribution writes:
> >
> >                    *************************************
> >                    FREE NFS SERVER RESPONSE TIME UTILITY
> >                    *************************************
> 
> ... cool program.  For anyone interested, here's an example output of
> the free demo they sent me.  Good teaser for the sharpshooter.
> ...

>                         NFS RESPONSE TIME REPORT
>                 Failure Rate       NFS Response Time (ms)      NFS Load
> Server            Over  20ms           Avg        Max         (ops/sec)
> -----------------------------------------------------------------------
> hammer                    0%           2.9        3.1               0.0
> darwin                    0%           1.0        1.0               0.0
> usenet                   20%          10.9       67.6               0.9
> mml0                      0%           2.5        2.7               0.1
> cds1                      9%           5.8       27.6               0.2


I'd be very surprised if the AIM test is a fraction as interesting as
LADDIS, the multi-vendor effort within SPEC to come up with an NFS
benchmark.

Deciding what to measure is always hard, and particularly difficult
with NFS.  Most people do not know and do not know they don't know
which operations are common between an NFS client and its server.  The
response time of the server depends radically on which operations you
assume are common, and about their nature.  Trivially, what block size
do you assume?  What about client or server caching?  Both of those
have 10X effects on read and write speed.  What about evil async
writes, which can have >2X effects?  I would not trust a benchmark that
did not spend a lot of time convincing me it's assumptions about client
working set are reasonable; you can expect "response times" to vary by
100X depending on the size of the client working set.

The hard problem with the operation mix is that reads and writes are
generally not as common as one would thing.  The exact operation mix
for LADDIS has been controversial.  They have switched to a synthetic
load generator from using real NFS clients.  I guess it was too hard to
define a standard client.  (It's well known that the speed of the
clients is at least as important as the speed of the server.)

Another controversy within the group working on LADDIS has been what,
if any, single "figure of merit" or number the LADDIS benchmark should
produce.  I don't recall seeing mention of "response time" for a long
time.

I'd be very surprised if the AIM test is very valuable.  I could be
wrong, of course, especially since I have never seen the AIM NFS test.

I don't make this attack lightly, but the history of commerical
benchmarks is not pretty.  If you think the system vendors are scum,
then you should check the benchmark slime.  Would it be fair to say
that SPEC was formed as response to Dhrystone and the so called
commercial CPU benchmarks?

If I wanted an NFS test, and if Silicon Graphics were not already
involved, I'd wait a few more weeks and then buy the tape of LADDIS
from SPEC.  (I think the SPEC CPU benchmark tape costs $200).

If I wanted a reasonable benchmark from people who understand NFS that
is free or next to free, I'd contact Legato for a copy of nfshstone (it
has an extra "h" in there somewhere).  It was the starting point for
LADDIS and is the benchmark that NFS server vendors are currently
competing on.  If you call a vendor and ask for numbers, you'll
probably be able to get nhfsstone numbers.

If I didn't want to spend any money, I would simply get a copy of the
Sun NFS validation suite.  It generates several numbers, and was the
de facto benchmark among NFS developers for years.  It's numbers are
not very good, but at least you get the source and can see exactly what
it thinks it is measuring.  Of course, you also get the source for
LADDIS and nfsshtone.


Vernon Schryver,  vjs@sgi.com
Newsgroups: comp.protocols.nfs