From: mjr@hussar.dco.dec.com (Marcus J. Ranum)
Subject: Re: Is local caching configurable?
Organization: Digital Equipment Corporation, Washington ULTRIX Resource Center
Date: Tue, 10 Nov 1992 01:36:56 GMT

>I'm using Ultrix 4.3, and have done some timing tests on
>local disk access vs. nfs disk access.  Nfs seems to be
>about twice as fast on writes of large files, and I am
>guessing this is because the data is cached. 

	Phew. This is a can of worms. In order to answer your
question accurately, I need more information. First, however,
I will answer your question directly: "no, there is no caching
done of NFS under ULTRIX"   Now for the "it depends" parts...

->	Does your server have PrestoCache? This is NVRAM on
the *file server* (not client) that makes NFS writes a few
hundred percent faster.

->	Are you running "biod"s on the client? These perform
a form of asynchronous NFS lying that permits higher I/O
thruput while giving somewhat asynchronous-like characteristics
to your client-side writes.

->	Are you measuring reads or writes? If you're measuring
reads, it's often faster, and for a lot of very cool and somewhat
odd reasons. When the server gets a block of a file for an NFS
client, it often triggers a read-ahead of the *next* block of
the file. So while the client is happily digesting the first block,
the server is already getting the next one! Then the client asks
for the next one, and >Zap< the server instantly replies with
the block out of memory.... And pre-fetches the next. Local disks
do some form of block prefetching, too, but they're contending
with local I/O and whatnot, and - even though your machine may
*look* like it's doing lots of things at once - it isn't. In
the NFS server read case, you actually can get 2 machines
working in a very sweet synchrony indeed.

->	Are you doing lots of small writes, or are you writing
reasonable sized chunks. NFS (sort of) delays writing out I/Os
to a specific block until the block is full, in some situations.
(otherwise writing code like:
	main() {
		fd = open("file",O_CREAT|O_WRONLY,0600);
		while(1)
			write(fd,"a",1);
	}

	would be utterly amazingly crawlingly slow)

	Since you're running 4.3, there are a *lot* of NFS I/O
enhancements on the server, that can make NFS service faster to
begin with.

	Are the disks on your server faster than the disks on
your client? If the server disks are a LOT faster than the client
disks, you can get some skewed results. For BSD filesystems, if
the filesystem on the client is nearly *full* the server can
be a lot faster.

	See what I mean? There's a lot of gotchas.

	ULTRIX doesn't do asynchronous NFS (it's unsupported),
other than with biods (read the man page) like other vendors
do. Some vendors do what they call "asynchronous NFS" in which
the server acks the write *before* it is written to stable
storage (usually the server's volatile buffer cache). This
violates the protocol but can give performance boost of up
to 700%. We don't do that. In fact, we don't even have kernel
support for it.


>Are there
>some parameters that can be tweaked to cause local data
>to be cached, to make local accesses as fast as nfs
>accesses?  (If so, would this be a bad idea?)

	accesses == reads, I assume?

	The way to improve read times on your local disk is to
increase the cache size. Edit the value of bufcache in your
system config file (/sys/conf/mips/MACHINENAME) and increase it
to maybe 20. Be *careful* to understand what you're doing because
you can make things worse if you just slap a big value in. Once
you're in system tuning land, you're in the land of tradeoffs and
there is NO such thing as a free lunch or performance boost.

	You can also enable cache_bufcache in param.c, which
can speed things up a little as well, by allowing parts of
your buffer cache table to be stored on the CPU's on chip
data cache.

>Are my findings consistent with anybody else's experience?

	Yes. The problem is that you're performance analyzing a
fairly complex configuration. I've seen what you're seeing several
times before, and for several different reasons. (Once, for example,
because I had a machine locally that was a MicroVAXII with Ra-81
disks, being served by a DECSystem3100 with a load of RAM, Rz56es,
a large buffer cache and [shh, don't tell Eric Schott] my kernel
whacked to do asynch NFS. I/O over the network was absurdly fast,
probably limited by the MicroVAX's ethernet board)

mjr.