From: joel@eco.twg.com (Joel Stevick)
Newsgroups: comp.os.vms,vmsnet.networks.tcp-ip,vmsnet.networks.tcp-ip.multinet,vmsnet.networks.tcp-ip.wintcp
Subject: Re: TGV MultiNet vs. WIN/TCP NFS Client
Keywords: WIN/TCP, TGV, NFS Client, VMS
Date: 8 Apr 92 21:00:16 GMT
Reply-To: joel@eco.twg.com (Joel Stevick)
Organization: The Wollongong Group (East Coast Operations)


David:

	I would like to comment on your response to Reece Pollack's
	BRIEF analysis.  You made some statements which I disagree with.
	In order to make this more digestable, I've removed the text 
	of the original article (by Reece) - I apologize if removing 
	the original text is in violation of accepted posting norms.

				- Joel Stevick

>
>	The semantics of segregating the file data from the
>	file attributes mimics the VMS filesystem semantics
>	where the attributes of the file (i.e. the file header)
>	are segregated from the file data.
>

The volume-structure management policy for a locally-attached disk is NOT 
a reasonable model for a distributed file system - the scope of operations 
performed are very different.  For instance, NFS Client does not
have to worry about allocation of physical disk resources, but it
does have to process page faults  - a task normally left to disk device 
drivers.

>
>	It is important to allow arbitrary ACLs to be attached
>	to any file.  ACLs for granting EXTRA access above and
>	beyond the UIC-based access are just a very small part
>	of the VMS ACL world.  Reece points out that you might
>	not WANT to allow ACLs because they could violate the
>	security policy on the server.  I will briefly mention
>	that only REAL security in NFS is that enforced by the
>	client.  Servers are pretty much wide open to anything
>	a client might want to do.  If you get the security ACL
>	semantics correct the VMS client would ensure the security
>	policy.  In any event -- the MultiNet NFS client allows
>	you to specify (when you mount the NFS server's file system)
>	whether or not to apply security ACL semantics.  The
>	ACL identifiers are applied in the context of the VMS client
>	system -- so to the extent that the ACLs would be meaningful
>	on a local disk they are meaningful on an NFS mounted disk.
>	Even if you don't want to allow EXTRA access -- due to security
>	policies on the server -- you CERTAINLY want to allow ACLs to
>	restrict access (at least on the VMS client) and to trigger
>	any appropriate security alarms.  You also need to support the
>	myriad other non-SECURITY ACLs on arbitrary files in order to
>	make the NFS server file system "feel" like a VMS file system.
>

Again you have assumed that the model for implementing local file systems
is meaningful in a distributed environment.  By implying that you support 
ACLs for security purposes means that the customer should feel secure
that file accesses from the context of his/her system will be controlled.
What happens in the case where a hard link (in a different directory)
is established to a file (with an associated alarm ACE) from a non-VMS 
client?  When the VMS client sees the new directory entry for the link 
will it also detect that the file has an alarm?  If not, then the
customer could be trusting that a file is being protected, when it is not. 
If this is the case, I hope that your documentation adequately warns customers 
of these security holes.  These kinds of problems do not occur on a local file 
system.  

>
>	Yes, PathWorks using the NFS client as a gateway to NFS
>	servers is a VERY useful function.  The resources used to
>	store the ACL information is typically much smaller than
>	the resources used to store the actual file itself.  Even
>	if it were as high as 20%, this is likely to be a trade-off
>	a customer would want to make for the transparent PathWorks
>	access to NFS servers.  Note, that nothing prevents a client
>	implementation to have mount option like /SEMANTICS=PATHWORKS
>	so that PathWorks generated ACLs only have to exist within
>	the "mind" of the client itself and don't have to be stored
>	on the NFS server at all.
>

I'm not sure how you are measuring resource consumption.  If you are
considering disk space as the resource, then perhaps 20% is a reasonable
number.  However, if you consider that file systems are configured
to support a FIXED quantity of files, having a meta file for each
PC file can become significant overhead.

Having a /SEMANTICS=PATHWORKS would place a SEVERE limitation on the use of 
VMS systems as gateways.  For example, if the ACL information 
is stored at the client, then you could not have multiple groups of PCs 
(i.e. through more than one VMS gateway) sharing files.

>
>	It is hard to imagine what kind of headache (other than what
>	Reece mentions below) is being referred to.  Files are files,
>	that is what the server is there for.
>

Customers typically acquire servers based on a projection of capacity
requirements.  It does not seem reasonable to tell custumers that
they must regen existing server file systems just because they
are introducing your VMS NFS Client into the environment.  This
would mean that they have to take their servers offline, back up
the file systems, re-make the file systems, and restore the
user files.  THIS IS A HEADACHE.

>
>	Now for the adminstrative burden on a UNIX NFS server...  In the
>	unlikely event that you have some of these unwanted files hanging
>	around (and over 2 years of having the MultiNet NFS Client in the
>	field has not brought any complaints about this) the following
>	very simple shell script (took me maybe 5 minutes to generate it)
>	will find all the files that don't also have a matching data file.
>	You can then do anything you like with the file (including delete it)
>	by just changing the "echo" line in the script to do something:
>

Consider the following (real) example:

One of our customers has a NETWARE-based PC network.  The
PC users operate on files that are managed using a VAX-based
document control system.  Word processing documents are created on a
VAX and then inserted into a document control library.  The documents
are then moved to an NFS disk which is served by a Novell server.  

A "monitoring" application performs some additional processing on the
documents and then moves them (via NETWARE) into an area which can be 
accessed by the PC users.  This process handles thousands of small documents
weekly.

These files are WORDPERFECT documents and are stored using 128-byte
fixed length records.  The meta file approach would not be useable
in this case.  Because the monitoring application knows nothing
about the meta files, the file system would quickly exhaust its resources
and user operations would be halted.  Would you suggest that the
customer pay for their software to be modified, or perhaps you
could provide an equivalent NETWARE script for cleaning up the
orphan meta files.  Most customers do not have resident NETWARE
expertise to consider either of these options.  THIS IS A HEADACHE.


>#!/bin/csh -f
>#
>#	This simple command procedure runs down a directory tree
>#	and finds any FDL file that has no accompanying data file
>#
>if $argv[1] == "CHECK-FILE" then
>	#
>	# This is done for each file FDL file that we find
>	#
>	set filename =  $argv[2]:h/`expr substr $argv[2]:t 7 1000`
>	if ! -e $filename then
>		echo $filename is missing for the FDL file $argv[2]
>	endif
>else
>	#
>	#	This searches for all the FDL files
>	#
>	find $argv[1] -name .\$fdl\$\* -exec $0 CHECK-FILE {} \;
>endif
>
>
>	If course if you want to it to run faster, a "C" program to
>	do this shouldn't be more than a page or 2 of code.
>
This shell script may work fine for UNIX, but what about NETWARE, or
any of the many other non-UNIX servers?  

>
>	A lot of work is done in the MultiNet NFS client to minimize the
>	need for these meta-files.  In reality you will find that this
>	tends to increase the number of files by a few percent.  The worst
>	case would be for something like PathWorks that wants to put ACLs
>	on everything -- but, as discussed above there are VERY good solutions
>	to this problem.
>

Do you describe in your documentation which file types require meta
files?  This is an important consideration for customers.  In the
case of our customer (described above) the files are stored using fixed
length 128-byte records.  Can you store preserve these attributes
without the use of meta files?  This is an important question considering
the volume of document processing performed.

>
>	The same scheme that minimizes the occurance of deleting
>	a file from the server side also serves to minimize this
>	occurance.  A UNIX user moving the file will not move the
>	hard link that has the version number -- certainly not
>	accidentally.  Of course, the client takes care of all the

Again you are assuming that file servers are UNIX-based.  What
about NETWARE, VM, etc.?  


>	accidentally.  Of course, the client takes care of all the
>	details when a file is moved using the VMS NFS client.
>	You will also note that the format of the meta-files were
>	carefully chosen to make recovery very easy in the unlikely
>	event of an accident.  The format is the same as a VMS FDL
>	File Definition Language file -- so VMS utilities can be used
>	analyze a similar file and get a meta-file that can be placed
>	on the server.
>

What if the meta file is lost, or the customer does not know from which
directory the file originated?  The customer may not know the original
RMS attributes for the file. THIS IS A HEADACHE.

>
>	You would need to allocate a fixed-size area for the meta-data
>	which would be VERY expensive to expand in a file were it ever
>	necessary.  Not being able to expand the meta-data area is a
>	problem -- as ACLs and other meta-data can grow beyond whatever
>	fixed size you decide to use.  Also note that just as in the
>	meta-file technique, you use up file system resources to store the
>	meta data.  The next problem, which Wollongong on partly solves
>

Since we don't have to be concerned about the management of on-disk
resources, the amount of storage required for meta data is quite small - much
less than the amount of space consumed by a typical server file system
"inode".  More importantly, the system manager does not have to regen
the file system to accomodate this meta data.  File systems are typically
configured into two regions - one for storing control information such
as "inodes", and a much larger region for general use.  Our meta
data resources are obtained from this larger pool.

>
>	meta data.  The next problem, which Wollongong on partly solves
>	is that there is a large class of VMS files that have identical
>	on-disk representations as their UNIX counterparts.  It is very
>	important to be able to share the data.  Just allowing a single
>	mapping from stream format to fixed-length records is not enough.
>	Your scheme does not work, for example, with VMS and Ultrix
>	bookreader files -- the MultiNet approach allows you to transparently
>	access Ultrix bookreader CDs from VMS bookreader.  We have run
>	across many examples of this.  Once you have to place the meta-data
>	in the data file, the UNIX applications will not be able to read
>	the files.
>                                      

I assume that you are refering to our /STREAM=<size> qualifier which
allows stream files to be represented to VMS NFS Client applications
as fixed length record files.  The reason that we implemented this
was that one of our customers has an application that was designed to
operate on files which have been obtained using an FTP command procedure.  
FTP stores them as fixed-length 512 byte record files.  The customer
did not want to have to pay someone to modify their application so that it
could operate on stream files.  Hence the /STREAM qualifier.

There is absolutely no meta data used under these circumstances.

>	Having a mount switch to force the conversion of fixed-length record
>	files to stream files is a problem when you have a mounted NFS disk
>	that you are using for several purposes, some of which require your
>	conversion in order to work and some of which break when the conversion
>	is done.  Forcing a VMS user (possibly through a system manager) to
>	make the decision on these semantics takes away from the transparency
>	people will want to see in the NFS client.
>

It would be easy enough to handle this through the use of device logical 
names.  However, I am not aware of any customers who have even encountered
this issue.

>	It also comes to mind that VMS file version limits are going to be
>	problem here as well.  When you do a VMS SET/FILE/VERSION_LIMIT=n
>	it sets a version limit for a particlar file in a particular
>	directory.  The semantics DO NOT allow you store that information
>	in the file itself -- this version limit is a property of the
>	directory entry, not the file.

An interesting point!  We do not claim to support version limit enforcement 
at the file-level.  We support version limit enforcement at the 
directory-level.  In this case RMS uses the version-limit field in the file 
header if there is none specified in the directory entry - we emulate this
field.  The interesting aspect to this question is that in our design we
did not completely eject the meta file approach.  If a file system is
mounted with version limit enforcement enabled (it is disabled
by default), then a meta-file (in that directory on the server) is
created to store the version limit information. 

>
>	This really only helps with NFS servers that have eNFS algorithms
>	that can delay updating certain file attributes on the server's
>	disk when there are other write requests coming in for the same file
>	before the initial write completes.  NFS servers using accelerator
>	boards and regular NFS servers don't really benefit from this.
>

We have found that on our in-house machines, pipelining consistently achieves 
a 50% improvement.  Among the servers tested are SUN 3/60, SUN Sparc I,
SUN Sparc II, and various DEC servers.  I have not encountered ANY file
servers where pipelining did not make a significant performance impact.

>
>	Unless I am mistaken the pipelining is accomplished by telling the
>	application that a write operation has been completed on the server
>	even though it has NOT.  If so, this is a problem -- as the semantics
>	of IO$_WRITEVBLK to a file say that the I/O operation does not complete
>	until the data is safely on the disk.  There are many applications,
>	like databases (and even RMS's indexed files) that can get corrupted
>	files on a client crash if this scheme is used.  The ability to have
>	more than one outstanding write request on a file on VMS is selected
>	(or not selected) by the application -- usually through RMS defaults.
>	RMS (or the applications themselves) do asynchronous I/O which would
>	allow a non-pipelined NFS client to get the same effect but only
>	when it was appropriate to.  So, a Multi-Threaded NFS client would
>	be able to get the same improvement in network bandwidth utilization
>	WITHOUT violating the semantics of the VMS file system.
>
This is true, and it is discussed in our documentation.

The pipelining feature was added to improve performance for large
sequential transfers, e.g. BACKUPs.  Customers don't have to be
concerned about configuring RMS parameters etc.

For cases where customers do not want to use pipelining, it can be disabled.

>>
>> 	Both NFS Client implementations automatically convert VMS text
>> 	files to Stream-LF for interoperation with non-VMS applications.
>>
>
>	I don't know if the WIN/TCP NFS Client allows you to disable this
>	on a per-mounted file system basis, but you need to be able to
>	disable it, as there are some small number of VMS applications that
>	just won't work with Stream-LF files.

Yes, we allow this feature to be disabled via /NOCONVERT.  This was
added to support CMS element files.


>   There are, I think, other issues in NFS client performance that have
>   not been touched upon.  The most important is the amount of information
>   the NFS client needs to extract from the NFS server before it can respond
>   to the VMS user.  The worst problem is when you first go to get a directory
>   listing, you have do an NFS lookup request on every file you find, as its
>   name may change on VMS depending on whether or not it is a directory file.
>   Even a directory with only a few hundred files can cause a significant delay
>   while the VMS NFS client beats up the NFS server trying to find out
>   information about all the files in the directory.  The MultiNet NFS client
>   has some very interesting algorithms in it and can typically respond to the
>   user after only looking at a couple of server files.
>

We also have such algorithms.  Any serious NFS Client design for VMS would
have to consider this.  We have implemented a configurable directory entry
caching scheme.  We have made it configurable so that the customer can 
trade off cache resource usage for response time.  For instance, operations
on small directories will not see a significant performance benefit from
caching, so there is no need to consume any resources.

>   When the number of files in a server directory becomes large (we have
>   customers storing many thousands of files in each directory) the 
>   computational complexity of just sorting the names of the files (they come 
>   from the server in random order and must be alphabetically sorted for VMS) 
>   is a problem.  The MultiNet NFS client has special algorithms that are 
>   highly tuned for for the kinds of data it is likely to see from the server.  
>   So, for example, a MicroVAX II (we keep one around for algorithm testing 
>   because it is so nice and slow) can start listing a directory that has 
>   3,000 files with about 2 seconds of computation.
>
>   Once you have seen a directory you can cache it so that later access
>   is fast.  The next problem occurs when a directory needs to be refreshed.
>   This can happen when a file is created or deleted in the directory by
>   another client -- the cache in the NFS client needs to be brought up to
>   date.  This also happens when the NFS client creates or deletes a file,
>   as you cannot be sure that some other client hasn't also done something to
>   the directory during the create/delete operation.  So a complete cache
>   refresh must be done to ensure a consistent view of the NFS server's file
>   system.  The MultiNet NFS client has an interesting algorithm that can
>   reduce the amount of work required to do this refresh by a couple of
>   orders of magnitude.

Again, any serious NFS Client implementation will address this.  Our
directory entry caching scheme prevents the redundant computations
that you are refering to.  During our initial field test, one of our
customers amused us by regularly storing 5000 entries in a single directory.  
The response time for creating and deleting files was very reasonable - 
on the order of a few seconds - on an active system.  It is important
to clarify that the "2 seconds" you mentioned above is "compute time".  This
is not the same as observed "wall clock response-time".


 - Joel