Mail-System-Version: <VAX-MM(301)+TOPSLIB(155)@TGV.COM>
Postal-Address: TGV Inc.; 603 Mission St.; Santa Cruz, CA 95060
Phone: (408) 427-4366 or (800) TGV-3440
Newsgroups: vmsnet.networks.tcp-ip.wintcp
Date:         Tue, 7 Apr 1992 10:03:22 -0700
From:         "David L. Kashtan" <KASHTAN@TGV.COM>
Subject:      re: TGV MultiNet vs. WIN/TCP NFS Client
X-To:         info-multinet@TGV.COM, info-vax@sri.com,
              WINTCP-L@ubvm.cc.buffalo.edu

Reece Pollack of The Wollongong Group in comparing their approach to
a VMS NFS Client vs TGV's talks mostly about the methodologies used
in storing VMS RMS file attributes on NFS file servers.

He (somewhat) correctly states that MultiNet's NFS client uses a second
file to describe RMS file attributes while the WIN/TCP NFS client stores the
attributes as part of the file data at the start of the file.  He then
goes on analyze what he considers the good and bad aspects of these 2
techniques:

>     1. Store the extended file attributes in a meta file.
>     -----------------------------------------------------
>
> 	For other than stream files (the natural file organization for
> 	NFS), a related file is created to store the RMS information.
>
> 	This is the scheme used by TGV.
>
> 	The primary advantage to this approach is that ACLs can be
> 	attached to any file. However, this advantage is limited,
> 	since applying Access Control Lists to files stored in a
> 	remote security domain may not be meaningful. In fact, some
> 	adminstrators may disapprove of the use of ACLs since it
> 	could violate the security policy used at the server.

There are MANY advantages to this technique:

	The semantics of segregating the file data from the
	file attributes mimics the VMS filesystem semantics
	where the attributes of the file (i.e. the file header)
	are segregated from the file data.

	It is important to allow arbitrary ACLs to be attached
	to any file.  ACLs for granting EXTRA access above and
	beyond the UIC-based access are just a very small part
	of the VMS ACL world.  Reece points out that you might
	not WANT to allow ACLs because they could violate the
	security policy on the server.  I will briefly mention
	that only REAL security in NFS is that enforced by the
	client.  Servers are pretty much wide open to anything
	a client might want to do.  If you get the security ACL
	semantics correct the VMS client would ensure the security
	policy.  In any event -- the MultiNet NFS client allows
	you to specify (when you mount the NFS server's file system)
	whether or not to apply security ACL semantics.  The
	ACL identifiers are applied in the context of the VMS client
	system -- so to the extent that the ACLs would be meaningful
	on a local disk they are meaningful on an NFS mounted disk.
	Even if you don't want to allow EXTRA access -- due to security
	policies on the server -- you CERTAINLY want to allow ACLs to
	restrict access (at least on the VMS client) and to trigger
	any appropriate security alarms.  You also need to support the
	myriad other non-SECURITY ACLs on arbitrary files in order to
	make the NFS server file system "feel" like a VMS file system.

>
> 	The other major use for ACLs is to allow the NFS client to
> 	serve as a file system gateway. For instance, a network
> 	of PCs accessing a VAX using PathWorks for VMS could
> 	be granted access to NFS-served file systems. This could
> 	be accomplished by exporting an NFS Client disk to the
> 	PathWorks for VMS server software. However, this too
> 	has a significant disadvantage: such software typically
> 	uses ACLs to store information such as user identification;
> 	as the users on the PC network begin to create documents,
> 	a proliferation of meta-files can appear at the remote NFS
> 	server, consuming file system resources ...

	The uses of ACLs are only limited by the imagination of
	of the VMS application programmer.  TGV felt that users
	were going to want to look at the NFS server as just another
	VMS file system and it was not appropriate to say that just
	because an application wants to use ACLs for something it
	cannot run over NFS.

	Yes, PathWorks using the NFS client as a gateway to NFS
	servers is a VERY useful function.  The resources used to
	store the ACL information is typically much smaller than
	the resources used to store the actual file itself.  Even
	if it were as high as 20%, this is likely to be a trade-off
	a customer would want to make for the transparent PathWorks
	access to NFS servers.  Note, that nothing prevents a client
	implementation to have mount option like /SEMANTICS=PATHWORKS
	so that PathWorks generated ACLs only have to exist within
	the "mind" of the client itself and don't have to be stored
	on the NFS server at all.

>						... and creating an
> 	(possibly unanticipated) administrative headache.
>

	It is hard to imagine what kind of headache (other than what
	Reece mentions below) is being referred to.  Files are files,
	that is what the server is there for.

>
> 	We have identified the following disadvantages to the use of
> 	meta-files:
>
> 	  - If a file is deleted by a non-VMS node (for example,
> 	    from the server), the meta file is left behind, and must
> 	    be explicitly deleted. This presents an administrative
> 	    burden.
>

	In designing the NFS client this event was considered.  The
	chances of this happening are actually quite slim.  Lets look
	at the most common case -- a UNIX-based NFS server.  Except in
	the case of a VMS file stored on the NFS server only having
	a ";1" version (in which case no version numbers are generated
	on the NFS server), the VMS files are stored on the server with
	VMS version numbers attached to them in the form:
			name.ext;ver
	That ";" for the version number separator was not just chosen
	because it is the one that VMS uses to separate version numbers.
	It is also a UNIX shell meta-character and MUST be quoted for
	the filename to be specified from the UNIX side.  This pretty much
	eliminates the accidental removal of these files.  A file of the
	form:
			name.ext
	Is also kept "hard" linked to the highest version of the VMS file.
	This allows UNIX users to see the highest version file in a more
	"natural" form and when a UNIX user removes the file the one with
	the version number still stays around unless explicity removed
	(which requires the above-mentioned shell-quoting of the ";").

	Now for the adminstrative burden on a UNIX NFS server...  In the
	unlikely event that you have some of these unwanted files hanging
	around (and over 2 years of having the MultiNet NFS Client in the
	field has not brought any complaints about this) the following
	very simple shell script (took me maybe 5 minutes to generate it)
	will find all the files that don't also have a matching data file.
	You can then do anything you like with the file (including delete it)
	by just changing the "echo" line in the script to do something:

#!/bin/csh -f
#
#	This simple command procedure runs down a directory tree
#	and finds any FDL file that has no accompanying data file
#
if $argv[1] == "CHECK-FILE" then
	#
	# This is done for each file FDL file that we find
	#
	set filename =  $argv[2]:h/`expr substr $argv[2]:t 7 1000`
	if ! -e $filename then
		echo $filename is missing for the FDL file $argv[2]
	endif
else
	#
	#	This searches for all the FDL files
	#
	find $argv[1] -name .\$fdl\$\* -exec $0 CHECK-FILE {} \;
endif


	If course if you want to it to run faster, a "C" program to
	do this shouldn't be more than a page or 2 of code.

>
> 	  - Meta files consume valuable file system resources. Most
> 	    systems are configured to support a fixed number of files.
> 	    Each meta-file consumes a resource that would otherwise be
> 	    available for allocation to user files.
>

	A lot of work is done in the MultiNet NFS client to minimize the
	need for these meta-files.  In reality you will find that this
	tends to increase the number of files by a few percent.  The worst
	case would be for something like PathWorks that wants to put ACLs
	on everything -- but, as discussed above there are VERY good solutions
	to this problem.

>
> 	  - If a file is moved by a non-VMS node without explicitly
> 	    moving the associated meta-file, the RMS attributes will
> 	    be lost, rendering the contents of the file unusable. In
> 	    order to use the file, the meta-file must be located or
> 	    an equivalent one created using FDL -- this is a major
> 	    inconvenience for users.
>

	The same scheme that minimizes the occurance of deleting
	a file from the server side also serves to minimize this
	occurance.  A UNIX user moving the file will not move the
	hard link that has the version number -- certainly not
	accidentally.  Of course, the client takes care of all the
	details when a file is moved using the VMS NFS client.
	You will also note that the format of the meta-files were
	carefully chosen to make recovery very easy in the unlikely
	event of an accident.  The format is the same as a VMS FDL
	File Definition Language file -- so VMS utilities can be used
	analyze a similar file and get a meta-file that can be placed
	on the server.

>
> 	It is our understanding that TGV has implemented an interesting
> 	parsing scheme which allows the Multinet NFS Client to store
> 	certain RMS file types (e.g. object modules and executables)
> 	without the necessity of a related meta-file.
>

	Now we are getting to the REAL thing that makes the meta-file
	technique really workable.  We designed the MultiNet NFS client
	so that it hardly ever has to create any meta-files.  It has a
	large set of default attributes based on the parse of the file
	name.  If the attributes of a file match the default attributes
	for the parse that was done, then no meta-file is required.  Of
	course, this does mean that there is a goodly amount of code in
	NFS client for managing the meta-files -- since the act of renaming
	a file may create, delete or rename a meta-file depending on the
	source/destination filenames.

	I would note that computationally this is not very expensive,
	as the parse rules are compiled into a Deterministic Finite-State
	Automaton at client startup time.

>
>
>
>     2. Store the extended file attributes in the first several bytes
>        of the file as meta-data.
>     -----------------------------------------------------------------
>
> 	This means that, for other than stream files, the RMS attributes
> 	are written in the first several bytes of a newly created file.
>
> 	This is the scheme used by Wollongong.
>
> 	In order to prevent a random pattern at the start of a native
> 	NFS file to be mistaken for RMS meta-data, the first few bytes
> 	of meta-data contain encoded information keyed from the file
> 	identifier.
>
> 	There are cases when fixed-length record files created by VMS
> 	applications must be shared with applications running at the
> 	remote server; an example is WordPerfect. If the files were
> 	to be stored with the RMS information, they would not be
> 	interpretable by the same application running on a non-VMS
> 	platform. To solve this problem, a usage mode is provided in
> 	which fixed-length record files may be transparently converted
> 	to stream format, and stored without any meta-data. If it is
> 	necessary for the file to retain its record structure (and we
> 	know of no such case), we also provide a usage mode in which
> 	stream files can be represented to the NFS client applications
> 	as fixed-length record files - but this should not be necessary.
>
> 	The main disadvantage to this approach is that ACLs may not be
> 	assigned to files. However, DDIF semantic ACEs are supported.
>

	We considered this in-band approach early-on in the NFS client
	design process and discarded it for several reasons.

	You would need to allocate a fixed-size area for the meta-data
	which would be VERY expensive to expand in a file were it ever
	necessary.  Not being able to expand the meta-data area is a
	problem -- as ACLs and other meta-data can grow beyond whatever
	fixed size you decide to use.  Also note that just as in the
	meta-file technique, you use up file system resources to store the
	meta data.  The next problem, which Wollongong on partly solves
	is that there is a large class of VMS files that have identical
	on-disk representations as their UNIX counterparts.  It is very
	important to be able to share the data.  Just allowing a single
	mapping from stream format to fixed-length records is not enough.
	Your scheme does not work, for example, with VMS and Ultrix
	bookreader files -- the MultiNet approach allows you to transparently
	access Ultrix bookreader CDs from VMS bookreader.  We have run
	across many examples of this.  Once you have to place the meta-data
	in the data file, the UNIX applications will not be able to read
	the files.

	Having a mount switch to force the conversion of fixed-length record
	files to stream files is a problem when you have a mounted NFS disk
	that you are using for several purposes, some of which require your
	conversion in order to work and some of which break when the conversion
	is done.  Forcing a VMS user (possibly through a system manager) to
	make the decision on these semantics takes away from the transparency
	people will want to see in the NFS client.

	It also comes to mind that VMS file version limits are going to be
	problem here as well.  When you do a VMS SET/FILE/VERSION_LIMIT=n
	it sets a version limit for a particlar file in a particular
	directory.  The semantics DO NOT allow you store that information
	in the file itself -- this version limit is a property of the
	directory entry, not the file.
>
>
>     In addition to the difference in methodologies described above, there
>     are some other implementation features worth mentioning:
>
> 	The Wollongong NFS Client supports RPC pipelining for write
> 	operations, providing significantly improved network bandwidth
> 	utilization (estimated at 50% better utilization).
>
	This really only helps with NFS servers that have eNFS algorithms
	that can delay updating certain file attributes on the server's
	disk when there are other write requests coming in for the same file
	before the initial write completes.  NFS servers using accelerator
	boards and regular NFS servers don't really benefit from this.

	Unless I am mistaken the pipelining is accomplished by telling the
	application that a write operation has been completed on the server
	even though it has NOT.  If so, this is a problem -- as the semantics
	of IO$_WRITEVBLK to a file say that the I/O operation does not complete
	until the data is safely on the disk.  There are many applications,
	like databases (and even RMS's indexed files) that can get corrupted
	files on a client crash if this scheme is used.  The ability to have
	more than one outstanding write request on a file on VMS is selected
	(or not selected) by the application -- usually through RMS defaults.
	RMS (or the applications themselves) do asynchronous I/O which would
	allow a non-pipelined NFS client to get the same effect but only
	when it was appropriate to.  So, a Multi-Threaded NFS client would
	be able to get the same improvement in network bandwidth utilization
	WITHOUT violating the semantics of the VMS file system.

	We are in the process of having the MultiNet NFS client use the
	same VMS kernel-based RPC transport that the MultiNet NFS server uses.
	The client will be able to dynamically choose the best RPC method based
	on certain parameters.  This will allow VMS asynchronous I/O to be
	mapped to these multiple-outstanding RPC requests without compromising
	file integrity and will benefit from the same 40% boost in performance
	that the VMS kernel-based RPC transport brings to the NFS Server.


>
> 	Both NFS Client implementations automatically convert VMS text
> 	files to Stream-LF for interoperation with non-VMS applications.
>

	I don't know if the WIN/TCP NFS Client allows you to disable this
	on a per-mounted file system basis, but you need to be able to
	disable it, as there are some small number of VMS applications that
	just won't work with Stream-LF files.

>
> 	Both NFS Client implementations provide a VMS-mode when interacting
> 	with a compatable VMS-based NFS Server. This mode provides enhanced
> 	capabilities when used between two VMS systems. However, the TGV
> 	and Wollongong VMS-modes are not compatable with each other.
>

	True.



There are, I think, other issues in NFS client performance that have
not been touched upon.  The most important is the amount of information
the NFS client needs to extract from the NFS server before it can respond
to the VMS user.  The worst problem is when you first go to get a directory
listing, you have do an NFS lookup request on every file you find, as its
name may change on VMS depending on whether or not it is a directory file.
Even a directory with only a few hundred files can cause a significant delay
while the VMS NFS client beats up the NFS server trying to find out
information about all the files in the directory.  The MultiNet NFS client
has some very interesting algorithms in it and can typically respond to the
user after only looking at a couple of server files.

When the number of files in a server directory becomes large (we have
customers storing many thousands of files in each directory) the computational
complexity of just sorting the names of the files (they come from the server
in random order and must be alphabetically sorted for VMS) is a problem.  The
MultiNet NFS client has special algorithms that are highly tuned for for the
kinds of data it is likely to see from the server.  So, for example,
a MicroVAX II (we keep one around for algorithm testing because it is so nice
and slow) can start listing a directory that has 3,000 files with about
2 seconds of computation.

Once you have seen a directory you can cache it so that later access
is fast.  The next problem occurs when a directory needs to be refreshed.
This can happen when a file is created or deleted in the directory by
another client -- the cache in the NFS client needs to be brought up to
date.  This also happens when the NFS client creates or deletes a file,
as you cannot be sure that some other client hasn't also done something to
the directory during the create/delete operation.  So a complete cache
refresh must be done to ensure a consistent view of the NFS server's file
system.  The MultiNet NFS client has an interesting algorithm that can
reduce the amount of work required to do this refresh by a couple of
orders of magnitude.

TGV NFS Client also uses the NFS Lock Manager to allow for file locking
in an NFS environment.  The ability to open a file and restrict access to
others while the file is being manipulated is an important VMS feature that
also maps into the NFS world.  It is amusing to have one VAX do the following:
	$ OPEN/WRITE FOO "FOO.DAT"
On an NFS volume and have another VAX 3,000 miles away do the following:
	$ DELETE FOO.DAT;
And have the 2nd VAX get a "file locked by another user" message.

The TGV NFS Client and Server both do NFS over TCP (in addition to UDP) for
use in Wide-Area networks.

*** NOTE *** Some of the TGV NFS Client algorithms I have mentioned are not
	     in the current 3.0 release but will be in the imminent 3.1 release.

David
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU