From: wen-king@cs.caltech.edu (Wen-King Su)
Subject: Re: FTP not timing out?
Organization: California Institute of Technology
Date: Fri, 10 Apr 1992 02:46:35 GMT

In article <1992Apr10.102228.2982@ucsvc.ucs.unimelb.edu.au> u8551044@ucsvc.ucs.unimelb.edu.au writes:
>Hello,
>subject: FTP not timing out?
[[ ftp didn't time out and exit after the server machine died ]]

The problem could be that "KEEPALIVE" is not enabled when socket is opened
by the ftp program.  It is often disabled by default to conserve bandwidth.

If you are interested in a more flexible solution, my I recommend that
you try the FSP software.  FSP is a connection-less, state-less, and
UDP-based internet archive server.  If the server dies during a transfer,
the transfer will automatically resume where it left off after the server
comes back.  Though the software is specifically designed for use in the
distribution of ftp-busting, high-demand materials, it will work quite
well for your application.  I have a copy of it on the public access ftp
archive on wuarchive.wustl.edu.  Compared to ftp, you will find the user
interface more suitable for shell scripts, and the software more suitable
for writing specialized client utilities.  Below is the INFO file that
comes with the software. 

=========================================================================

	FSP is a set of programs that implements a public-access archive
	similar to an anonymous-FTP archive.  It is not meant to be a
	replacement for ftp; it is only meant to do what anonymous-ftp
	does, but in a manner more acceptible to the provider of the
	service and more friendly to the clients. 

	Providing anonymous-FTP service can be costly --- each active
	session consumes one process slot in the OS and one stream socket
	entry in the network sub-system.  The servers can also run
	concurrently, adding to the system load.  A popular archive site
	can easily be overwhelmed as a result.  Some were forced to
	shutdown or and some impose inconvienent access restrictions. 

	Unlike FTP, FSP is connection-less and virtually state-less.  One
	server handles requests from all clients machines.  Each active
	client machine takes up 16-bytes in a dynamically extensible
	table.  Since only one server runs at any time, the load added
	to the server machine is no more than one.

	In exchange for allowing site operators to keep their sites open
	and do away with cumbersome access restrictions, this is what the
	clients accept with FSP: 

	 1) Lower transfer rate.  The maximum rate is 1 kbyte per UDP
	    message round-trip time between the client and the server.
	
	In addition to the potential for more abundant sites and more
	accessible sites, this is what the clients gain with FSP:

	 1) Robustness.  Since FSP is connectionless, flucturations in
	    the network will not abort a FSP transaction.  Furthermore,
	    the 16-bytes of data for each client can be regenerated at
	    any point during any transaction.  Thus, if the server goes
	    down at any point during a transaction, the transaction will
	    resume when the server is restarted.  (like NFS) 

	 2) Friendlier user interface.  FSP does not have its own command
	    interpretor like FTP.  Since it is connectionless, there is
	    no reason to carry much information from one command to the
	    next, and the commands can all be made into individual unix
	    programs.  For instance, there is one program you run to list
	    the directory and another you run to download a file. 

	 3) Client protection.  FSP oversees a directory structure similar
	    to that of an anonymous-FTP.  However, a directory created
	    via FSP transaction is owned by the client machine that issued
	    the creation request.  The client can create and delete files
	    and subdirectories in that directory.  In addition, the client
	    can enable any of the two attributes for that directory: 

		A) Give all other clients the permission to create files
		   and subdirectries.

		B) Give all other clients the permission to delete files
		   and subdirectories.

	    Note: A subdirectory can be deleted if it is empty and the
		  client owns the subdirectory.

	 4) Server protection.  FSP server does not spawn sub-programs.
	    It will accept only paths that are downward relative to its
	    designated working directory.  On systems with symbolic links,
	    the server will follow symbolic links, but it does not follow
	    uplinks ("..").  Clients cannot create symbolic links and
	    care should be taken so that other users on the server machine
	    cannot create symbolic links in the server's work space. 

	    It is also fairly difficult to formuate an attack to force a
	    shutdown of a FSP site by actions of a rogue site.  About the
	    only way to distrupt a FSP service is to flood the FSP site
	    with network packets.  FSP server prevents itself from
	    'counter-flooding' by filtering for legitimate requests using
	    the following method:

		A) Each request message contains a key.  For each client,
		   server database contains the keys to be used for the
		   next client request and for the previous client request.

		B) If the next request does not contain a key that matches
		   either of the two keys, it is accepted only if at least
		   one minute has elapsed since the last time a request
		   is accepted.  If the key does match the old key
		   (retransmit) it is accepted if the elapse time is
		   greater than 3 seconds.

		C) Every request message accepted is acknowledged with
		   one reply message.  The reply message contains a new
		   key to used for the next request.  The new key is
		   computed by the server with a pseudo-random number
		   generator. 

	    Flooding is a ballant violation of network etiquette because
	    a site can be subjected to flooding attack whether it has FSP
	    running or not, and flooding congests every link and gateway
	    between the rogue client and the server.  As a further measure
	    of protection, the server loads a table of rogue clients on
	    startup.  The server will not respond to requests from any of
	    those clients.

    The software set:

	common_def.h	This C header file contains definitions common to
			both the server code and the client code.

	client_def.h	This C header file contains definitions for the
			client code.

	server_def.h	This C header file contains definitions for the
			server code.

	udp_io.c	This file contains the lowest level routines that
			deal with the unix inet sockets.  This file is
			used by both the server code and the client code.

	server_main.c	Main routine and dispatch loop for the server.
	server_host.c	Routines for maintaining client database.
	server_file.c	Routines for file i/o.
	server_lib.c	Routines for inet socket i/o.

	client_lib.c	Core routines of the client library.
	client_util.c	Supplementry routines of the client library.
	client_lock.c	udp packet multiplexing mechanism.

	bsd_src/	Directory containing additional sources derived
			from those in public archive on uunet.uu.net.  It
			contains a BSD random/srandom routine, a modified
			BSD globbing routine, a modified "ls" source.

	fcdcmd.c	These compiles into individual client utilities.
	fgetcmd.c	Those with a "cmd" in their name will do their
	flscmd.c	own globbing on their argv base on directory
	fprocmd.c	information obtained from the server.
	frmcmd.c
	frmdircmd.c
	fcatcmd.c
	fmkdir.c
	fput.c
	fver.c
	fgrab.c

    Compilation:

	FSP has been compiled and tested on a SS-2 running SunOs 4.1.1,
	a HP-9000 running HP UNIX, a VAX-780 running 4.3-tahoe, and a 386
	box running system-V UNIX with old Excelan ethernet interface.
	It has also been compiled on a variety of machines by over a
	hundred users across the net. 

	To compile the software, you must first successfully complete a
	"make" in the bsd_src directory.  You may have to change a few
	files.  In particular, you may have to edit "Makefile" and "tweak.h"
	in bsd_src directory. 

	When that is done, you can edit the Makefile on the top directory
	and run "make" in the top directory.  You may have to read through
	the rest of this document first before making changes to the Makefile.

    Server Administration:

	The only things you need for setting up a FSP server is a work
	directory for the service and and the FSP server itself (fspd).
	fspd can run independently or it can be run under inetd.  When
	running independently, fspd waits for messages through a UDP
	socket whoes port number is defined in the Makefile.  When running
	under inetd, fspd is involked as in.fspd.  inetd will spawn fspd
	when a message arrives for the FSP socket.  The fspd process will
	take over and stick around to wait on additional messages.  After
	it has become idle for 2 minutes, fspd will exit and return control
	to inetd. 

	Sample setup for inetd operation:

	    In /etc/services file:

		fsp             21/udp          fspd

	    In /etc/inetd.conf file:

		fsp dgram   udp wait ftp /usr/etc/fspd in.fspd

	    In this sample, the same port number for ftp is used for the
	    fsp socket.  There will not be a conflict because ftp uses
	    stream protocol, and fsp uses UDP protocol.  The fspd program
	    in this example is ran under user 'ftp'. 

	In addition, fspd will accept these flags:

	    -h absolute_path    Set fsp work directory.  Overrides the
				compiled-in default.

	    -p udp_port_number  Set UDP port number.  Overrides the
				compiled-in default.

	    -u uid_number       Assume this uid after startup.  If present,
				fspd will attempt a setuid() to this uid
				number.  It will exit if setuid() fails.

	    -d                  Turn on debug mode.  The stdio files will
				remain open in debugging mode.

	When fspd starts, it chdir to its work directory where it looks
	for (and reads in if found) a list of internet numbers in the
	standard 4-part form: ddd.ddd.ddd.ddd in the file ".ROGUE_HOSTS".
	This file is prepared by the FSP maintainer, and is used to
	indicate that fspd should not respond to any requests from these
	machines.   After that, it begins to service any requests it gets
	on the UDP socket.

	If a file .OWN.XXXXXXXX, where XXXXXXXX is an 8-digit hex number,
	exists in a directory in fspd's work space, the directory is owned
	by the machine whoes inet number is XXXXXXXX, where the number
	is printed as a hexadecimal number.  If no such file exists, the
	directory has no owner.  (Note, the 'dot' files are hidden from
	clients).

	If the file .FSP_OK_DEL does not exists in a directory, only the
	owner is allowed to remove items from that directory. 

	If the file .FSP_OK_ADD does not exists in a directory, only the
	owner is allowed to add items into that directory. 

	Thus, you typically want to protect the top directory by leaving
	out the .FSP_OK_DEL, .FSP_OK_ADD files, and .OWN.XXXXXXXX files
	in the top directory. 

	Clients do not get to read the directory information directly.
	Instead, fspd maintains a directory listing in the file .FSP_CONTENT
	in each directory.  When a client requests information for a
	directory, the .FSP_CONTENT file is created if it doesn't exist,
	and it is rebuilt if it is out of date.  The information is
	accessed by having the client read the directory listing file.
	Care is taken so that the client will not get corrupted entries
	when the directory is changed while the listing is being read. 

	Files being uploaded are first written to a temporary file in the
	work directory: .TXXXXXXXXYYYY where XXXXXXXX is the inet number
	of the client, and YYYY is the port number of the client program.
	When upload is compelete, the file is moved into the intended
	location. 

	An 'alarm' interrupt will cause fspd to dump its current client
	database into the file .HTAB_DUMP in the work directory.  This
	can be useful for debugging and for catching rogue clients.

    Client utilities:

	All inter-command states are kept in these four shell environment
	variables.

	    FSP_PORT		Port number of the fspd you wish to contact.
	    FSP_HOST		Host name or number of the fspd.
	    FSP_DIR		Your current working directory in the archive.

	When multiple client utilities are run at the same time on the
	same client machine, packet multiplexing mechanisms can be used
	to enable concurrent access to the same fsp database.  If none
	of the mechanisms are selected at compile time, FSP_LOCALPORT
	can be used to ensure that only once client utility can run at
	any time.  In this case, FSP_LOCALPORT can be set to any port
	number not current used on the client machine.

	FSP_TRACE can be set if you want status reports be printed while
	files are being transferred.  FSP_DELAY variable can be used to
	set the retransmit interval for client utilities (in thousandth
	of a second).  The retransmit rate is adjusted in an exponential
	manner, until the retry rate reaches 5 mintes per retry.

	FSP_BUF_SIZE can be set to a positive number less than or equal
	to 1024.  When set, it determines the size of data to be send for
	each request during file and directory information transfer.  The
	default is 1024.  Some sites are connected via links that cannot
	transmit buffers containing 1024 bytes of data in addition to the
	header information.  Setting FSP_BUF_SIZE to a lower value will
	allow these sites to access fsp archives.

	A typical setup looks like this:

	    setenv FSP_PORT	 21
	    setenv FSP_HOST	 131.215.131.97
	    setenv FSP_DIR	 /
	    setenv FSP_TRACE
	    setenv FSP_DELAY	 3000
	    setenv FSP_BUF_SIZE  1024

	(All examples will be in csh.  However, it is assumed that similar
	 things can be done with other shells)

	For commands that do globbing using remote directory info, normal
	shell globbing needs to be turned off.  In csh, it can be done
	with a set of aliases: 

	    alias fcd setenv FSP_DIR \`\(set noglob\; exec fcdcmd \!\*\)\`
	    alias fls    \(set noglob\; exec flscmd    \!\*\)
	    alias fget   \(set noglob\; exec fgetcmd   \!\*\)
	    alias fgrab  \(set noglob\; exec fgrabcmd  \!\*\)
	    alias fcat   \(set noglob\; exec fcatcmd   \!\*\)
	    alias frm    \(set noglob\; exec frmcmd    \!\*\)
	    alias frmdir \(set noglob\; exec frmdircmd \!\*\)
	    alias fpro   \(set noglob\; exec fprocmd   \!\*\)
	
	In addtion, this alias is useful:

	    alias fpwd echo \$FSP_DIR on \$FSP_HOST port \$FSP_PORT

	Commands:

	    fver	display server's version number.
	    fcd		change current remote directory, like cd.
	    fls		list directory.  works like ls.
	    fget	get the named files.
	    fgrab	get the named file and delete it from remote directory.
	    fput	put the named files.
	    fcat	get the named files and send them to stdout.
	    fmkdir	make named directories.
	    frm		delete named files.
	    frmdir	delete named directories.

	    fpro	no arg: display directory protection modes.
			    +c: give others permission to create new items.
			    -c: deny others permission to create new items.
			    +d: give others permission to delete old items.
			    -d: deny others permission to delete old items.

    ***********************************************************************

    This is a free software.  Be creative; make your own macros and tools
    and let me know of any bugs and suggestions.
Xref: sparky comp.protocols.tcp-ip:2534 comp.sys.sun.admin:2542