Date: Wed, 2 Sep 92 20:48:53 PDT
From: benseb@SDSC.EDU (Booker Bense)
Original-To: gopher-news@boombox.micro.umn.edu
Subject: Additional Support for Anonymous Ftp gateways
Newsgroups: comp.infosystems.gopher
Distribution: comp

-Howdy,

	Unfortunately for me, SDSC's anonymous ftp site is a VMS
machine. I've been working for some time to make it and some other
very cool stuff available via gopher. The problem is that many
ftp servers ( and some clients ! ) are not Unixy enough to work with
the gopher ftp gateway. What I've done is set up gopher to use a
gopherftp file something like :

#
#	This is a sample gopherftp file
#
# The format is :
# host:dir list command:field to examine:dir special chars:???other fields:
#	host - the machines domain name
#	dir  - the appropriate ftp list command
#	field to examine - the SPACE delimited field to
#	                    examine on the return line
#	dir special chars - directories end with these chars
# For example  here's the default "unixy" file
default.edu:ls -F:1:/:
# For a multinet VMS machine
sdsc.edu:nlist:1:.dir:
# for some less "unixy" machines
fred.edu:ls -F:9:/:
--------------EOF----------------------------------
I've also fixed it to understand VMS pathnames so you can have
something like this


#
Type=1
Name=SDSC FTP
Path=ftp:sdsc.edu@[.sdscpub]
Host=+
Port=+

and it will descend the directories and work correctly!!!!  It's set
up so if that it doesn't find the host it uses a default setting that
you specify at compile time ( I had to have this for the newer ftp
clients. ) Perhaps the gopherftp file is something that should
go in the gopherd.conf file. This code needs to be cleaned and merged
into the gopher1.03 stuff,but it works with the 1.2 gopherd on a 
NeXT. One possible change I have in mind is to also put an entry
in the format for what fields you want to go in the Name. Some ftp
servers will give the file name and a 1-line description of the file,
the size of the file might also be a nice thing to know. For a final
entry you could specify which fields would be used for the Name entry
and it what order with a format string. Something like:

fred.edu:ls -F:9:/:9,5:%s %s bytes:

-Anyway, Here are the initial patches to gopher1.02. Let me know 
what you think!!

- Booker C. Bense               "News, News, It's long, It's ASCII?,  
prefered: benseb@grumpy.sdsc.edu   It's Wood. News, News, It's better
NeXT Mail: benseb@next.sdsc.edu      than bad, It's Good!"  Ren&Stimpy

-------------Cut-Here------8<---------8<-------------------
===================================================================
RCS file: /Booker/CVSRoot/gopher/Makefile.config,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 Makefile.config
41c41
< CLIENT_HOST	= gopher.micro.umn.edu
---
> CLIENT_HOST	= next.sdsc.edu
61,62c61,62
< SERVEROPTS	= #-DDL # -DLOADRESTRICT
< DLPATH		= /home/mudhoney/lindner/src/describe
---
> #SERVEROPTS	= #-DDL # -DLOADRESTRICT
> #DLPATH		= /home/mudhoney/lindner/src/describe
94c94
< DOMAIN		= .micro.umn.edu
---
> DOMAIN		= 
105c105
< SERVERDATA	= /home/mudhoney/gopher-data
---
> SERVERDATA	= /LocalLibrary/Gopher/gopher-data
107c107,113
< 
---
> #-------------------------------------------------------------------
> #
> # Local SDSC mods 
> #
> LOG_FILE        = /LocalLibrary/Gopher/log/logfile
> ACCESS_FILE     = /LocalLibrary/Gopher/access/main.gopher
> GOPHERFTP_FILE  = /LocalLibrary/Gopher/access/gopherftp
===================================================================
RCS file: /Booker/CVSRoot/gopher/conf.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 conf.h
70c70
< #define BUMMERSTR "We cannot off campus allow connections to this server. Sorry"
---
> #define BUMMERSTR "We cannot allow off-site connections to this server. Sorry"
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/Makefile,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 Makefile
68a69,71
> 	-DLOG_FILE=\"$(LOG_FILE)\"\
> 	-DACCESS_FILE=\"$(ACCESS_FILE)\"\
> 	-DGOPHERFTP_FILE=\"$(GOPHERFTP_FILE)\"\
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/daemon.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 daemon.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:03 $
<  * $Source: /Booker/CVSRoot/gopher/gopherd/daemon.c,v $
---
>  * $Author: jmsellens $
>  * $Revision: 1.2 $
>  * $Date: 1992/02/10 07:36:28 $
>  * $Source: /source2/gopher/gopher0.7/server/RCS/daemon.c,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/ftp.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 ftp.c
1a2,13
>         8/19/92
> 	Booker C. Bense 
> 	EVIL hack to allow gopher to use VMS ftp entries
> 	and to allow it to adapt to more non-standard servers
> 	This should grok things like
> 	ftp:sdsc.edu@[.sdscpub.next.graphics]
> 	ftp:sdsc.edu@[.sdscpub.next.graphics]imtool_v_2_0_cpio.z
> 	ftp:sdsc.edu@pub/gopher/
> 
> 	It also needs a gopherftp parse description file.
> 
> 		
7,18d18
< *	To be run by inetd.
< *	For installation, read the companion README.... or
< *	if you don't feel like doing that, the brief instructions
< *	are below:
< *	
< *	First edit the local parameters in defines below.
< *	Then do a "make g2fd"
< *	Then place the binary (g2fd) in some reasonable place,
< *		eg in /usr/local/bin
< *	Finally, edit /etc/services and /etc/servers or
< *	 	/etc/inetd.conf as the case may be.  Kill
< *		and restart inetd.
19a20
> *
22a24
> #include "ftp.h" 
47a50,53
> int             Icky_VMS = FALSE; 
> parse_ftp *Parse;                               /* Ftp Parse Structure */ 
> parse_ftp *ReadGopherFtp();                     /* What Reads it */ 
> 
54c60
<      
---
> 
60a67
>      
63c70,72
< 	  Abort("Not a valid ftp query.");
---
> 	  Abort("Not a valid ftp query."); 
> 
>      /* check for icky vms pathname */ 
64a74,87
>      if ( strchr(query,'[') != NULL )
>        {
> 	 if ( strchr(query,']') != NULL )
> 	   {
> 	     Icky_VMS = TRUE; 
> 
> 	   }
> 	 else
> 	   {
> 	     Abort("Not a valid vms ftp query."); 
> 	   }
>        }
> 
> 
67c90,91
<      sprintf(ftp + strlen(ftp), " -n %s > %s", host, tmpList);
---
>      /* Get Parse Structure */ 
>      Parse = ReadGopherFtp(host); 
68a93,94
>      sprintf(ftp + strlen(ftp), " -n %s > %s", host, tmpList);
>      
73c99,100
< 	
---
>      
>      
81,94c108,139
< 	  printf("At this point host: %s   thing: %s\n", host, thing);
< 
<      if (termCh == '/') {	/* We have a directory */
< 	  gettingFile = 0;
< 	  if (strlen(thing) > 0) 
< 	       FailErr(fprintf(fd, "cd \"%s\"\n", thing));
< 	  FailErr(fprintf(fd, "ls -F\n"));
<      } 
<      else {	                /* We have a file */
< 	  gettingFile = 1;
< 	  if (gettingBinary = IsBinaryType(thing))  
< 	       FailErr(fprintf(fd, "binary\n"));
< 	  FailErr(fprintf(fd, "get \"%s\" \"%s\"\n", thing, tmpData));
<      }
---
> 	  printf("At this point host: %s   thing: %s list_cmd: %s\n", host, thing,Parse->dir_cmd);
>      if ( ! Icky_VMS )
>        {
> 	 if (termCh == '/') {	/* We have a directory */
> 	   gettingFile = 0;
> 	   if (strlen(thing) > 0) 
> 	     FailErr(fprintf(fd, "cd \"%s\"\n", thing));
> 	   FailErr(fprintf(fd, "%s \n",Parse->dir_cmd));
> 	 } 
> 	 else {	                /* We have a file */
> 	   gettingFile = 1;
> 	   if (gettingBinary = IsBinaryType(thing))  
> 	     FailErr(fprintf(fd, "binary\n"));
> 	   FailErr(fprintf(fd, "get \"%s\" \"%s\"\n", thing, tmpData));
> 	 }
>        }
>      else
>        {
> 	 if (termCh == ']') {	/* We have a directory */
> 	   gettingFile = 0;
> 	   if (strlen(thing) > 0) 
> 	     FailErr(fprintf(fd, "cd \"%s\"\n", thing));
> 	   FailErr(fprintf(fd, "%s \n",Parse->dir_cmd));
> 	 } 
> 	 else {	                /* We have a file */
> 	   gettingFile = 1;
> 	   if (gettingBinary = IsBinaryType(thing))  
> 	     FailErr(fprintf(fd, "binary\n"));
> 	   FailErr(fprintf(fd, "get \"%s\" \"%s\"\n", thing, tmpData));
> 	 }
> 	 
>        }
97c142,143
< }
---
> 
>    }
104c150
<      char  buf[BUFSIZ], theName[SLEN];
---
>      char  buf[BUFSIZ], theName[SLEN],theThing[SLEN];
114c160
< 	       fd = uopen(tmpData, "r");
---
> 	       fd = uopen(tmpData, O_RDONLY);
140c186
< 	       GopherType(buf, theName);
---
> 	       GopherType(buf, theName,theThing);
142c188,190
< 		       host, thing, buf, Zehostname, GopherPort);
---
> 		       host, theThing, buf, Zehostname, GopherPort);
> 	       if (DEBUG)
> 		 printf("%s",outputline); 
226,227c274,275
< GopherType(buf, theName)
<   char *buf, *theName;
---
> GopherType(buf, theName,theThing)
>   char *buf, *theName,*theThing;
234,235c282,286
<      static char *ext9[] = {".tar", ".Z"};
<      int extType, i, last;
---
>      static char *ext9[] = {".tar", ".Z" };
>      static char *vms_dir = ".dir" ; 
> 
>      char *semi,*cp,*cp_temp,*cp_temp2;
>      int extType, i, last,field;
236a288,290
> 
>      if ( DEBUG && Icky_VMS )
>        printf("buff:%s:theName:%s:",buf,theName); 
240,244c294,354
<      if (buf[last] == '/') {
< 	  tmpName[last] = '\0';
< 	  sprintf(theName, "%d%s", GDIR, tmpName);
< 	  return;
<      }
---
> 
>      strcpy(theThing,thing); 
> 
>      /* Find the pesky Field */ 
>      field = Parse->field; 
>      /* Sure wish NeXT had strtok ! */ 
>      cp = tmpName; 
>      /* Note this will break horribly if there is a space in the filename */ 
>      /* Get to first non-space char */ 
>      while ( isspace(*cp)) cp++ ; 
>      i = 1 ; 
>     
>      while ( i < field && cp != '\0' )
>        {  
> 	 while ( !isspace(*cp)) cp++ ; 
> 	 i++ ; 
> 	 while ( isspace(*cp)) cp++ ; 
>        }
>      if ( cp == '\0' ) /* field value was wrong or not found use tmpName */ 
>        cp = tmpName ; 
>       
>      cp_temp = index(cp,' ');
>      if ( cp_temp != 0 ) 
>        *cp_temp = '\0'; /* sneaky string splitting */ 
>      /* cp now points to the correct field , put that in buf and tmpName */ 
>      strcpy(buf,cp); 
>      strcpy(tmpName,cp); 
>      cp_temp = &cp[strlen(cp)-strlen(Parse->dir_string)]; 
> 
>      if (DEBUG )
>        {
> 	 printf("Dir str :%s: Compare :%s: \n",cp_temp,Parse->dir_string); 
> 
>        }
> 
>      if ( strncmp(cp_temp,Parse->dir_string,strlen(Parse->dir_string)) == 0) 
>        { /* It's a dir ! */ 
> 	 if (DEBUG )
> 	   printf("%s is a dir\n",cp); 
> 	 *cp_temp = '\0' ; /* more sneaky string splitting */ 
> 	 if ( Icky_VMS ) 
> 	   {
> 	     /* Need to add to theThing in a correct way and Blast buf */
> 	     *buf = '\0'; 
> 	     cp_temp2 = index(theThing,']');
> 	     *cp_temp2 = '.' ;
> 	     cp_temp2++;
> 	    
> 	     
> 	     strcpy(cp_temp2,cp);
> 	     strcat(cp_temp2,"]"); 
> 	   }
> 	     
> 	 if (DEBUG )
> 	       { printf("The thing is %s \n The Name is %s\n",theThing,theName); }
> 	
> 	 sprintf(theName, "%d%s", GDIR, cp);
> 	 return;
> 	   
>        }
>     
306a417,493
> /*
>   Booker C. Bense : 9/1/92
>   routines for parsing ftp gopher files 
> 
>  */ 
> 
> /*
>   ReadGopherFtp reads the gopherftp file and returns
>   either the default structure or the listed one. 
>   As a cheesy first pass we will follow the Security Access File method
>   although this should really be done once and kept in memory. 
> */ 
> 
> parse_ftp *ReadGopherFtp(char *hostname)
> {
>   char inputline[MAXLINE];
>   char *cp,*cp_temp; 
>   int KeepGoing = TRUE;
>     
>   fseek(FTPFileHandle,0,0); /* back to the beginning */ 
>  
>    while (fgets(inputline, MAXLINE, FTPFileHandle) != NULL && KeepGoing) 
>      {
>        ZapCRLF(inputline);
>        cp = inputline;
>        
>        if ( *cp != '#' ) /* not a comment line */ 
> 	 {
> 	   if ( strncasecmp(cp,hostname,strlen(hostname)) == 0 ) 
> 	     /* if host matches parse */ 
> 	     {
> 	       strcpy(Current.host,hostname);
> 	       cp = index(inputline,DELIM);
> 	       if ( cp != 0 )
> 		 { cp++ ; 
> 		   cp_temp = index(cp,DELIM); 
> 		   if ( cp_temp != 0 ) 
> 		     {
> 		       *cp_temp = '\0'; /* sneaky string chopping */
> 		       strcpy(Current.dir_cmd,cp);
> 		       cp_temp++; 
> 		       cp = cp_temp; 
> 		       cp_temp = index(cp,DELIM); 
> 		       if ( cp_temp != 0 )
> 			 {
> 			   *cp_temp = '\0' ; /* sneaky string chopping */
> 			   Current.field = atoi(cp);
> 			   cp_temp++; 
> 			   cp = cp_temp; 
> 			   cp_temp = index(cp,DELIM); 
> 			   if (cp_temp != 0) 
> 			     {
> 			       *cp_temp = '\0' ; /* sneaky string chopping */
> 			       strcpy(Current.dir_string,cp); 
> 			       if (DEBUG )
> 				 {
> 				   printf("Host %s \n",Current.host);
> 				   printf("Dir cmd %s \n",Current.dir_cmd);
> 				   printf("Field %d \n",Current.field);
> 				   printf("Dir_string %s \n",Current.dir_string);
> 
> 
> 				 }
> 			       return( &Current ); /* Current is filled GO FOR IT! */ 
> 			     }
> 			 }
> 		     }
> 		 }
> 	       KeepGoing = FALSE; /* line did not parse correctly so we're giving up */ 
> 	     }
> 	 }
>      }
>   /* hostname wasn't in the file so we're punting and using the default */ 
>   strcpy(Default.host,hostname); 
>   return ( &Default ); 
> 
> }
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/globals.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 globals.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:03 $
<  * $Source: /Booker/CVSRoot/gopher/gopherd/globals.c,v $
---
>  * $Author: lindner $
>  * $Revision: 1.1 $
>  * $Date: 91/03/25 21:11:53 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/globals.c,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/globals.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 globals.h
28a29,32
> /* Cheesy ftp file mods */
> EXTERN char      FTPParseFile[256];
> EXTERN FILE      *FTPFileHandle INIT(NULL);
> 
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/gopherd.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 gopherd.c
121a122
> 
122a124,130
> 
>      /* Local SDSC mod */ 
>      /* Want to log and use default access files */ 
>      strcpy(LOGFile, LOG_FILE);
>      strcpy(SecurityFile, ACCESS_FILE);
>      strcpy(FTPParseFile, GOPHERFTP_FILE ); 
> 
138c146
<      while ((c = getopt(argc, argv, "mCDIcL:l:s:u:U:")) != -1)
---
>      while ((c = getopt(argc, argv, "mCDIcL:l:s:u:U:f:")) != -1)
190a199,208
> 	  
> 	     case 'f': /** gopherftp file name **/
> 	       if (*optarg == '/')
> 		    strcpy(FTPParseFile, optarg);
> 	       else {
> 		    getwd(FTPParseFile);
> 		    strcat(FTPParseFile, "/");
> 		    strcat(FTPParseFile, optarg);
> 	       }
> 	       break;
224c242
< 	  fprintf(stderr, "Usage: %s [-CDIc] [-u userid] [-U uid] [-s securityfile] [-l logfile] <datadirectory> <port>\n", argv[0]);
---
> 	  fprintf(stderr, "Usage: %s [-CDIc] [-u userid] [-U uid] [-s securityfile] [-l logfile] [-f gopherftpfile ] <datadirectory> <port>\n", argv[0]);
231a250
> 	  fprintf(stderr, "   -f  specifies the name of a gopherftp file\n");
276a296,298
>      if (*FTPParseFile != '\0' && !RunFromInetd)
> 	  printf("Using GopherFtp file %s\n", FTPParseFile);
> 
311a334,343
>      /* Same deal for ftp Parse file */ 
>      if (*FTPParseFile != '\0') {
> 	  FTPFileHandle = ufopen(FTPParseFile, "r");
> 
> 	  if (FTPFileHandle == NULL) {
> 	       printf("Can't open the FTPParse file: %s\n", FTPParseFile);
> 	       exit(-1);
> 	  }
> 	  fclose(FTPFileHandle);
>      }
350a383,393
>      }
>      
>      /* Same deal for ftp Parse file */ 
>      if (*FTPParseFile != '\0') {
> 	  FTPFileHandle = ufopen(FTPParseFile, "r");
> 
> 	  if (FTPFileHandle == NULL) {
> 	       printf("Can't open the FTPParse file: %s\n", FTPParseFile);
> 	       exit(-1);
> 	  }
> 	 
===================================================================
RCS file: /Booker/CVSRoot/gopher/gopherd/wais.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 wais.h
10,13c10
<  * $Log: wais.h,v $
<  * Revision 1.1.1.1  1992/09/03  02:54:05  booker
<  * UMN 1.2 distribution
<  *
---
>  * $Log:	wais.h,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/conf.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 conf.h
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/conf.h,v $
---
>  * $Author: lindner $
>  * $Revision: 1.1 $
>  * $Date: 91/03/25 21:11:32 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/conf.h,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/daemon.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 daemon.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/daemon.c,v $
---
>  * $Author: lindner $
>  * $Revision: 1.1 $
>  * $Date: 91/03/25 21:10:22 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/daemon.c,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/globals.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 globals.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/globals.c,v $
---
>  * $Author: lindner $
>  * $Revision: 1.1 $
>  * $Date: 91/03/25 21:11:53 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/globals.c,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/globals.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 globals.h
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/globals.h,v $
---
>  * $Author: lindner $
>  * $Revision: 1.1 $
>  * $Date: 91/03/25 21:12:22 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/globals.h,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/gopherd.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 gopherd.h
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/gopherd.h,v $
---
>  * $Author: $
>  * $Revision: $
>  * $Date: $
>  * $Source: $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/radiod.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 radiod.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:22 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/radiod.c,v $
---
>  * $Author: lindner $
>  * $Revision: 1.4 $
>  * $Date: 91/04/10 12:37:09 $
>  * $Source: /export/mermaid/staff/lindner/gopherd/RCS/gopherd.c,v $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/Radio/radiod/util.c,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 util.c
3,6c3,6
<  * $Author: booker $
<  * $Revision: 1.1.1.1 $
<  * $Date: 1992/09/03 02:54:21 $
<  * $Source: /Booker/CVSRoot/gopher/misc/Radio/radiod/util.c,v $
---
>  * $Author: $
>  * $Revision: $
>  * $Date: $
>  * $Source: $
===================================================================
RCS file: /Booker/CVSRoot/gopher/misc/gopher2ftp/gopherftp,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 gopherftp
6,9c6
< # $Log: gopherftp,v $
< # Revision 1.1.1.1  1992/09/03  02:54:16  booker
< # UMN 1.2 distribution
< #
---
> # $Log: g2ftpd,v $