From: Eric Lease Morgan <eric_morgn@ncsu.edu>
Subject: Re: mailing info to gopher/wais servers - how ??
X-Xxmessage-Id: <A754A7A2EC01184D@ericmorgan.lib.ncsu.edu>
X-Xxdate: Wed, 16 Dec 92 09:30:10 GMT
Organization: NCSU Libraries
X-Useragent: Nuntius v1.1.1d12
Date: Wed, 16 Dec 1992 14:23:42 GMT

In article <1992Dec15.223305.21532@cc.uow.edu.au> Kirk,
kirk@cc.uow.edu.au writes:
>	I am interested to find out if anyone has setup programs or 
>procedures that enable them to email files to gopher or wais servers and
have 
>the information automatically placed into the relevant area/index. A
while 
>back someone posted an article about how to do something like this which
I 
>think also included converting Microsoft Word documents to plain text.

Below is an essay and two scripts I wrote to begin addressing this
problem. The essay also breifly outlines my plans for the future.
They are here to share.

Eric Lease Morgan
NCSU Libraries
(919) 515-6182

=====================

A Memo From Mr. Serials.


A few weeks ago I inquired to the folks of PACS-L about methods for
systematically collecting ejournals. I got a few responses. To all of you
who responded, "Thank you." Only a couple of people said they were
collecting ejournals. Unfortunately, no one had the sort of systematic
method I was looking for.

Consequently, I hired a consultant to investigate the matter. His name is
Mr. Serials. Below is a memo I got from Mr. Serials outlining his
progress using the LISTSERV, FTP, Gopher, Perl, and WAIS technologies. I
am forwarding it on to you for your information.

Eric Lease Morgan

============================

  To: Eric Lease Morgan
  From: Mr. Serials
  Date: December 8, 1992
  Re: Ejournal collection

I am happy to report I have made significant progress towards discovering
methods for collecting ejournals. Although the project is still under
development, I believe you can look forward to having a system in place
before the end of Spring. This memo outlines what I have done so far.

As I see it, I have five collection management issues to address:

  1) selection,
  2) storage,
  3) organization, 
  4) access, and
  5) acquisitions.


SELECTION

My first task is to select and subscribe to the ejournals. I took it upon
myself to collect only a small segment of the population, specifically
the titles dealing with library and information science. These titles
include:

  * ALAWON,
  * ALTCS Network News,
  * Citations for Serial Literature,
  * Current Cites,
  * IR List Digest,
  * Newsletter for Serials Pricing Issues,
  * Public Access Computer Systems News, and
  * Public Access Computer Systems Review.

This part of the process was easy. I just used Charles Bailey's list of
ejournals as a guide and followed the instructions listed there. (There
are a few other titles listed in Bailey's guide, but I haven't gotten
around to them yet.) 


FTP FOR STORAGE AND RUDIMENTARY ACCESS

I decided to solve my storage problems using anonymous FTP. This way
anybody can get the articles if they have access to the Internet. FTP
also provides a mechanism for archiving the materials. It is important to
note I created a single directory for each title. Within that directory
are all the issues and articles from that title, each with a unique but
identifying name. For example, every file from Citations for Serial
Literature (CSL) is designated to begin with "csl". Since CSL is issued
in volume and number, each designation is followed with something like
"-v1n22". Therefore a complete designation is "csl-v1n22". Issues from
Public Access Computer Systems News looks like "pacsn-v1n03". IR List
Digest, which comes out sequentially, is saved as "irld-001". Take note
of the leading zeros in the preceding examples. They make it possible to
accurately sort the files by volume and number, and consequently date.
Also, keep in mind, the serial codes I have created are arbitrary. As
long as I am  consistent and there is a method to my madness, then it
doesn't matter what I call them.


GOPHER FOR ENHANCED ACCESS AND ORGANIZATION

FTP is strong on storage, but weak on access. This is where a Gopher
comes in. The Gopher protocol excels at maintaining pointers to Internet
resources. Since anonymous FTP sites are Internet resources, and since I
have a Gopher server in place, I created Gopher pointers (called link
files) to my ejournals. This solved a number of problems. First, the link
files provide the capability to list files in a more readable fashion.
Second, the user no longer has to use FTP commands to access the
collection. Below is a fictitious, but correctly formatted link file:

  Name=ALAWON v1n35 (May 5, 1992)
  Type=0
  Port=70
  Path=ftp:dewey.lib.ncsu.edu@pub/stacks/alawon/alawon-v1n35
  Host=dewey.lib.ncsu.edu

When the patron uses the Gopher they see "ALAWON v1n35 (May 5, 1992)." By
selecting the item the Gopher transparently opens up an anonymous FTP
connection to dewey.lib.ncsu.edu, changes to the /pub/stacks/alawon
directory, retrieves the file "alawon-v1n35" as a text (ASCII) file,
closes the connection, and displays the file. From there the patron can
save the file on their computer.

Furthermore, the Gopher protocol allows me to organize my ejournals into
volumes or years with an abbreviated link file called a "cap" file. Thus
I can organize my ejournals and eliminate very long menu listings.


WAIS FOR KEYWORD ACCESS

I am now using FTP primarily for storage and rudimentary access. With
Gopher I ehnanced access and organized the information. Unfortunately FTP
and Gopher are only useful if the patron knows exactly which issue or
article they want. WAIS now enters the picture allowing the patron to
search the collection via keywords.

Since I own all the articles associated with any particular ejournal, and
since I have a WAIS server in place, then it is a simple matter of
creating a WAIS index for each of my titles and putting these indexes in
the appropriate directory of the Gopher server. An unexpected benefit
became of this. It will be described later.


AUTOMATED ACQUISITIONS WITH PERL

Finally, after all this planning, I had a place to put my ejournals once
I collected them. I began by manually retrieving the files with the
LISTSERV "get" command. Once retrieved, I manually edited the files,
saved the edited versions in their appropriate FTP directory, wrote
Gopher link files, and updated the WAIS indexes.

Needless to say, this got old quickly! From the comp.infosystems.gopher
USENET newsgroup I had heard a lot about a programing language called
"perl". I retrieve the perl archive and wrote my very first perl script.
This script, which I call "makelink" automates much of the process listed
above. It:

  1) reads a file (email message) given from the command line,
  2) removes the header,
  3) displays the first 19 line of the file,
  4) queries the user for a Gopher name for the file
     (ex: ALAWON v1n12 (May 13, 1992))
  5) queries the user for a FTP name for the file
     (ex: alawon-v1n12)
  6) saves the edited file in the appropriate FTP directory, and
  7) creates a link file and saves it the appropriate directory.

Now, all I have to do is wait for mail to arrive, save the message as a
file, and use makelink to make it available to the world.


SUMMARY AND FUTURE CONSIDERATIONS

Looking back, the most labor intensive aspect of this ejournal collection
process has been collecting all the back issues. Now that I have them,
this problem is behind me.

It was tempting to begin the acquisitions process without a structure for
storage and access, but now that the structure is in place things are
going smoothly.

The acquisitions process can be further automated. There is a program
called "procmail". Procmail "listens" to your email and processes it when
it arrives. I plan to use procmail to save incoming files from my
subscriptions into a directory called "process-us". I will then rewrite
makelink to run once a day. It will read all the files in the process-us
directory, examine the contents of the files, and automatically create
its own link and FTP files. This in turn will decrease the probability of
data-entry error and increase my lunch hour. The success of this
automated acquisitions process will be based on the consistency of the
ejournals. If they are created with a basic form, and they consistently
use that form, then it will a simple matter of extracting the identifying
information from them.

You may ask, "How do people get access to these ejournals if the don't
have a computer or Internet access?" The answer is the OPAC. It is
probable that the opening screen of our new OPAC will contain a list of
databases. One will be the card catalog. Another will the the CWIS.
Another choice may be the Gopher. Patrons will be able to access the
ejournal collection through terminals throughout the library.

"What about bibliographic control?" Well, I haven't addressed that
problem. I suppose it would be possible to catalog the ejournals and
create a note saying that the title lives on the Gopher. A better
solution would be if the OPAC could include Gopher client capabilities,
and thus the OPAC would retrieve a selected file. Hmmm...

Previously I mentioned a side benefit of WAIS indexing. Here it is. Since
I have a (almost) complete set of the library and information science
titles I can (and have) created an electronic version of _Library
Literature_. I call it Electronic Library Literature Index (ELLI). With
ELLI I can search the entire body of electronic library literature and
retrieve articles from a number of titles. For example, I can search for
"NREN" can come up with articles from Public Access Computer Systems
News, ALAWON, and IR List Digest. There is no reason why this idea can't
be used for other ejournals and subjects.

This concludes my memo. All of the ejournals and their indexes can be
found in the Stacks of the Library Without Walls at the Gopher of the
NCSU Libraries. ELLI is located in the Library and Information Science
Study Carrell. Point your Gopher client to dewey.lib.ncsu.edu on port 70.

If you have any questions, then please do not hesitate to ask. Anybody
interested in obtaining the scripts I wrote are encouraged to send my
boss a note.

Mr. Serials

============================

Eric Lease Morgan is a Systems Librarian at the North Carolina State
University Libraries. "My professional goal is to discover new ways to
use computers to improve library service." Eric can be reached at (919)
515-6182 or eric_morgan@ncsu.edu.

Appendix To A Memo From Mr. Serials


First, here is the shell scipt I use to reindex the ejournals.

#
# this file creates a wais index for the specified command
# line input
#
# usage: index <leaf name>
#
# first, IR List Digest
if "$1" == "irld" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# ALCTS Network News
if "$1" == "ann" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# next, Public Access Computer Systems Review
if "$1" == "pacsr" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# Citation for Serial Literature 
if "$1" == "csl" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# the Newsletter for Serials Pricing Issues
if "$1" == "prices" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# Publics Access Computer Systems News   
if "$1" == "pacsn" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# how about ALAWON?
if "$1" == "alawon" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
# and now Current Cites
if "$1" == "currentc" then
        set theIndex=/usr/users/temp/gopher/data/.wais/$1
        set theDirectory=/usr/users/temp/ftp/pub/stacks/$1
        waisindex -d $theIndex -r -t text -export $theDirectory
        exit
endif
#
#
#
# index all the titles in the their own indexes
if "$1" == "all" then
#
# pacs news
        set theIndex=/usr/users/temp/gopher/data/.wais/pacsn
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsn
        waisindex -d $theIndex -r -t text -export $theDirectory
# pacs review
        set theIndex=/usr/users/temp/gopher/data/.wais/pacsr
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsr
        waisindex -d $theIndex -r -t text -export $theDirectory
# alawon
        set theIndex=/usr/users/temp/gopher/data/.wais/alawon
        set theDirectory=/usr/users/temp/ftp/pub/stacks/alawon
        waisindex -d $theIndex -r -t text -export $theDirectory
# current cites
        set theIndex=/usr/users/temp/gopher/data/.wais/currentc
        set theDirectory=/usr/users/temp/ftp/pub/stacks/currentc
        waisindex -d $theIndex -r -t text -export $theDirectory
# newsletter on serials pricing issues
        set theIndex=/usr/users/temp/gopher/data/.wais/prices
        set theDirectory=/usr/users/temp/ftp/pub/stacks/prices
        waisindex -d $theIndex -r -t text -export $theDirectory
# irl digest
        set theIndex=/usr/users/temp/gopher/data/.wais/irld
        set theDirectory=/usr/users/temp/ftp/pub/stacks/irld
        waisindex -d $theIndex -r -t text -export $theDirectory
# aclts netork news 
        set theIndex=/usr/users/temp/gopher/data/.wais/ann
        set theDirectory=/usr/users/temp/ftp/pub/stacks/ann
        waisindex -d $theIndex -r -t text -export $theDirectory
# citations of serial literature
        set theIndex=/usr/users/temp/gopher/data/.wais/csl
        set theDirectory=/usr/users/temp/ftp/pub/stacks/csl
        waisindex -d $theIndex -r -t text -export $theDirectory
# 
# ell
        set theIndex=/usr/users/temp/gopher/data/.wais/ell
        set theDirectory=/usr/users/temp/ftp/pub/stacks/alawon
        waisindex -d $theIndex -r -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/currentc
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/irld
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsr
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/prices
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsn
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/csl
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/ann
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        exit
endif
#
#
# now the moment you've all been waiting for... create
# the "Electronic Library Literature Index"
if "$1" == "ell" then
        set theIndex=/usr/users/temp/gopher/data/.wais/ell
        set theDirectory=/usr/users/temp/ftp/pub/stacks/alawon
        waisindex -d $theIndex -r -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/currentc
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/irld
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsr
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/prices
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/csl
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/pacsn
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        set theDirectory=/usr/users/temp/ftp/pub/stacks/ann
        waisindex -d $theIndex -r -a -t text -export $theDirectory
        exit
endif
#
#
# echo the input error
echo "Usage: index <nspi | irld | pacsr | alawon | currentc | ell>"


Second, here is the perl script I use (makelink) to do acquisitions.

#!/usr/bin/perl
#
# makelink, a perl script to edit a saved mail
# message, move the edited message to an FTP
# directory, and create a link file for the message.
# 
# copyright 1992 by Eric Lease Morgan
#
#
#
# make sure the command line arguments where given
if ($#ARGV < 1) {
        print STDERR "Usage: makelink <serial code> <filename>\n";
        exit(1);
        }
#
#
# check for a valid serial code
$basicGopherPath = "/usr/users/common/gopher/data/library/stacks/";
if ($ARGV[0] eq "csl") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/csl/";
        $gopherPath = $basicGopherPath . "csl/";
        }
elsif ($ARGV[0] eq "pacsr") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/pacsr/";
        $gopherPath = $basicGopherPath . "pacsr/";
        }
elsif ($ARGV[0] eq "prices") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/prices/";
        $gopherPath = $basicGopherPath . "prices/";
        $FTPExample = "nspi-ns001";
        $gopherExample = "NSPI ns 001 (August 5, 1991)";
        }
elsif ($ARGV[0] eq "pacsn") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/pacsn/";
        $gopherPath = $basicGopherPath . "pacsn/v3/";
        $FTPExample = "pacsn-v1n02";
        $gopherExample = "PACSN v1n2";
        }
elsif ($ARGV[0] eq "alawon") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/alawon/";
        $gopherPath = $basicGopherPath . "alawon/v1/";
        $FTPExample = "alawon-v1n02";
        $gopherExample = "ALAWON v1n15 (May 15, 1992)";
        }
elsif ($ARGV[0] eq "irld") {
        $FTPPath = "/usr/users/common/ftp/pub/stacks/irld/";
        $gopherPath = $basicGopherPath . "irld/";
        }
else {
        die STDERR "Error: $ARGV[0] is unknown.\n";
        exit(1);
        }
# open the file
open (THEFILE, $ARGV[1]) || die "Error: Can't open the file.\n";
#
#
# read the file
@theText = <THEFILE>;
close (THEFILE);
#
#
# find the begining of the file without the header 
$theFirstLine = -1;
do {
        $theFirstLine++;
} until $theText[$theFirstLine] eq "\n";
$theFirstLine++;
#
#
# display the first 19 lines of the file
print "\n\n";
for ($i = $theFirstLine; $i < ($theFirstLine + 19); $i++)
        {print $theText[$i];}
print "\n\n";
#
#
# ask the user for the "human name" of the file
print "What will be the Gopher title of this file?\n";
print "example: $gopherExample  ";
$gopherName = <STDIN>;
#
#
# ask the user for the "FTP name" of the file
print "What will be the FTP name of the file?\n";
print "example: $FTPExample  ";
$FTPName = <STDIN>;
#
#
# create the  full path name of the link file
$linkFile = $gopherPath . "." . $FTPName;
chop $linkFile;
#
#
# create the full name of the FTP file
$FTPPath = $FTPPath . $FTPName;
#
#
# save the FTP file
open (FTPFILE, ">$FTPPath") || die "Can't open FTP file\n";
print FTPFILE "\n";
print FTPFILE "Title: $gopherName";
print FTPFILE "File: $FTPName\n";
$i = -1;
do {
        $i++;
        print FTPFILE "$theText[$theFirstLine + $i]";
} until $theText[$theFirstLine + $i] eq '';
close (FTPFILE);
#
#
# write the link file
open (LINKFILE, ">$linkFile") || die "Can't open link file\n";
print LINKFILE "Name=$gopherName";
print LINKFILE "Type=0\n";
print LINKFILE "Port=70\n";
print LINKFILE "Path=ftp:dewey.lib.ncsu.edu@pub/stacks/$ARGV[0]/$FTPName";
print LINKFILE "Host=dewey.lib.ncsu.edu";
close (LINKFILE);
Newsgroups: comp.infosystems.gopher