1.  Introduction
``Harvest'' is an integrated set of tools to gather,
extract, organize, search, cache, and replicate relevant
information across the Internet.  With modest effort users
can tailor ``Harvest'' to digest information in many
different formats, and offer custom search services on the
Internet.  ``Harvest'' consists of five subsystems.
For the immediate CHISSA needs, only two of the ``Harvest''
subsystems need be used.  These subsystems are described in
the following sub-sections.
1.1  ``Gather''
The ``Gather'' subsystem collects indexing information (such
as keywords, author names, and titles) from the resources
available at Provider sites (such as FTP and HTTP servers).
1.2  ``Broker''
The ``Broker'' subsystem retrieves the information of the
``Gather'' subsystem, suppresses duplicate information,
indexes the collected data, and provides a WWW query
interface to it.
1.3  Structured Query
The ``Harvest'' Broker handles many types of queries.  The
simplest query is a single keyword, such as: _l_i_g_h_t_b_u_l_b
``Harvest'' supports many different index/search engines,
with varying capabilities.  At present, the most powerful
(and commonly used) search engine is _G_l_i_m_p_s_e i.  which
supports:
  o+ case-insensitive and case-sensitive queries;
  o+ the ability to match parts of words, whole words, or
    multiple word phrases (like "resource discovery");
  o+ Boolean (AND/OR) combinations of keywords;
                           Page 1
                                           September 7, 1995
  o+ approximate matches (e.g., allowing spelling errors);
  o+ structured queries (which allow you to constrain matches
    to certain fields);
  o+ the ability to show matched lines or entire matching
    records (e.g., for citations);
  o+ the ability to specify limits on the number of matches
    returned; and
  o+ a limited form of regular expressions (e.g., allowing
    "wild card" expressions that match all words ending in a
    particular suffix).
You use the same syntax regardless of what index/search
engine is running in a particular Broker, but that not all
engines support all of the above features.
The different options - case-sensitivity, approximate
matching, the ability to show matched lines versus entire
matching records, and the ability to specify match count
limits - can all be specified with buttons and menus in the
Broker query forms.
A structured query has the form:           tag-name : value
where ``tag-name'' specifies the context for interpreting
the ``value''.  Each Broker can support different ``tag-
name'' attributes, depending on the data it holds.  The
common valid ``tag-name''s are:
Abstract
Author
Description
File-Size
Full-Text
Gatherer-Host
Gatherer-Name
Gatherer-Version
Keywords
Last-Modification-Time
                           Page 2
                                           September 7, 1995
MD5
Refresh-Rate
Time-to-Live
Title
Type
URL-References
Update-Time
and ``value'' specifies the search value of the ``tag-name''
attribute.  If you click on a Content Summary, you will see
what attributes are available for a particular Broker.
Keyword searches and structured queries can be combined
using Boolean operators (AND and OR) to form complex
queries.  Lacking parentheses, logical operation precedence
is based left to right.  For multiple word phrases or
regular expressions, you need to enclose the string in
double quotes, e.g.,
          "internet resource discovery"
or
          "discov.*"
1.3.1  Examples
Simple keyword search query:           Arizona</pre>
This query will return all objects in the Broker containing
the word _A_r_i_z_o_n_a.
Boolean query:           Arizona AND desert
This query will return all objects in the Broker that
contain both words anywhere in the object in any order.
Phrase query:           "Arizona desert"
This query will return all objects in the Broker that
contain _A_r_i_z_o_n_a _d_e_s_e_r_t as a phrase.  Notice that you need to
put double quotes around the phrase.
Boolean queries with phrases:           "Arizona desert" AND
windsurfing
Simple Structured query:           Title : windsurfing</pre>
This query will return all objects in the Broker where the
_T_i_t_l_e attribute contains the value _w_i_n_d_s_u_r_f_i_n_g.
                           Page 3
                                           September 7, 1995
Complex query:           "Arizona desert" AND (Title :
windsurfing)
This query will return all objects in the Broker that
contain the phrase _A_r_i_z_o_n_a _d_e_s_e_r_t and where the _T_i_t_l_e
attribute of the same object contains the value _w_i_n_d_s_u_r_f_i_n_g.
1.3.2  Query options selected by menus or buttons
These checkboxes allow some control of the query
specification.
Case insensitive By selecting this checkbox the query will
 become case insensitive (lower case and upper case letters
 differ).  Otherwise, the query will be case senstive.  The
 default is case insensitive.
Keywords match on word boundaries By selecting this
 checkbox, keywords will match on word boundaries.
 Otherwise, a keyword will match part of a word (or phrase).
 For example, _n_e_t_w_o_r_k will matching _n_e_t_w_o_r_k_i_n_g, _s_e_n_s_i_t_i_v_e
 will match _i_n_s_e_n_s_i_t_i_v_e, and _A_r_i_z_o_n_a _d_e_s_e_r_t will match
 _A_r_i_z_o_n_a _d_e_s_e_r_t_n_e_s_s.  The default is to match keywords on
 word boundaries.
Number of errors allowed Glimpse allows the search to
 contain a number of errors.  An error is either a deletion,
 insertion, or substitution of a single character.  The Best
 Match option will find the match(es) with the least number
 of errors.  The default is 0 (zero) errors.
9NOTE: The previous three options do not apply to attribute
names.  Attribute names are always case insensitive and
allow no errors.
9
1.3.3  Result set presentation
These checkboxes allow some control of presentation of the
query return.
Display matched lines (from content summaries) By selecting
 this checkbox, the result set presentation will contain the
 lines of the Content Summary that matched the query.
 Otherwise, the matched lines will not be displayed.  The
 default is to display the matched lines.
Display object descriptions (if available) Some objects have
 short, one-line descriptions associated with them.  By
 selecting this checkbox, the descriptions will be
 presented.  Otherwise, the object descriptions will not be
                           Page 4
                                           September 7, 1995
 displayed.  The default is to display object descriptions.
Verbose display This checkbox allows you to set whether
 results are displayed listing the filename, host, path, and
 Content Summary each on separate lines, or just with two
 lines listing the filename (without a label) and the
 Content Summary (with a label).  The default is verbose.
1.3.4  Regular Expressions
Some types of regular expressions are supported by Glimpse.
A regular expression search can be much slower that other
searches.  The following is a partial list of possible
patterns.  (For more details see the URL
"http://glimpse.cs.arizona.edu:1994/glimpse.html" Glimpse
manual pages.)
^joe will match _j_o_e at the beginning of a line.
joe$ will match _j_o_e at the end of a line.
[a-ho-z] matches any character between a and h or between o
 and z.
. matches any single character except newline.
c* matches zero or more occurrences of the character _c.
.* matches any number of wild cards.
matches the character *  ( escapes any of the above special
 characters).
Regular expressions are currently limited to approximately
30 characters, not including meta characters.  Regular
expressions will generally not cross word boundaries
(because only words are stored in the index).  So, for
example, "lin.*ing" will find "linking" or "flinching," but
not "linear programming."
1.4  On-line Demonstration
The latest version of ``Harvest'' (Version 1.3, September 7,
1995) has been installed on ``video''.  Access to the demo
is obtainable from a WWW Browser via the URL
http://video/Harvest/brokers/CHISSA/summary.html.
                           Page 5
                                           September 7, 1995
1.4.1  Setup for CHISSA Document Search
All the documents available for the CHISSA Document Search
are contained in /export/home/tools/harvest-
1.3/progs/CHISSA/DOCs.  The program ``RunGatherer'' in
directory /export/home/tools/harvest-
1.3/progs/gatherers/CHISSA enumerates and summarizes data
objects obtained from the CHISSA Document Search directory.
This gathered data is represented as a list of attribute-
value pairs using the _S_u_m_m_a_r_y _O_b_j_e_c_t _I_n_t_e_r_c_h_a_n_g_e _F_o_r_m_a_t
(SOIF).
To generate an updated SOIF archive for CHISSA documents:
          cd /export/home/tools/harvest-
1.3/progs/gatherers/CHISSA           ./RunGatherer
To allow WWW Browser's to access the CHISSA SOIF archive, a
server (the ``Harvest'' Broker) has to be running on _v_i_d_e_o.
To initiate the CHISSA Broker:           cd
/export/home/tools/harvest-1.3/progs/brokers/CHISSA
          ./RunBroker           ps -ax | grep broker    #
Ensure "glimpse" server active before continuing
The current URL to search the CHISSA Document Server is
http://video/Harvest/brokers/CHISSA/summary.html.
_1._4._1._1  Updating search data
Modify configuration file _C_H_I_S_S_A._c_f if needed to include new
files.
          /export/home/tools/harvest-
1.3/progs/gatherers/CHISSA
Gather the new data.            cd
/export/home/tools/harvest-1.3/progs/gatherers/CHISSA
          rm -rf data           ./RunGatherer Update the
Broker data to use the update gathered data.            cd
/export/home/tools/harvest-1.3/progs/brokers/CHISSA
          ps -ax | grep broker    # Ensure "glimpse" server
active before continuing           kill pid_broker
          rm -rf objects           ./RunBroker           ps
-ax | grep broker    # Ensure "glimpse" server active before
continuing
9NOTE: If two files contain the same data, ``Broker'' will
only report on one of the files.
9
                           Page 6
                                           September 7, 1995
1.5  Changing Broker Output
File:           /export/home/tools/harvest-1.3/progs/cgi-
bin/lib
is the configuration file for the Broker Query results.
This file appears to be written in HTML for deciphering by
Perl.  If this is true, should be able to change this file
to satisfy specific requests.
                    2.  Accomplishments
2.1  WWW Access to CHISSA Tools
2.1.1  Constraints
Providing access and useability of CHISSA developed tools at
NIST/CSL for all users on the WWWW independent of the
hardware/software used to attain the access was determined
to be too broad a scope.  Also, to provide access by
allowing remote logins was considered dubious because of
security considerations.
This scope was reduced to encompass those capabilities
needed to process the current set of tools that CHISSA has
available for demonstration.  These tools used Xwindows to
provide the user interface.  By requiring the accessor to
support Xterm and requiring the user to configure one's
system to allow the NIST/CSL host running the CHISSA
demonstration tool to establish an Xwindow on one's
workstation, the availability and useability of the CHISSA
tools becomes attainable.
Access to CHISSA tools can be provided via WWW pages at
designated hosts.  Useability of the CHISSA tool is via the
WWW page spawning a process that runs the selected CHISSA
tool.
2.1.2  It Works
Host "video" was used to develop a WWW page and a program to
provide this capability.  Once the tool is selected-User
invokes a WWW Browser to get to CHISSA's home page.-from a
WWW menu page depicting multiple tools, an independent
process is spawned to run the selected tool.-The "httpd"
server generated process that processed the WWW Browser
request exits, after initiating a new process.- The user now
has two active windows on one's monitor.  One window is the
user's local process displaying the WWW Browser output from
the CHISSA home page script and the other window displays
the initial output from the selected CHISSA tool.  Both
                           Page 7
                                           September 7, 1995
windows remain active.  The CHISSA tool window provides the
ability to run the CHISSA tool as though the user was logged
in on the NIST/CSL host.
This capability actually allows a user to open as many
independent CHISSA tool windows as desired.
2.1.3  Code
The actual code used to provide the capability on "video" is
provided below.
        /*    Spawn another process to run CHISSA Tool.
            The current Tool run for this cgi-bin module is "xforecast".
        */
#define    _POSIX_SOURCE
#include <unistd.h>
#include <sys/types.h>
#include <fcntl.h>
#include <string.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
#include <errno.h>
char *conhtml = "Content-type: text/html0;        /* note dual linefeeds */
extern  void est_env(char **);
#define    TRUE    1
extern  char    **environ;
#define    USER_NAME    "cincotta"    /*    This process runs as this user.    */
#define    TESTpath    "/usr/local/bin/xforecast"
#define    REMOTE_HOST    "REMOTE_HOST"
#define    CONSOLE        "console"
    /*    Establish proper environment.    */
/*        Note:  This program requires a number of environmental parameters. */
#define    LD_LIBRARY_PATH    "LD_LIBRARY_PATH=/usr/local/ORBeline/solaris_gcc/lib:/usr/openwin/lib"
#define    OPENWINHOME        "OPENWINHOME=/usr/openwin"
#define    ORBELINE        "ORBELINE=/usr/local/ORBeline/solaris_gcc/adm"
#define    PATH            "PATH=/usr/local/bin:/usr/ucb:/bin:/usr/bin:/usr/ccs/bin:/usr/local/ORBeline/solaris_gcc/bin:/usr/openwin/bin:/usr/openwin/lib:/usr/ccs/bin:/usr/sbin"
                           Page 8
                                           September 7, 1995
main(int argc, char *argv[], char *env[])
{
    pid_t    pid;
    char    *remote_host;
    char    display[100], console[100];
    est_env(env); /* Establish environment based on WWW browser client data. */
    if(*(getenv(CONSOLE)) == NULL) {    /*    Determine console address.    */
        remote_host = getenv(REMOTE_HOST);
        if(remote_host == NULL)
            _exit(1);
    }
    else
        remote_host = getenv(CONSOLE);
    pid = fork();
    if(pid == (pid_t)0) {
        setuid(200);        /*    Run CHISSA tool as a specific user.    */
            /*    Setup needed environment, which will vary for each tool.  */
        sprintf(display, "DISPLAY=%s:0", remote_host);
        putenv(display);
        putenv(LD_LIBRARY_PATH);
        putenv(OPENWINHOME);
        putenv(ORBELINE);
        putenv(PATH);
        fclose(stdout);        /*    Required for parent process to exit.    */
        execle(TESTpath, "path_test", (char *)0, environ);
        _exit(1);    /*    Should never get here.    */
    }
/*        Generate WWW Browser display.    */
    fprintf(stdout, "%s", conhtml);
    fprintf(stdout, "<br>");
    fprintf(stdout,    "<TITLE>CHISSA Tool Executed</TITLE>");
    fprintf(stdout, "<hr>");
    fprintf(stdout,    "<H1>CHISSA Tool -<br></H1>");
    fprintf(stdout,    "<br><br><br>");
    fprintf(stdout,    "<H2>");
    fprintf(stdout,    "<pre>       xforecast  <a href=%s>%s",
                "http://video.ncsl.nist.gov/~cincotta/st_demo.html",
                "<img align=middle src=/ICONs/unravel.gif></a></pre>");
    fprintf(stdout, "<br>");
    fprintf(stdout, "has been spawned as an independent process.<br>");
    fprintf(stdout, "<br>");
    fprintf(stdout, "<br>");
    fprintf(stdout, "User's console address:  %s", remote_host);
    fprintf(stdout, "<br>");
    fprintf(stdout, "<hr>");
                           Page 9
                                           September 7, 1995
    fflush(stdout);
    _exit(0);
}
As you can see the code needed is minimal.  The code can be
viewed as consisting of three sections:
  1.  Establish process environment based on WWW Browser
      input, then spawn a new process.
  2.  New process establishes the needed process environment
      to run the CHISSA tool on the NIST/CSL host and use
      the user's monitor as its Xterminal, then executes the
      selected CHISSA tool.
  3.  Server process establishes a new display for the WWW
      Browser and exits.
9NOTE: To establish the proper display for Xterm the code
will use the address of the WWW Browser, UNLESS the user
provides the address of one's host.  That is, when a user
initiates a WWW Browser from other than one's own
workstation the user MUST input the proper address of one's
workstation.
9
2.1.4  Availability
The code above was ported to "hissa" and a new home page was
written.  The CHISSA tool ``unravel'' will now operate
properly in this environment.
2.2  Search and Retrieval
2.2.1  Tool Search
A minimal report was generated on the availability of tools
available (at no cost), which will provide search and
retrieval of data on the WWW.  The tools described were
gopher, archie, veronica, jughead, WAIS, WAIS-sf, and
Harvest.  Of these, Harvest seems capable of providing for
the current CHISSA needs.
``Harvest'' is described as:
    an integrated set of tools to gather, extract, organize,
    search, cache, and replicate relevant information across
    the Internet.  This tool can be tailored to digest
    information in many different formats and to offer
    custom services on the Internet.
Quickly skimming the ``Harvest'' manual, this looks like the
tool that can handle all the CHISSA documentation search and
                          Page 10
                                           September 7, 1995
retrieval requirements.
Although, at this time, I Tony, don't know enough about
``Harvest'' to provide comments on its usage and actual
capabilities.  I have installed ``Harvest'' on "video" and
plan to try it out.  Jim has installed ``Harvest'' on
``hissa''.
Some specific ``Harvest'' capabilites:
  1.  Selectively choose the type of data to be gathered
      from a database and/or the file types to be used.
  2.  Data can be gathered at specific times of day.
  3.  Supports attribute querying (e.g., Title: UNIX,
      returns all documents with UNIX in title).
  4.  Supports UNIX style  regular expression querying
      (e.g., ^joe, matches ``joe'' at beginning of line;
      joe$, matches ``joe'' at end of line; etc.).
  5.  Custom output of query results.
  6.  Caching.
  7.  Replicatation of Database to other ``Harvest'' hosts.
                        3.  Tomorrow
3.1  Plans
Become familiar/knowledgeable with the
capabilities/limitations of ``Harvest'' and its affect on
the _k_n_o_w_n CHISSA requirements.
3.2  Billing
                    Job Number 872-7278 Billing
          Used      Available Total_Allocated
                    -----     -----------------------------
          Tony      44        46        90
                          Page 11
                                           September 7, 1995
                4.  Support of CHISSA Tools
Assuming the CHISSA Tool facility is a successful venture.
That is, it contains many useful tools from many
contributors.
4.1  Questions
How is this facility setup:
o+ Is there a single WWW site that provides a Web page that
  access all the tools?
o+ Are the tools at one site, at multiple sites, or multiple
  copies at multiple sites?
Is there a single point of contact for all the tools,
multiple contacts, or a single contact per unique tool?
4.2  Comments
To provide maximal expertise for each CHISSA Tool, the point
of contact for each tool must be the tool developer.  This
may not be practical in many cases, but should be the goal
for naming the contact point.  A single contact for ALL the
CHISSA Tools is a poor approach to Tool support.
Since the environment of the tool dictates where the tool
can run, not all tools will run on all systems.  Therefore,
muliple hosts will be needed to run ALL the tools.
To ensure that a CHISSA Tool remains on-line and up-to-date,
the tool developer must be an active participant in
providing the support of the CHISSA Tool.
By having the Tools runnable on different host sites,
distributes the overhead of running the Tools.
                          Page 12