Report of Progress for TREC
                                        II

                                  William Kelleher


                            Systems Environments Corporation


                                  November 1,1993

1.0  Introduction
Systems Environments is a commercial category B partici-
pant in TREC. For the past two years we have been devel-
oping a software system starting on a DOS based PC and
recently moving to Windows NT on the same 486/25 PC.
Much development time has been diverted toward getting
around bugs and trying to fit the system on small (200 Mb
disks and within the memory constraints of the DOCS sys-
tem). At this time the system is still under development.

FORMS is designed and implemented using the object ori-
ented approach. Object oriented methods were chosen to
create a system architecture that can be installed in a vari-
ety of different environments using different architectures.
In its smallest implementation FORMS can function as a
personal information system on a single PC, or it can be
installed as a organization wide system using client server
approach. The basic objects and their relationships are
shown in the following diagram and described in the fol-
lowing sections.

                                   2.1  SGML Documents

2.0   System Architecture

FORMS (Feedback, Object-oriented Retrieval Methods) is
an object oriented, concept based information retrieval
system.

FORMS is concept based because it creates a profile of the
users information need and matches that profile to the doc-
ument profiles. Profile can be stored in a library of con-
cepts which is retained from one use of the system to
another. Relevance feedback is an essential ingredient in
the design of FORMS although so far it has not been used
inTREC.


                                         275

A document arrives and first must be translated into an
SGML (Standard Generalized Markup Language) Docu-
ment.

                                   User Interface

F

,SGML~
%i2-ments


                      /rn'Document
                `-<Index)

,Document~
    Profiles

  Document
  Display
  Process


  Query
  Process


  Query
  Results
~ Process

(



Query
Profile

        Element
  ~     Index


Concept
Library

2.2  Document Profiles

Elements are the things found in text documents that cap-
ture the meaning imbedded in the documents. The basic
element in text is obviously the words. But there are other
elements that can be very helpful. Identifying certain kinds
of proper nouns can be crucial to determining the rele-
vance of text to an information need. Important proper
nouns might be persons, places, and some kinds of things,
e.g. companies.

There are two distinct types of elements used in FORMS,
simple and complex. A simple element consists of a char-
acter string, e.g. a word, and an element type, e.g. a noun
or a noun phrase. Complex elements are various combina-
tions of simple elements. For example a boolean `and' of
several simple elements occuning in the same sentence,
paragraph or document would be a complex element.

In general simple elements are only used to create docu-
ment profiles elements. A document profile element IS-A
simple element that includes the document id and the num-
ber of occurrences of Th. element in the document.

full text of the document is also created so that the user
can retrieve a document for inspection.


3.0   User Interface
The user intefface objects (grey area in the diagram) pro-
vide object to display documents, retrieve documents
using a natural language query as well as stored concepts,
and a window that displays results.


3.1   Document Display
The document display window permits a user to enter a
document ID and the system will simply retrieve the docu-
ment from one of the text files and display it on the screen.
When a document is displayed, the user has the option to
indicate that the document is or is not relevant to the cur-
rent query (or concept). When an indication of relevance is
given the system modifies the concept/query profile and
provides a new rank list of retrieve documents.

                                          3.2  Query Window

2.3  Indexes
Mter the document profiles are created, those elements
that occur in more than one document are stored in the ele-
ment Index. An second index from the document ID to the

The query window actually consists several parts. A natu-
ral language part for entering a query, and a query profile
part that displays the elements of the query/concept along
with various statistics about each element. Most important

                                                               Report of Progress for TREC II


                                            276

                                 Basic AssumpUons

 is the probability that a document is relevant if it contains
 a particular element. Each element is treated as being inde-
 pendent of every other element.


 3.3  Results Window
 The results window contains the list of all documents with a
 probability of relevance to the current query/concept. Documents
 are ranked by probability of relevance. For each document, the
 elements found in the document are listed as well as any previous
 relevance information. The document D selected from the
 results window can be used in the document window to examine
 the document.


 4.0 Basic Assumptions
 The approach being taken by FORMS is based a number
 of critical assumptions.
   Using the axioms of probabilities as a foundation for
   determining the relevance of documents to a query
 * Elements besides words, such as phrases, can be found
   to make a significant contribution to retrieval accuracy
 * A number of different approaches to identif~ing rele-
   vant elements are worth pursuing. These include
   proper noun identification, part of speech tagging,
   noun phrase tagging, and as yet undetermined relations
   that can be extracted from natural language.


 5.0 Progress to Date
 Frankly, there has been little progress to date beyond basic
 system development. In both TREC I and II, we have per-
 formed routing queries with very poor results. One obvi-
 ous reason is that we have had to perform the analysis by
 breaking the texts into small sections and doing even class
 B in sections.


 6.0 FutureWork

 FORMS is designed to provide information on the effec-
 tiveness of different approaches.

 * To examine different approaches to incorporating rele-
   vance feedback into evaluating relevance of other doc-
   uments.


 Report of Progress for TREC II


                                          277

*  To examine whether queries can be analyzed and clas-
   sified so that the system can determine which approach
   is most likely to be successful for a particular query or
   concept.
*  To examine whether certain kinds of elements (where
   an element is a word, a phrase, a proper noun, a verb, a
   cooccurrence etc.) can be predetermined to be helpful
   in certain types of queries.


7.0 Results & Conclusion

At this time no results are supported by work performed
with FORMS. However; it is worth emphasizing that the
results in TREC seem to support the view that there is no
specific approach that is going to revolutionize informa-
tion retrieval. Rather, it seems that improvements are
going to come from attention to details and fmding the
right element to use at the right time.