From: sshws@convx1.lerc.nasa.gov (Herb Schilling)
Subject: SUMMARY : Integrating text and data
Organization: NASA Lewis Research Center
Date: Fri, 16 Oct 1992 14:20:33 GMT


Here is a summary of the responses I got to my question about wanting to 
integrate a DBMS with a text retrieval package. 


===============================================================

Hi Herb,


   I saw your message at comp.database. We are right now doing something
   similar to what your want, but much less ambitious. I even think that
   what we are doing here is a kind of "poor's man solution.

   On the service we have here (AlterNex), we have databases available for
   users to query it. We developed an "interface engine" that "talks" with
   the end-user and helps him on formulating his query. Then, using
   CodeBase on SunOS, we retrieve structured data.


   There are control files to this interface engine, that specifies what
   are the fields on each database (by now, a database can just be just a
   flat file...). We are now looking at OpenText's full-text software. Our
   partner in Canada (AlterNex is a node of APC Networks) recently acquired
   a copy and here, at AlterNex, we are going to integrate it to our
   "interface engine". Updating is mostly done by us, not by our end-users,
   but we are going to further develop this too.


   As you can see this is rather small, but, I hope it adds to your survey.


   If you get anything useful, please forward me or to comp.database.


Regards,
Villas.


 ---------------------------------------------------------------------------
Marcos Villas                                       Internet: villas@ibase.br
Instituto Brasileiro de Analises Sociais e Economicas, Rio de Janeiro, Brasil
 ---------------------------------------------------------------------------

========================================================================
Hello,


I realize your question was more about "how can I use existing commercial 
products to work on my application" but I have some academic work that
may be useful. I believe it is practical as I have worked with a couple
of industrial applications and by using some of these ideas they were
able to resolve some real problems.

Part of the work I am on doing on my PhD thesis is devoted to using existing
traditional database systems to assist in text search. Its
not a new idea, but it may be that its more relevant now as more and more
applications are being developed that have your requirements (some structured
data and some text).


I'll spare you the long description of everything and just say that some
of my work has been to use existing relational systems that speak SQL to
do text search. I can send you a couple of papers on it that have recently
been published. The bottom line is that with a little bit of work on your
part you might be able to get sufficient text search capability out of the
existing DBMS that you are using and you could forego the need for a 
specialized text retrieval system like VERITY. The extra amount of work 
depends on your application but my papers have database designs for 
keyword searches, proximity searches and relevance ranking. The nice thing
about using one system is that you get a lot of added functionality such
as concurrency control and recovery that text retrieval packages typically
don't have. Anyway, all my work has been with relational but I've always
thought it could be easily modified to work with object oriented systems
as well. 


I'd be happy to send you the papers. I'd be very interested to hear
more about your application and I'd be interested to hear what you finally
decide to do.


Good Luck.


David Grossman

==========================================================================


Hello again,


Have you looked at BRS Search?  According to their literature, they provide
full integration of relational database and Text Retrieval in their
document management system.


Here is a new (to me) company to check out:


        OCE - Wissenschaftliches
        Forschungsinstitut AG (C)
        Asylstrasse 125
        8032 Zuerich
        Switzerland
        011-41-1-422-0024 (tel)
        011-41-1-422-7001 (fax)


        John Glascock, Marketing Manager
        jg@OWF.ch       E-MAIL


Their package seems quite capable and they seem eager to do business.


We are using a "hack" system consisting of an in-house developed text
retrieval system coupled to the FREEWARE database system, "PostGress".
Text data is fed to the text-retrieval system with the following "virtual"
attributes:


       Each record appears as a separate file to the TR system, with the
        "file name" being the record number.


        Each field of the record is assigned a unique "zone" number.  Text
        from the field is indexed with the zone number so that TR querries
        can (optionally) be constrained to one or more zones (fields).


        Textual data is stored in separate files from the regular database.


       The postgress database contains pointers to (and into) the file
        containing the text for its field(s).  A single file may contain
        text for many records and many fields.  [or all]


        The querry interface is set up so that it all appears to be one
        database using a single software package.


        There is about 3 man-months of development in this system.  As soon
        as we can afford a commercial replacement, IT WILL BE REPLACED!

Disclaimer:
        The above information pertains to the situation as I perceive it at
        Micronetics.  My statements have nothing to do with Sequent
        Computer Systems, Inc., or any other customer of Micronetics.


-_-_-_-_
Daryl V. McDaniel       Symmetry Systems Engineering  (Contractor)
Sequent                         (503) 578-3383          darylm@sequent.com
Micronetics                     (503) 643-6137          darylm@mnet.com

==============================================================


In comp.databases you write:


> We are looking to create a database in which some of the fields will be 
>full text searchable. Since DBMS do not have all the sophisticated 
>capabilities of a full text retrieval package ( like Fulcrum , Verity )
>we would like to use one of these packages to do the searching in the
>text fields and let the DBMS do the querying in more structured fields
>such as date, time,etc... 


That's not quite true.  Some databases do have sufficient flexibility
to handle both structured and unstructured data.  For example, the
several variants of Pick available under Unix provide this capability,
and we have found it to be most useful in a number of applications.
(So, of course, do a number of library systems based on Pick.)  A number
of the Pick variants now support integrated Alternate Key indexing, so
our approach would generally be to create an index on individual 
keywords extracted from the text, use this for initial selection and
then do more detailed comparisons on the actual text in the selected
records if required.


A lot of our work involves a mixture of structured data and what I call
"messy" data (unstructured, variable length fields, multivalues) and we
have found the Pick-style systems (actually, Prime "Information",
although I believe the product has now been sold to V-Mark) to be
excellent for this kind of application.


> What we are looking for is information on how to go about doing this. We
>have some ideas and the vendors have helped some but not much. Is
>there anyone out there who has done something like this ? Actually, we are 
>going to use an OODBMS ( Object Store ) as the DBMS but examples
>using a relational database would also be useful. I will summarize
>any replies I get.
Is it permissible to ask why select a product which does not
provide all the functionality you require?


> By the way, we have not selected a text retrieval product yet.


I really have doubts about the desirability and feasibility of
grafting this functionality onto a system which does not support
it intrinsically.  However, I hope you get it to work.  Certainly
my preferred approach would be to look for a single product
which either provides the functionality up front or at least
provides an environment in which it can be rapidly developed.


Geoff Miller  (g-miller@adfa.oz.au)
Computer Centre, Australian Defence Force Academy
-- 
Herb Schilling , NASA Lewis Research Center , 21000 Brookpark Road, MS 142-5
Cleveland, Ohio 44135 . (216) 433-8955       sshws@convx1.lerc.nasa.gov
Newsgroups: comp.databases