From: atherton@unixg.ubc.ca (Bruce Atherton)
Subject: Re: Adapting to post-boolean searching
Organization: University of British Columbia, Vancouver, B.C., Canada
Date: Wed, 19 Aug 1992 21:47:25 GMT

In article <1992Aug18.090308.29292@dcs.glasgow.ac.uk> sanderso@dcs.glasgow.ac.uk (Mark Sanderson) writes:
>
>Thats all very well saying that but if the user doesn't know any other words
>then there is a problem. James Allan's boolean system produced better results
>than WAIS.

It is very easy for a boolean system to get better results than one using
one of the new systems for text, such as the Vector Space Model.  If
you specify all of your query terms connected by OR, you will get the best
recall of relevant documents that you can get.  Unfortunately, you will also
get a lot of junk.  If you give a query where the terms are all joined by AND,
you get the best precision you can while losing a lot of recall.  Many relevant
documents will never be returned.  "secret" AND "document" will miss references
to confidential or classified documents, no matter how much they may be
discussed.

It depends on your particular search.  If you are just trying to find the
answer to a question, all you need is one document that contains the answer.
It doesn't matter what your recall is, as long as you get one.  In that
case, relevance feedback is irrelevant and you may be better off with boolean.

If instead you are trying to find several relevant documents, boolean becomes
a real hit-or-miss operation, especially for someone unskilled in its use.

As an example, we are writing software that allows lawyers to retrieve
relevant cases to use as precedence when they go to court.  There are several
systems available which allow boolean queries to be made, but they are all
but incomprehensible to most lawyers.  The results that are returned are
lacking as well.  The end result is that law firms still keep massive
libraries of both cases and indices so they can find the case manually. 

>The document ranking procedure in WAIS should be changed, to put a document
>with multiple single hits above a document containing all the user's query
>words seems wrong. Probably the better way of organising the ranking is to 
>rank documents based on a binary occurance (is the term in the doc or not ?) of
>query words in the document. This way documents containing the largest number
>of query words will apear at the top of the ranking.

There have been a lot of tests done on this problem, and the solution you
describe does not give the best results in general.  I have spent a great
deal of time researching the results given by various systems in the last
six months, and returning documents based on how many terms match the
query gives a real boost to documents that mention terms in passing.  It
also hurts when synonynms are used.

The trouble is that no one system can best meet all needs.  There are a number
of enhancements that could make searches more effective under WAIS, however:

  1) Allow each query term to be weighted.  If a single term seems likely
     to swamp a search, lower its weight and increase the weight of other
     terms more important to your search.  You could even allow for a
     negative weight, which simulates a boolean NOT without causing such
     drastic consequences.

  2) Allow for phrases.  That seems to be the biggest problem that people
     are complaining about right now.  Because you start out with a weak
     query (such as "boolean search") where most terms are common ones,  
     the results aren't all that good.  Specifying that the words form
     a phrase solves that problem.  Note that this solution is far better
     than just going back to boolean, because it can handle problems like
     finding documents about "The Who", which causes boolean to choke as
     badly as the VSM.

  3) Adjust the document term weight calculation so that the term frequency
     has less of an impact.  I'm not familiar with how WAIS calculates
     the weight, but it could be that it needs tweaking for this application.

At the 1990 SIGIR conference, Cyril Cleverdon who is one of the original
investigators into text retrieval said that he had devoted his life to trying
to eliminate boolean systems.  Anyone who has studied the limitations that
boolean has on text systems can't help but applaud that, IMO.  Don't go back to
a terribly flawed system when some simple changes could fix the current one.
--
UBC Faculty of Law Artificial Intelligence Research (FLAIR) Project
atherton@unixg.ubc.ca or | You're built too low to the ground, boy!  The fast
bruce@flair.law.ubc.ca   | one go over your head. Ya got a hole in your glove.
Newsgroups: comp.infosystems.wais