X-Mailer: ELM [version 2.3 PL11]
Newsgroups: bit.listserv.notis-l
Date:         Fri, 2 Oct 1992 17:36:28 EDT
From:         timothy@ASA.UGL.LIB.UMICH.EDU
Subject:      Re: Location Based Catalogs
Comments: To: NOTIS-L@VM.TCS.Tulane.EDU

We were a beta site for LMS 5.1, and have had location-based
catalogs in production since the end of May.  Impact on resource
usage has been minimal; but only because use of the feature has
also been minimal.  Only about 1% of searches are being done
using scoped catalogs.  Of those searches however, response
time is up dramatically; the reason is primarily increased
I/O, but there's also an increase in cpu useage.  Let me try to
explain why.

When a search is scoped, each hit returned from the index is
processed against the location filter file by reading the filter
file record assocated with the notis record id from the index
hit (that's the I/O part), and comparing each location in the
filter record with each location in the searcher's location based
catalog (that's the cpu part).

Generally, searching Notis blocked indexes (such as the MHIX) are
very efficient with respect to I/O's.  For example, the search
"a=king", when unscoped, gets 3753 hits, does 81 I/O's to the MHIX,
and has a response time of about 3.5 seconds.  If I do the same
search scoped to the Undergraduate Library (about 200,000 records),
we get 472 hits and response time jumps to a whopping 91 seconds.
I/O's to the MHIX have stayed the same, but now we have 3500 or so
I/O's to the LBC filter file.  In this case, the I/O time for the
filter file was about 84 seconds, averaging about .025 seconds per
I/O.  And there's also a pretty hefty increase in cpu time used,
from .325 seconds to 5.25 seconds, which comes from location
comparisons.  In general, the I/O part of the response time increase
can be calculated as (# of hits) * (avg I/O time for filter file).
Cpu time increase probably depends more on how many locations are in
each LBC, and I don't have a formula for it.  In our system,
I/O time dominates.

The above is true whether you use LBC index option 1 (only display
entries within current scope) or option 2 (highlight entries within
scope), since the filter file must be consulted in either case.
A worst case scenario occurs when option 1 is uses, and a search is
entered which, when unscoped, has more than 5000 hits.  When this
search is scoped, it's possible that many more than 5000 hits will
be processed against the filter file, since hits will continue to
be processed until either they are all processed, or until 5000
hits have been found within the current scope.  In our system,
searches like these generally get abended by TMON, exceeding our
CPU time limit.

In general, searches which return a smaller number of hits when
unscoped (less than 500) don't suffer so much when scoped.  For example,
"s=dental caries" gets 244 hits, response time 1.14 secs, I/O time of
1.08 seconds, cpu time of .038 seconds when unscoped.  When scoped to
the Dentistry Library, the same search gets 231 hits, response time is
5.6 seconds, I/O time is 5.3 seconds and cpu time is .38 seconds.

Notis may be working on tuning this, but I don't know how much
they can do with the current design.  One thing we suggested was
to have a different index hit threshold (significantly lower than 5000)
for scoped searches.

Timothy Prettyman
Library Systems Office		313-764-0412
University of Michigan		INTERNET:  timothy_prettyman@um.cc.umich.edu
Ann Arbor, MI  48109-1185	BITNET:  USERLM5C@UMICHUM
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU