- 32 -Chapter 6.
CONCLUSIONS AND RECOMMENDATIONS

The aims of this project were to conduct a comparative experimentusing Boolean and Weighted retrieval, and to establish the operational
feasibility or otherwise of a weighted searching system implemented on afront end. Despite the difficulty of obtaining sufficient searches toallow strong statistical conclusions, it can nevertheless be claimed
that Weighted retrieval is robust and at very least, is capable ofachieving results comparable to those obtained with Boolean searching.
All of this has been accomplished in an operational environment using
real users, queries, databases, host and intermediaries, on a front-endsystem.
Looking at the comparison in more detail, it is apparent that theWeighted searching was achieved using fewer terms on average that
Boolean (see comment below). Weighted searching seems to cost somewhat
more, both in on-line time and in telecommunication costs; both are afunction of the necessity to transform the weighted search into a seriesof Boolean searches. (In comparison with other recent front-end
systems, little attention has been paid in the design of Cirt toreducing on-line time, e.g. by logging off and on again).
Separating the effects of weighting, ranking and relevance feedback
within Weighted searching is not easy. It would appear that therelevance feedback component has had little direct effect (but seecomment below).
Some limitations imposed by the environment have clearly affected
the potential of Weighted retrieval. Two related points stand out.
(a)

The time taken to search discourages the use of many terms. Thiseffect is as much a matter of perceptions of the Intermediary anduser as of objective time since Weighted searches did not in facttake much longer than Boolean. However, in principle Weighted
searching should benefit from the inclusion of many terms; such- 33 -

benefits are not likely to emerge in the present environment,(b) The implementation of query enhancement (ie adding new terms
automatically or semi-automatically) is not really feasible in thisenvironment, though it might be with a different host. The lack ofthis facility again may have limited the benefits of relevance
feedback.
The fact that Weighted retrieval performed adequately even withthese limitations indicates the possibility at least of genuine
performance advantages, in conditions where these limitations can beovercome.
6uK

Experimental methodology

The lack of statistically significant results confirms the generaltrend of the argument of paragraph 3.1 and Appendix A2, although clearly
the actual numbers required for significance cannot be confirmed
empirically, when significance was not actually achieved.
The overall assessments made by the intermediaries and users
immediately after the search were not of great value in distinguishing
the systems; the more detailed information provided by the logs andrelevance judgements turned out to be more useful in that respect.
The initial intention of using mainly searches performed in otherinstitutions proved extermely difficult to maintain. Free searches
offered at City, while perhaps going against the strict "operationalenvironment" philosophy, proved a very much more successful way ofobtaining a reasonable number of searches. Also the difficulty ofextending the project (5.5) contributed to the final shortfall ofsearches.
6_.2_. Proposals for future research
During the course of the present project ideas for future research
have emerged. One is a diagnosis of system perfomance, which might beinvestigated as follows:(a) Trying to determine the circumstances under which Cirt's innovative
features are useful. This would involve clearly defining thecomponents to be examined, and trying to categorise as precisely as
possible the circumstances and the manner in which these components
perform.

- 34 -(b)Attempting to evaluate whether relevance feedback does contributeto perfomance and if so how, if not why not.

(c)

Deciphering the constraints placed on a weighting ranking andrelevance feedback system forced to operate in a Booleanenvironment.

The proposed methodology for the above would involve:
Examining logs, presearch strategies and the questionnaires fromthe present project.
Expanding the data set of searches to enhance our knowledge of its
capabilities.
Extensive matched pairs on existing searches.
A second area for investigation concerns possible enhancements toCirt, in particular the introduction of a query expansion facility, andtheir evaluation.

Some diagnostic work of the type indicated abovecould provide some evidence as to the possible value of query expansion
in a real-life environment.

Although automatic query expansion has been

investigated in the laboratory, semi-automatic expansion (i.e. offeringterms to the user) can only be properly evaluated in an operationalcontext.

Cirt therefore would provide an invaluable mechanism for suchevaluation.
A third possible area is the role of weighted searching within amore sophisticated front-end. For an intermediary, it would seemappropriate to provide the option to switch between weighted and Boolean
searching, but the logic of such a mechanism is not obvious and needs
further consideration. For the end-user, it would make more sense toembed the weighting facility in a more user-friendly (perhaps expertsystem based) front-end.
6^. 3^

Final remarks

Despite the practical difficulties of providing a usable Weighted
searching facility in an operational environment, and the experimental
difficulties of evaluating it, this project has shown the following:(a)

Weighted searching, according to the Robertson-Sparck Jones
independence model with subsequent enhancements, is a feasible way
to do real searching.

- 35 -(b)Such Weighted searching can be implemented as a front-end to aremote Boolean database, though with difficulty and in a somewhat
limited way.

(c)

Performance of Weighted searching implemented in this fashion iscomparable to that of Boolean searching.