- 21 -Chapter 4.
RESULTSThe results discussed in this chapter are based on the 190 searches
finally obtained.
The analysis was essentially directed at the comparison question —i.e. how do Weighted and Boolean searches compare. Essentially eachvariable included was subjected to a basic distributional analysis, andalso cross-tabulated with the "Boolean or Weighted" variable. In thecase of (nominal or ordinal) category variables, a chi-square test wasthen applied; in the case of quantitative variables, the Mann-Whitney U
test was used.
Because of the time available, not all variables were included in
the analysis, and even for those included, not all possibly interesting
analyses were performed. There remains, therefore, a quantitiy of data
for future researchers to work on. In particular, there is some scopefor diagnostic work on the logs, with a view to understanding andcategorising the particular situations in which Weighted searching doesor does not work well.
Full tables of the results are given in Appendix A8. Tables arenumbered to correspond to each of the sub-sections below (e.g. 4.2.1),
and the existence of a table in A8 is indicated in the appropriate subsection. "Significance" in the tables and "p" in the discussion below
mean the same thing; a significance criterion of 5% has been used, i.e.
a p value of less than 5% has been taken as significant. The following
is a brief discussion of each variable analysed. The results arefurther discussed by category of variable in section 4.6.
4_.j_. Results from the presearch form
These results are not expected to show any difference between
Boolean and Weighted, and serve as background information only.

- 22 -Among the categories of users (Consultant/ Doctor/ Lecturer/ Postgraduate/ Researcher/ Other) approximately 40% were postgraduate
students, with doctor, researcher, other each accounting for a littleover 15%.

Almost all (85%) of searches were for research purposes.

60%

of users described their search topics as Precise; also 60% wanted abroad search.

60% again had previously had online searches done forthem; 15% had done searches themselves (without an intermediary).
4_.2_. Results from the post-search user questionnaire4^.2.K Indicate your SATISFACTION with the search on the basis of thescale below (excellent/ good/ satisfactory/ poor/ bad).
(Table in A8.)
For both Boolean and weighted, the modal response was Good, withvery few Poor or Bad. No significant difference between Boolean andWeighted, although there may be some advantage to Boolean.
4_.2_.^. Please provide a^ general assessment of the SEARCH (easy/average/ difficult).
(Table in A8.)
Mode is Easy; there is absolutely no difference between Boolean andWeighted.
4^.2^.3_. Generally speaking, were the RESULTS of the search:
good/ satisfactory/ poor/ bad?

excellent/(Table in A8.)
For both Boolean and Weighted, the modal response was Good, withvery few Poor or Bad.

No significant difference.

4^.4^. Please assess the SEARCHER'S CONTRIBUTION t£ the_ search(essential/ helpful/ satisfactory/ poor/ bad).
(Table in A8.)
About 75% said Essential. Difference between Boolean and Weighted
not significant (p*13%), but a slight tendency to give a lower rating
for Weighted.

- 23 4_.2_.!>.• How close was the online search to your original or intended
enquiry (exact/ fairly close/ considerably altered)?
(Table in A8.)
Mode is close; difference not significant (p=12%), but there issome indication of a greater spread (i.e. more extreme values) forBoolean.
^.2_.6^. Did £Ou GET th£ number of_ REFERENCES EXPECTED (less/ asexpected/ more)?
(Table in A8.)
Mode is As Expected; no significant difference.
4^.2^.7^. (Weighted searches only) Did you mark any references asrelevant?
(No table.)
Of the users who answered this question, almost all said Yes, andabout 75% said that it appeared to make the search more effective.
However, only about half the users answered the question.
4_.2_.8_. (Boolean searches only) Did you view any references while
online?
(No table.)
Similarly, only about half answered the question; of these, almost
all said Yes; about 65% said that the search was modified as a result.
{*•$•

Results from the post-search Intermediary's questionnaire

Apart from the questions discussed below, the remaining questions
were not expected to show differences between Boolean and Weighted. The
average presearch time reported was 18 minutes.
4^.U
Indicate your OVERALL SATISFACTION with the search (excellent/good/ satisfactory/ poor/ bad).
(Table in A8.)
The mode is Good. The difference between Boolean and Weighted isnot quite significant (p«6%); although each has the same proportion of

- 24 -

Excellent and Good, Weighted gets somewhat fewer Satisfactory and more
Poor and Bad ratings•
4^2.2_.

Please provide £ general assessment of the SEARCH PROCESS (easy/average/ difficult).
(Table in A8.)
Mode is Average; difference not significant (p»17%), but there isapparently a slight advantage to Weighted.
^•3_.3_.

Generally speaking were the RESULTS of the search: excellent/

good/ satisfactory/ poor/ bad?
(Table in A8.)
Again the mode is Good; no significant difference.
4_.3_.4_. What was your REASON FOR FINISHING the search?

(Found what wasrequired/ Technical difficulties/ Search strategy failed/ Other)(Table in A8.)
Virtually all searches indicated "Found what was required", but
technical difficulties were also reported in 7% of searches. (It shouldbe mentioned that searches which were aborted at an early stage or
before any output was obtained were either re-run or excluded from thesample).
4_.4^Results from logs

60% of searches were on Medline only, and 25% on Inspec only; theremainder involved Psychological Abstracts and/or more than one
database.
flLm!L*Lm PSS packets sent and received(Table in A8.)
Skew distribution with a long tail to the right; highly significant
difference, with the mean for Weighted more than twice that for Boolean.

- 25 *.•*.•£• Q n l i n e time
(Table in A8.)
Average 22 minutes.

Skew distribution; difference not significant(p-20%), but some indication of a longer time for Weighted: mean for
Weighted about 25% higher than for Boolean,^-•—•2/

Online citations(Table in A8.)
Skew distribution; significant difference (p<5%), mean for Weighted
35% less than for Boolean.
A_.4_.4_.

Offline citations(Table in A8.)
Skew distribution; significant difference (p<5%) with fewer for
Weighted (mean 30% less than for Boolean).
4_.4^.

Terms used in the search, terms added or ammended(Table in A8.)
Both variables show a significant difference (p<l%), with fewer for
Weighted.
4_.5_. Results from the relevance assessments!t*l.mLm

Total number of documents assessed(No table.)
Difference not significant.

Note that this number is generally thesame as the number of offline citations, except that it is truncated to50.

Average is 36.

4_._5_._2_. Total relevant retrieved(Table in A8.)
Difference not significant, either for relevancel (p=ll%) or for
relevance2 (p*15%). Nevertheless, for both the total is smaller for
Weighted (by about 12%).

- 26 -4_.5^2-Precision(Table in A8.)
Difference again not significant, for either precisionl (p«36%) orprecision2 (p=22%).

In both cases mean precision for Weighted isslightly higher than for Boolean: 50% against 45% for precisionl, 73%
against 69% for precision2.
4^.6_.

Summary of results by category of variable

4^.6_. 1_. Retrieval effectiveness
Essentially, there appears to be little difference between thesystems.

There could be a tendency for Weighted searches to achieve

higher precision with lower recall, but the difference is within thebounds of sampling error.

If there is such an effect, it is likely torelate to the number of terms used (see 4.6.7 below).
The matched-pair data and the novelty data have not yet been
analysed.
4_.6^2_.

User effort

The users' subjective assessments of the difficulty of searchshowed absolutely no difference between weighted and Boolean. Users
spent slightly more time on Weighted searching simply because the online
time was greater (see next paragraph). Fewer terms were used (see 4.6.7below) and fewer citations looked at online; on the other hand, thesystem required an online relevant/non-relevant response in respect ofeach item looked at.
4_.6_.3_.

Cost

Online time was apparently slightly greater for Weighted (thoughnot significantly so), almost certainly because of the large number ofBoolean search statements generated by the weighted searching algorithm.
This would have affected four separate categories of cost: host charges,
telecommunications charges, intermediary's time, user's time. On- andoffline citations were fewer with Weighted. It is not quite clear why
this should be (though for offline citations, it is consistent with thelower recall result).

- 27 -!Lm*Lm!Lm Subjective user reactions
There were no significant differences in this category at all.

Thevery slight differences observed do not appear to form part of aconsistent pattern.
4^.6u5_.

User characteristics, request characteristics

Previously summarised from the presearch form.

Obviously we do notlook for differences between the systems in these variables.

It might

be appropriate to use them as independent variables against which toevaluate changes in performance etc., but this has not been attempted inthe present analysis.
4^.6^.6^.

Intermediary7 s contribution

One might look for a reduced intermediary's contribution for
Weighted, in that it should (in principle) involve less effort inconstructing search statements. In the users' opinion, this may be thecase, though the difference observed is not significant. But various
indirect pointers suggest something a little more complex.
Intermediaries seemed to take the task of learning how to use the system(both at the low level of learning the command language and at thehigher level of learning how to make best use of the system) as
seriously as any other online search system. They seemed to find it
easier for some queries and harder for others; they would perhaps havewelcomed weighting techniques as an addition to their range ofpossiblities, rather than as an alternative to Boolean searching.
4^.6^.7_. Perceived online time
The most consistent group of observed differences between thesystems apparently related to the perception of online time on the partof the intermediaries. It was apparent from comments that they feltthat weighted searches were taking too long, in particular at the Searchcommand (when the algorithm is generating and transmitting numbers ofBoolean statements). Since this time is critically dependent on thenumber of terms, they tended to cut down the number of terms. This wasobserved directly, and also probably led to the apparent reduced-recall,increased-precision result for Weighted searches.
It is interesting to note that the average time for Weighted
searches was in fact very little higher than that for Boolean. It issuggested that the intermediaries' perception was perhaps a little
distorted, probably because they had nothing to do while the Search- 28 -

command was running.