Chapter 6
Assessment of Evaluative Methods
Introduction
Whilst taking into account the results of the experiments reported in the previous
chapter, the following discussion primarily aims to provide a critical assessment of theevaluative methods applied to address the different research questions. In testing thevarious methods and approaches, the criteria which were considered included, a) the
information which could be elicited from the user, b) the validation of the data byother methods, and c) the practical limitations in applying the methods.

6.1

Holistic vs 'black-box' approachThe holistic approach adopted in Experiment 1, which included the observation ofsearching at the library shelves by catalogue users and non-users, revealed a) a
predominance of subject searching, b) an inherent need for subject searching and c) thelimitations of the library catalogue in meeting that need. This brings into question thedual assumption of information need and the use of the library catalogue in meetingthat need. The 'black-box' approach of previous studies emphasized the distinction
between known-item and subject searching at the library catalogue, and concluded
that the use of the catalogue as a finding tool reflected users' basic requirement for
specific item searching, thus minimizing the need for subject searching.
The study of information seeking in the context of the library environment as awhole also brought to light the role of known citations in subject searching. The
identification of hybrid searches whereby one in three searches initiated as specificitem searches developed into subject searches either at the catalogue or at the shelvesis clearly significant (Section 5.1.2).
Although following users to the shelves and directly observing the searching activity
at the shelves proved to be fruitful, it is an approach which is very time consuming andcan only realistically be applied in a limited way. Hence the project also consideredother methods to see which could be applied more easily and also be as informative.
59

6.2

Observation combined with talk-aloud technique

By adopting a holistic approach it was possible to record observable behaviour in acontinuous fashion from the beginning to the end of a search. The talk-aloud techniqueapplied in Experiment 1, was used to complement or confirm what was being observed
without having the user provide any explanations for the actions taken. Searchers
were asked to talk-aloud but were left to choose their own level of verbal comment sot h a t the amount of information volunteered varied with individuals and with different
search situations.
Great care was taken by the experimenter not to interrupt but to leave the searcher
in control. As such there appeared to be no evidence of interferring with the continuityof the search process. Whether or not interference could be manifested in other ways
has not been determined. In leaving the level of interaction between the experimenterand searcher at a minimal level, the method did not inform on searchers' full intentions,motivation or perception of the task or problems encountered. It may have beenpossible to have gathered the same information by observation alone particularly at
the shelves, but being observed in silence may have been more intrusive. Equally it
may have been more disruptive for the searcher if the experimenter had interrupted
intermittently.
At the online catalogue, the verbal data provided by the user was essential toindicate choices and decisions made in the course of the consultation and particularly
the information extracted for further searching at the shelves. Through the combinedd a t a gathering methods of observation and talk-aloud, the object was to record thefull progression of searches. Any attempt to seek more indepth information, relating
for example to search formulations, would certainly have encroached on the searchingprocess.
Searchers did not have any difficulty in providing a running commentary. At thecatalogue the experimenter recorded the data which would assist in observing thesearcher at the shelves, thus duplicating some of the data already recorded in the log.
The real time interpretation and recording of data on a structured form served as anefficient way of getting around the tape recording, transcription and coding associatedwith the use of protocol analysis.

6.3

Basic screen logging facility

The screen logging facility used in Experiment 1 provided a comprehensive source ofd a t a of the searching activity at the terminal but did not include any data pertaining
to the user's interpretations, judgements or intentions. The latter was obtained fromthe user's verbal data. Hence the analysis of each search required both sources of data.
Interpreting the logs independently from the recorded verbal data even with user
sessions being clearly demarcated still proved to be difficult, particularly in deciphering
individual searches within a search session and establishing whether a session pertained
to an individual search or multiple search. In fact there were very few multiple searchsessions. In the vast majority of cases what in the log may appear as possible separate60

searches, were in fact interrelated.

6.4

Combining screen logging with questionnaires

Experiments 2 and 3 explored the possibility of complementing screen logs withinformation obtained from written questionnaires as a possible alternative to theholistic, observation, talk- aloud approach adopted in Experiment 1. The questions
were thus designed to inform not only on the catalogue search but on the information
seeking activity as a whole.
In addressing the question of subject searching strategies, the results in Experiment
2 show users' clear intent to undertake subject searches via a specific item search, i.e. ahybrid search, irrespective of when (pre-search or post-search) or how (offline or online)
the questionnaire was put to them. These findings which are comparable to the resultsof Experiment 1, confirm in turn the reliability of eliciting this type of informationfrom users' through the combined methods of questionnaires and screen logs. Users'intent appears to be independent of the catalogue consultation, i.e. users expect toundertake subject searching at the shelves in addition or irrespective of the catalogue
consultation.
However, if the logs are taken on their own, just over a third of the searches do notin fact represent users' full intent. It would appear that basic screen logs alone are nota reliable source for identifying types of searches and are thus not an adequate tool formonitoring searching behaviour.
The strength of the combined methods of eliciting information from users fromquestionnaires as well as the logs is also confirmed in Experiment 3, where users
were asked to articulate their information need in a formalised statement. Again
there seemed to be little difference between the information elicited before or after thecatalogue consultation online or offline.
However seeking that type of information from the user was difficult. Although 48%of searchers did distinguish between their search articulation and search formulationto access the catalogue, 52% did not. Encouraging users to articulate a ' t r u e ' orfull representation of their need (if known) remains problematic. It would seem that
this type of information is too complex to elicit by using a direct open ended written
question. Nevertheless both of these experiments established the reliability of presearch and post-search online questionnaires.

6.5

Online interactive questionnaires (Olive)

In Experiment 4, the online interactive version of the pre-search and post-searchquestionnaires used in Experiments 2 and 3, tested the viability of intercepting theuser in the course of a search. Although the samples are small, the results do not seemto differ from those obtained from the pre-search and post-search questions. There
were no obvious indications that the interceptions and question interferred with thesearching process.
61Interceptions could only be triggered after a restart, and conditions for the questionswere limited to events in the previous chain or search sequence. It soon became
apparent that the combination of determining other appropriate interception points
in addition to setting conditions across chains, would prove to be very complex inpractice and would require extensive experimentation. Moreover the implementationof multiple interceptions and multiple sets of conditions would also necessitate a set ofclearly defined research questions as well as questionnaires carefully designed in putting
appropriate and well formatted questions to the user.
There is scope in further testing the existing facility more vigorously. Although
the development of more complex mechanism would seem to be feasible this should beparalleled with prerequisite information on users' internal cognitions.
It must also be borne in mind that the necessary testing of such a facility in an
operational library setting to take into account a sufficient number of cases for each of
the experiments particularly for paired sets, requires an considerable amount of effort
and resources. The five experiments conducted in the present project included aroundsix hundred individual cases.

6.6

Search replays

Although the use of replay facility in Experiment 5 was exploratory, this method
demonstrated much potential and deserves further investigation.
Users had no difficulty in recalling the motivations underlying their search as theysaw it replayed. At the same time users did not perceive the replay as a new search andactively search the second time around. The method clearly provided greater insight
into user's perception of the searching process than any of the other methods describedabove.
Through the replay facility it should be possible to focus on specific aspects of theuser interaction. This in turn could contribute to the formulation of more appropriatequestions for use on a wider scale in online interactive questionnaires.

62