Workshop report: Usability of Digital Libraries @ JCDL’02
Ann Blandford
UCL Interaction Centre
University College London
London WC1H 0AB, U.K.
A.Blandford@ucl.ac.uk&

George BuchananInteraction Design Centre
Middlesex University
London N14 4YZ, U.K.
G.Buchanan@mdx.ac.uk

The motivation for organizing a workshop on “Usability of Digital Libraries” at JCDL’02
was the recognition that digital libraries will only realise their potential when they areusable, useful and used by a broad cross-section of users. An interviewee in one of ourearlier studies on experiences of working with digital libraries commented that “It’s likebeing given a Rolls Royce and only being able to sound the horn”; in other words, digital
libraries are potentially powerful tools, but only if users are able to harness that power
and take it in fruitful directions. As with all new technologies, there will be early adopters
who will persevere with using systems almost regardless of the costs and benefits;
however, to have the impact commensurate with current investment, it is essential that
there be take-up by a much broader community of potential users. As one participant in
the workshop observed, the notion that “‘build it and they’ll come’ is a lie”.
The workshop took place on Thursday 18th July, with 32 participants. Participants
represented a variety of backgrounds and perspectives on usability, from practitioners
with responsibility for introducing digital resources within their own libraries, whose
experience was well grounded in the demands of delivering effective services here andnow, to researchers from various disciplines (human–computer interaction, information
science, social science, computer science), who took more theoretical approaches toissues of usability.
Ten papers were presented at the workshop. These covered both practical and theoreticalaspects of the topic. Practical presentations described both empirical and analytical
approaches to evaluating the usability of particular digital libraries.
Matt Stoeffler described a “User study of multi-collection text retrieval”, in which theytested users’ ability to complete pre-set tasks within a library comprising multiple
collections. One of their important findings was that, while the concept of a ‘collection’within a library is often helpful to librarians and curators, it causes great confusion for
most end-users, who have difficulty anticipating which collections desired information
might be found in. In a similar vein, Jessica Hovater presented the results of “Usability
testing of the virtual data center”, through which users can both access data and perform a
range of analyses on that data (e.g. statistical tests). One of the striking features of thissystem is the approach to integrating information seeking and information use within one
streamlined system. Taking a more strategic approach, Casey Jones and Tamara Sumnerpresented a discussion paper on issues surrounding “Evaluation of the National Science
Digital Library”. Here, the question is not how to evaluate a particular system, but how toevaluate and improve an organisational structure and development process – a much lesswell defined problem.

Cédric Dumas entertained everyone with a demonstration of three contrastingvisualisation systems that are supporting a formal usability evaluation of various
representations of the same library system. This was accompanied by a presentation of apaper, “Libraries: comparisons between the real and the virtual in 3D, 2D zoomable and2D arborescent”. This provoked substantial discussion on the relationship between users’
experience of physical and virtual libraries, and how these experiences might reinforce or
disrupt each other. In a presentation entitled “An investigation into the application ofClaims Analysis to evaluate usability of a digital library interface”, Suzette Keithaddressed a different kind of gulf – that between usability specialists and developers. In
particular, she highlighted the difficulty for developers of recognising potential user
difficulties when working with a new system because they have (typically) built “the best
possible system”. Whereas the approaches presented in the first few talks were essentiallycontrolled empirical studies or analytical, the following two talks took a more explicitly
qualitative approach to evaluation and design.
Julie Parker discussed a user study based on in-depth interviews with specialist end-users
on “Searching digital image libraries”. This talk raised issues such as how the activities
of direct searching compare with using an expert intermediary (a librarian who is familiar
with the image library), of how the activity of searching for images fitted in with otheraspects of a user’s work, and of how images with particular, impossible to articulate,features could be located or eliminated from the search.
After lunch, the focus shifted somewhat from evaluation to design. Bob Sandusky
presented a paper on behalf of Ann Bishop and Chip Bruce on “Usability research as
participative inquiry”. This work again raised issues about how to engender effective
interactions between different stakeholder groups – in this case between (often
disadvantaged) user groups, system developers and information providers. The
importance of valuing users’ prior experience as contributing to design was emphasisedin this work. The following two papers both sought to develop a better abstract
understanding of the information search process, independent of users working with aparticular system or in a particular context. Hanna Stelmaszewska presented work on“Patterns of interactions: user behaviour in response to search results”, in which she
focused on the difficulties users experience when search results fail to match user
expectations, particularly in number: different user strategies for narrowing results sets
(when too large) and for expanding sets (when too small – typically ‘no matches’) were
presented; in particular, users’ tendency to give up searches if they fail to get good results
within two or three attempts was highlighted. Kyunghye Kim presented “A model-based
approach to usability evaluation for digital libraries”, in which she proposed a processmodel for understanding the search process, and illustrated how the process model can beused to support usability evaluation and re-design.
Finally, Bob Sandusky presented preliminary work on “Digital Library Attributes:
Framing usability research”. In this he proposed a classification scheme for categorisingvarious aspects of digital libraries, including system designs, contexts of use and design
processes. This framework was welcomed by many practitioners in the audience asproviding a good ‘checklist’ for aspects to consider when designing and deploying new
digital library services.

One of the great strengths of the workshop format is that it provides plenty of opportunity
for discussion, and this workshop was no exception. Inevitably, discussion revolved morearound unresolved issues than (for instance) past achievements, although the importanceof these should not be underestimated. Various themes emerged through the presentations
and discussions, as follows.
On evaluation:• One question that arose several times was: “what is usability?” There was broad
agreement that the term includes many aspects: as well as performance measures suchas efficiency of interactions, avoidance of user errors and basic ability of users toachieve their goals with a system, ‘usability’ should account for affective aspects ofinteraction – e.g. ‘frustration’. In addition, it is important to recognise that usability
includes the larger context within which the search is taking place.

• In the course of the workshop, as illustrated above, a variety of methods of usability
evaluation – qualitative, quantitative, model-based, participatory – were presented. At
present, we have a relatively immature understanding of what techniques areappropriate for addressing particular aspects of design and evaluation. This isdefinitely an area for further work: while usability specialists may have the skills todefine suitable methods depending on the context in which they are working, theconstraints they are operating under and the particular questions a study is aimed ataddressing, this understanding needs to be codified so as to be accessible to nonspecialists such as librarians. A particular question concerned sample sizes: many of
the studies reported were based on small numbers of subjects (e.g. 6 - 10); in somecases the small sample size may represent a ‘short cut’, or lack of rigour in the study;
in other cases, notably where the analysis of each user is in great depth, suchapparently small numbers may be appropriate. In summary, there is a need for further
work on methods – including an understanding of how to balance rigour,
appropriateness of techniques and practical limitations – for analysing usability.

• One of the particular difficulties faced by many developers of digital libraries is thatof knowing who the users are. This presents a challenge in selecting representative
samples for (empirical) user testing, for creating scenarios of use to guide design andevaluation, and for identifying user groups for participatory design. An associatedchallenge is that of devising appropriate incentives for study participants: why should
particular users invest time in helping evaluate novel digital library systems? This, ofcourse, recognises the need to study representative user groups, as other studies areliable to result in designers / evaluators making inappropriate inferences.

• Associated with the difficulty of accurately identifying user groups is that ofaccurately identifying core classes of user tasks that the system is expected to support.
There was general agreement that user goals and design solutions need to ‘fit’ … but
there is a cycle of interactive development, whereby systems are (in principle)
designed to address recognised user goals, but users may then recognise new
possibilities which leads to the definition of new goals, which the system is not (yet)designed to support effectively. Particularly in the context of empirical evaluation,
therefore, one of the outstanding challenges is to identify reliable means of defining
appropriate user tasks for testing.

• It was recognised that – probably more so for digital libraries than most other kinds ofcomputer systems – there are different types of users. One obvious dimension is thatof novice / expert (or ‘power’) user; the process whereby general information users
gain expertise in information seeking is as yet poorly understood. Another dimension
is that of end-users and librarian, where the main differentiating feature between thesegroups is the way in which information retrieved is subsequently used; as one
librarian put it (with a smile on her face): “we’re not normal people working onreference desks”.

• It was recognised that there is little understanding about what user assumptions areabout what constitutes a library. In one of the presentations, the library was described
as a ‘buffer’ between user and data, which is rather different from a user’s experienceof a physical library. The ‘library metaphor’ that is intended to support users anddevelopers in thinking about what is possible within, and gain appropriate
understanding of, a digital library may at times help and at other times hinder theuser’s developing understanding and ability to work effectively with a particular
digital resource.

• There is a general problem of user familiarisation – with process, content, and otherfeatures of any particular library. Pierre Cubaud presented a graph giving one
measure of user competence with a particular library that indicated – though further
research would need to be done to validate the assumption – that general usercompetence is not perceptibly changing over time (meaning, here, two years, rather
than hours or days), although it is likely that that of individuals is improving. One of
the tensions in library development is that between rapid evolution, as new problems
are recognised and new design solutions become possible, and maintaining
consistency of interaction so that users’ familiarity with a particular system – andhence users’ competence in using it – can be retained.

• One particular consensus that emerged was the inappropriateness of the ‘collection’as a user concept. While this concept has a place within many physical libraries(especially where users can easily ask a librarian where particular resources can befound), and is perceived as significant by many information professionals, it has beenfound to create a barrier for end users, for whom the focus is generally on theinformation rather than the route to finding it.

• Similar problems were observed with the use of metadata fields for search – manyusers simply not comprehending the role and purpose of different fields. However,for some specialist user groups, that very facility was very important where the roleof words (e.g. in either author or title field) had a significant impact on meaning.
Again the target user group would have a significant impact – general access libraries
tending to rely less on metadata for access. Problems with cross-collection searching,where metadata formats varied, and metadata quality could vary, were also identified
as related problems.

• One theme that emerged in discussing the relationship between physical and digital
libraries was that of understanding the roles of information intermediaries as experts
mediating search. The conditions under which expert intermediaries contributepositively to the user’s interaction with information, and when users are better off
searching for themselves, are not yet at all well understood.

• We were, rightly, reminded that even though digital library use can be difficult foranyone, the needs of less able users present additional challenges for design.
Users and uses:
The requirements on design and evaluation are strongly informed by an understanding of
how systems are used, and here the experiences of professional librarians wereparticularly informative.

• It was observed more than once that, for most users under most circumstances, use ofdigital libraries is discretionary. While use is made attractive by the ease of accessing
the desktop (you don’t even need to leave your office!), there are generally otherpossible sources of information, such as the physical library or asking a friend.
Indeed, with the proliferation of digital resources, users now have wider access to
information and may use libraries other than the ones designed specifically for them.
As one participant observed, “what they need is information, not to use this library”.
However, it has also been observed that digital library access can sometimes increase
access to the physical library resources too, as users gain awareness of theinformation resources that are available.

• There is a widespread need for integrating physical and digital resources andunderstanding who has access to one, the other or both, and hence how the different
resources may complement each other effectively. The value of instant gratification –e.g. for physical users who can quickly be shown to the right place by aknowledgeable librarian – cannot be underestimated.

• Users don’t use systems as expected! For example, users have often been found using
internet chat to communicate with librarians from within the library, when thelibrarians are in the same physical space. The impact of available techniques foraccessing information and support in different contexts need to be better understoodin order to design better information use environments.

• A need to log in has been found to be a deterrence for many users (yet another username and password); there is a need to work towards integrated information services
for each user that address security, privacy and related issues effectively.

• Students have poor information handling skills. This deficit needs to be addressed –whether by improved training in information sciences or by devising design solutions
that enable users to better develop skills through the activity of information search.
Design:While many of the presentations focused primarily on evaluation of systems, it wasagreed that the main need is for solutions, not just problem identification (though the one
rarely happens without the other). There is usually a need for making trade-offs in design,and one of the key design challenges is knowing how these trade-offs are best made forparticular design contexts.

• Effective design demands effective communication between different stakeholder
groups (librarians, users, developers, HCI specialists …). The roles of users as
designers and of design as a social process were discussed.

• Some practitioners applauded the idea of producing checklists of design issues to beconsidered. More generally, there was a recognised need for tools and techniques tosupport designing for usability.

• There is a need to consider potential users as well as existing ones – e.g. as socialgroups. This links back to an earlier observation made: that it is difficult but
necessary to know (as well as possible) who the intended users of a system are. It is
important to recognise the essential difference between publicly accessible and‘closed’ libraries, and the different possibilities in the different situations – e.g.
publicly accessible libraries cannot make assumptions about what technicalinfrastructure is available to users. At times, it may be necessary to choose the focus
of users, accepting that other people may access the system even though they are notnecessarily well catered for by it.

• The issue of interface design and exploiting users’ existing knowledge and capability
was discussed. For example, the question was raised as to whether being ‘amazonlike’ or ‘google-like’ was an appropriate target for design. However, no conclusion
was reached, so this has to remain an outstanding research question.

• Issues of tailoring interfaces to different user groups – e.g. students, academics, others(maybe even other librarians as a particular expert group) were discussed. The
relative merits of adaptive, tailored and customisable interfaces were discussed,
without reaching any strong conclusions. It was, however, observed that users have atendency to believe that other user groups may be ‘getting a better deal’ if they areaware that there are multiple interfaces, which may lead to resentment.

• It was observed – somewhat forcibly – that core services are not ‘usability neutral’.
That is: technical specialists cannot design and implement fundamental structures andalgorithms (e.g. protocols) on which libraries are built, serenely assuming that theirwork is “neutral” with respect to the way in which the libraries are subsequently used.
Every important design decision has an impact on what is subsequently possible, andusability cannot be ‘added on’ at the end. User needs have to be taken into accountfrom the earliest stages and the deepest levels of design.

• Of course, one of the challenges for all who are concerned with the users’ views indesign is communicating those views effectively to others involved in the process ofdesign and deployment of digital information resources. We need to learn to ‘market’
the need for usability, and to know the audience for ‘marketing’. Maybe this remains
as one of the biggest challenges for us, because usability is a privative – you don’t
notice when it all works well, you may not even notice if it doesn’t work well (after
all, fewer people will use it, and very few will complain or suggest that things might
have been done better). And making systems truly usable and rewarding to use isdifficult – probably as difficult as getting a novel piece of system infrastructure to
perform well, but difficult in a different, ‘softer’, kind of way.

To close on a more optimistic note: one attendee reported that a library he is involved insaw 1.5m copies of articles downloaded in the past year. This is a finding that will bereflected in many places, representing a real increase in the accessibility of information.
Digital libraries are providing a new kind of service that is increasing the use of pertinent
information. They are not impossible to use, but there is scope for improvement, andmany exciting challenges lie ahead in this regard. This workshop represents a step
forward in the slow march towards useful, usable and used digital resources.
The
papersfromtheworkshopareall
http://www.uclic.ucl.ac.uk/annb/DLUsability/JCDL02.html .

availablefrom

We would like to thank the programme committee:
Katy Börner
Indiana University, USA
Sally-Jo Cunningham
University of Waikato, NZ
Christine Borgman
UCLA, USA
Matt Jones
University of Waikato, NZ
Hanna Stelmaszewska
Middlesex University, UK
Tamara Sumner
University of Colorado, USA
Yin Leng Theng
Nanyang TU, Singapore
Harold Thimbleby
University College London, UK
We also wish to thank Richard Furuta, Lynnetta Sacharek and others on the JCDL
organising committee, Kathleen Casey (student volunteer) for her support on the day, andall speakers and workshop participants for making this such a stimulating and thoughtprovoking workshop.
Ann Blandford & George BuchananWorkshop chairs