ECIR WORKSHOP REPORT

Report on the First Workshop on
Supporting Complex Search Tasks
Maria Gäde1Marijn Koolen3Mark M. Hall2
Mette Skove4

Hugo Huurdeman3
Elaine Toms5

Jaap Kamps3
David Walsh2

1

Humboldt Universtity Berlin, Germany
2
Edge Hill University, UK
3
University of Amsterdam, Netherlands
4
Aalborg University, Denmark
5
University of Sheffield, UK

Abstract
There is broad consensus in the field of IR that search is complex in many use cases andapplications, both on the Web and in domain specific collections, and both in our professional
and in our daily life. Yet our understanding of complex search tasks, in comparison to simple
look up tasks, is fragmented at best. The workshop addressed many open research questions:
What are the obvious use cases and applications of complex search? What are essential
features of work tasks and search tasks to take into account? And how do these evolve
over time? With a multitude of information, varying from introductory to specialized, andfrom authoritative to speculative or opinionated, when to show what sources of information?
How does the information seeking process evolve and what are relevant differences between
different stages? With complex task and search process management, blending searching,browsing, and recommendations, and supporting exploratory search to sensemaking andanalytics, UI and UX design pose an overconstrained challenge. How do we know that
our approach is any good? Supporting complex search tasks requires new collaborations
across the whole field of IR, and the proposed workshop brought together a diverse group of
researchers to work together on one of the greatest challenges of our field.
The workshop featured three main elements. First, a keynote on an emerging theory oftask difficulty by Diane Kelly. Second, a lively boaster and poster session in which sevencontributed papers were presented. Third, three breakout groups on: 1) user interfaces anduser experience, 2) tasks and users, and 3) information needs on controversial topics. There
was an general feeling that the discussion made progress, and built new connections between
related strands of research in IR.

1

Introduction

One of the current challenges in information access is supporting complex search tasks. A user’s
understanding of the information need and the overall task develop as they interact with the
ACM SIGIR Forum

50

Vol. 49 No. 1 June 2015

system. Supporting the various stages of the task involves many aspects of the system, e.g. interface features, presentation of information, retrieving and ranking. Many search systems treat
the search process as a series of identical steps of submitting a query and consulting documents.
Yet information seeking research has shown that users go through different phases in their searchsessions, from exploring and identifying vague information needs, to focusing and refining theirneeds and search strategies, to finalizing their search. To be able to support exploring and discovering strategies we need to understand the characteristics of different tasks including open-ended,leisure-focused sessions. This is a highly complex problem that touches upon and bridges areas of
information seeking, interactive information retrieval, system-centered (ranking, evaluation), anduser interface design.
The background for this workshop is derived from the CLEF/INEX Interactive Social Book
Search Track [6], which investigates scenarios with complex book search tasks and develops systems and interfaces that support the user through the different stages of their search process.
Book search provides an excellent scenario to investigate these issues. Information needs in booksearch are highly complex, combining aspects of topical relevance (genre, subject), user relevance(background knowledge, reading level, preferences and interests) and social relevance (recommendations and opinions of friends and other trusted sources). Moreover, book search needs develop
from vague notions of interest (books similar to X) to more specific criteria (likable characters,
academic treatment of topic, etc.) This change in the users needs, and the development of thetasks associated with those needs, demonstrates that current search systems provide little active
support for such scenarios. Examples from the ISBS collection, findings based on the user studies,and prototypes of information seeking stage sensitive search systems are available, and will be
used to focus the discussion in the breakout groups.
The overall goal of the workshop was to create and foster an interdisciplinary forum where
researchers can exchange and contribute to the development of alternative experiments and prototypes. The main aim was to better understand how to support complex search tasks, addressing
many open research questions to be explored, including:Context What are the obvious use cases and applications of complex search? In what sense arethese “complex”? What generic characteristic do they share? How can search become an
integral part of its context, and the context integral part of search?
Tasks What are essential features of work tasks and search tasks to take into account? And
how do these evolve over time? How do can complex tasks be decomposed into manageable
sub-tasks, and partial results composed into comprehensive answers? How can we monitor
and support task progress?
Heterogeneous sources With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of
information? When to show more or other types of information than directly requested by
the searcher? Do we know when the user has gotten enough?
Search process How does the information seeking process evolve and what are relevant differences between different stages? What search tactics and search strategies are effective? How
can we promote the use of effective search strategies? How does the information need evolve

ACM SIGIR Forum

51

Vol. 49 No. 1 June 2015

and what are relevant success criteria for the end result and intermediate steps? How can
we cast these as effective complex queries, and how to (interactively) construct such queries?
UI and UX Does the need of complex task and search process management, blending searching,browsing, and recommendations, and supporting exploratory search to sense-making andanalytics, make UI and UX design an overconstrained challenge? What affordances are
required and in what stage of the search process? How can we make the search processtransparent to the user? How and when does the initiative shift between system and user?
Evaluation How do we know that our approach is any good? Can we carve out one or a range
of generic aspects testable on a suitable benchmarks? Is there enough empirical evidence
to ground simulated interactive search? What kind of novel retrieval models are needed to
combine topical, contextual and preferential aspects?
SCST 2015 was a half day workshop on supporting complex search tasks—a work shop proper
where discussion was central, and all attendees were active participants. The workshop brought
together a varied group of researchers with experience covering both user and system centeredapproaches, to work together on the problem and potential solutions, and identify the barriers to
success and work on ways of addressing them.
The rest of this report will follow the program of the workshop. The workshop started witha round of introductions where each attendee introduced him- or herself, and explained theirown interest in the topic. Next, it featured a keynote (discussed in §2) which helped frame theproblems and reach a shared understanding of the issues involved among all workshop attendees.
Diane Kelly (University of North Carolina at Chapel Hill) talked about an emerging theory oftask difficulty. This was followed by a boaster and poster session in which seven papers (discussedin §3) were presented. The lively discussion extended over the coffee break. In the next session,
participants divided over three breakout groups (discussed in §4), with in-depth discussion on
three topics of importance in the area: 1) user interfaces and user experience, 2) tasks and users
and 3) information needs on controversial topics. In the final session the results and progress of
the workshop was discussed and preliminary conclusions were drawn (discussed in §5).

2

KeynoteThe workshop started with a keynote to set the stage and ensure all attendees were on the same
page.

2.1

Theory Of Search Task Difficulty

Diane Kelly (University of North Carolina at Chapel Hill) talked about “When Effort Exceeds
Expectations: A theory of search task difficulty.” She presented an overview of key literature ontask complexity and task difficulty, noting the lack of consensus on the meaning and use of theterms over different studies. Task difficulty is usually associated with the user’s perception of
the task (such as the searcher’s performance or the solvability of the task), and task complexity
usually associate with the task’s characteristics (such as the number of subtasks or subtopics orthe cognitive complexity).
ACM SIGIR Forum

52

Vol. 49 No. 1 June 2015

The cognitive complexity framework from educational theory was introduced—consisting ofsix stages: remember, understand, apply, analyze, evaluate, create—and applied to the searchtask design.
This framework and Campbell’s model of task complexity were used in a recent empirical
study on expected and experienced task difficulty. Despite exhibiting varying amounts of effort,
searchers rated most of these tasks the same with respect to difficulty, and it seems clear that
more effort does not necessarily correspond to more difficulty or less satisfaction.
This led to the proposition of a theory of search task difficulty, based on the (lack of) match
between the expected effort and the experienced effort: when the experienced effort conformthe expectations, the difficulty is moderate; when the experienced effort is larger than expected
the task is considered hard, and when the experienced effort is lower than expected the task is
considered easy.

3

Accepted Papers

All papers had an one minute boaster and were presented as posters in an interactive poster session.
We received 8 submission and accepted 7 papers (87.5% acceptance rate). All submissions werepeer reviewed by at least three reviewers. In this section we briefly describe each of the sevenaccepted papers.
Dori-Hacohen et al. [3] present searching on controversial topics as a complex search task.
Challenges include how one searches for such views on such topics and in what way a systemshould engage users, by e.g. showing all the different views or censoring certain views.
Gunadi et al. [5] highlight the problems of distributed IR, such as dealing with heterogeneous
document collections and information systems, which presents a challenge for systems to support
users requiring information from multiple resources.
Balog [1] introduces the notion of task completion engines and argues for re-thinking the searchexperience. The author presents use cases such as travel planning and shopping and sketches aninterface that integrates functionality to support users in the various steps of their complex tasks.
Walsh and Hall [9] discuss the scenario of users with no particular goal or information need,
who engage the system with the aim to explore collections in the domain of cultural heritage,
where the system helps them in the initial phase to discover areas or topics of interest.
Kumpulainen and Huurdeman [7] describe the relation between searcher’s confidence in theirsearch skill and understanding of the search topic and/or the system, focusing on those users
with limited knowledge but high confidence. For such users, the system could shake up the searchexperience to help them formulate their search goals better and change their search behaviour.
Toms [8] argues that search tasks in and of themselves are often not complex, but that thecomplexity arises from the related work task for which the searches are conducted. The focus of
the system should be on supporting search within the larger work task. The author also describes
different kinds of support that a system could or should give.
Dean-Hall et al. [2] present an evaluation framework for Point-Of-Interest recommender systems and scenarios where users want personalised suggestions that takes many aspects of theircontext into account, such as geographical location, time of day, season as well as their interestsand previous experiences.

ACM SIGIR Forum

53

Vol. 49 No. 1 June 2015

4

Breakout Sessions

The second half of the workshop consisted of 3 breakout groups, seeded from the open researchquestions (see §1) and the contributed papers (see §3). The three themes were 1) user interfaces
and user experience, 2) tasks and users and 3) information needs on controversial topics.

4.1

User Interfaces and User Experience

Mark Hall chaired a breakout group on user interfaces and user experience. The discussion initially
focused on defining what are “complex search tasks.” There was a distinction between “complex”
and “complicated” tasks, with the consensus in this group going toward a definition in terms oftasks with a non-trivial internal task structure, rather than in terms of inherent difficulty of therequest. There was also clear consensus that a standard search box is not sufficient, and that
it is crucial what you do after (and for) basic, atomic search. As complex tasks can pose very
different requirements, there is not a single solution and need for a “toolbox” of relatively genericcomponents that can composed into a tailored user interface for a specific task—as it were a multipurpose swiss army knife rather than a single purpose phillips screwdriver. There was continued
discussion on the exact shape and form of the resulting system, and the integration of such incomplex tasks.

4.2

Tasks and Users

Elaine Toms chaired a breakout group on tasks and users. The discussion was essentially a continuation of the keynote, focusing on task complexity, task difficulty and effort, and the distinction
between perceived and experienced complexity, difficulty, and effort. The general feeling was that
more clarity and consensus is needed on the core task and user variables that play a role, and weshouldn’t just take their meaning for granted. There was great support for attempts to clear up
basic dependencies between these variables, as in Diane’s proposed theory of search task difficulty
based on divergence or congruence between the expected effort and the experienced effort. Yetat the same time, variables must be contextualized, for example, effort may be a negative thing
in one context (as the user has to invest extra time and energy to obtain the same result, hencerisking failure) but a positive thing in another context (as a user that reads information about
multiple perspectives is more informed and can make better decisions).

4.3

Controversial Topics

Shiri Dori Hacohen chaired a breakout group on information needs on controversial topics. This
was a rather specialized group, dealing cognitively challenging topics for which information with
opposing facts, views, or opinions can be found. The discussion focused on how to help users deal
with such controversial topics. There are challenges in detecting controversy or diverging views, inmeasuring the degree of controversy, and in the presentation of information with opposing facts,
views, or opinions.

ACM SIGIR Forum

54

Vol. 49 No. 1 June 2015

5

Conclusions

Finally, the breakout groups reported to the audience and a panel of experts, with continued
discussion on what we learned, and concrete plans for the next year. The workshop provided acomprehensive overview of current work on supporting complex tasks in a variety of settings, andfostered new collaboration within our field on one of the most important topics in the coming years.
Although there is a clear sense of direction emerging, it is less easy to pinpoint concrete insights orlessons learned. There was a general feeling that we made progress, and that the open discussion
with participants across the fields of user-centered and system-centered information retrieval andhuman computer interaction was useful. There was great support for holding another edition of
the workshop at a future ECIR.
Last, but certainly not least, the workshop lived up to it’s proud reputation of social events,
leading to new papers, spin-off workshops, and new friendships. This tradition was continued withan informal program in the 1516 Brewing Company in Vienna, attended by workshop participants
and other ECIR attendees interested in the workshops topic, combining great discussion witha sheer endless supply of food and drinks. Intense discussion about complex search tasks and(scientific) life in general continued far into the Vienna night...

6

Acknowledgments

We thank the ECIR workshop chairs Guido Zuccon and András Benczur and the local organizers
for their support. We also thank the programme committee members for their timely and constructive reviews. Last, but not least, we thank Diane Kelly for her keynote address that help
frame the problems and work toward a shared understanding.
The proceedings are available at http://ceur-ws.org/Vol-1338. .

References[1] K. Balog. Task-completion Engines: A Vision with a Plan. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.
[2] A. Dean-Hall, C. Clarke, J. Kamps, and J. Kiseleva. Online evaluation of point-of-interest
recommendation systems. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.
[3] S. Dori-Hacohen, E. Yom-Tov, and J. Allan. Navigating Controversy as a Complex Search
Task. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.
[4] M. Gäde, M. Hall, H. Huurdeman, J. Kamps, M. Koolen, M. Skov, E. Toms, and D. Walsh,editors. Proceedings of the first international workshop on Supporting Complex Search Tasks,
volume 1338 of CEUR Workshop Proceedings, 2015. CEUR-WS.org. URL http://ceur-ws.
org/Vol-1338.
[5] E. Gunadi, T. Plumbaum, and S. Albayrak. Applied Distributed Information Retrieval inEnterprise Search. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.

ACM SIGIR Forum

55

Vol. 49 No. 1 June 2015

[6] M. M. Hall, H. C. Huurdeman, M. Koolen, M. Skov, and D. Walsh. Overview of the INEX 2014
interactive social book search track. In L. Cappellato, N. Ferro, M. Halvey, and W. Kraaij,
editors, Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15-18, 2014.,
volume 1180 of CEUR Workshop Proceedings, pages 480–493. CEUR-WS.org, 2014. URL
http://ceur-ws.org/Vol-1180/CLEF2014wn-Inex-HallEt2014.pdf.
[7] S. Kumpulainen and H. Huurdeman. Shaken, not steered the value of shaking up the search
process. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.
[8] E. Toms. Complex Tools for Complex Tasks. In Gäde et al. [4]. URL http://ceur-ws.org/
Vol-1338.
[9] D. Walsh and M. M. Hall. Just looking around: Supporting casual users initial encounterswith Digital Cultural Heritage. In Gäde et al. [4]. URL http://ceur-ws.org/Vol-1338.

ACM SIGIR Forum

56

Vol. 49 No. 1 June 2015