CHIIR 2017 WORKSHOP REPORT

Report on the Second Workshop on
Supporting Complex Search Tasks
Marijn Koolen1
Nicholas Belkin4

Jaap Kamps2
Diane Kelly5

Toine Bogers3
Emine Yilmaz61

Huygens ING/KNAW, Netherlands
University of Amsterdam, Netherlands
3
Aalborg University Copenhagen, Denmark
4
Rutgers University, NJ, USA
5
University of Tennessee Knoxville, TN, USA
6
University College London, UK
2

Abstract
There is broad consensus in the field of IR that search is complex in many use cases andapplications, both on the Web and in domain-specific collections, and both in our professional and in our daily life. Yet our understanding of complex search tasks, in comparison
to simple look up tasks, is fragmented at best. The workshop addressed many open researchquestions: What are the obvious use cases and applications of complex search? What are
essential features of work tasks and search tasks to take into account? And how do theseevolve over time? With a multitude of information, varying from introductory to specialized,
and from authoritative to speculative or opinionated, when should which sources of information be shown? How does the information seeking process evolve and what are relevant
differences between different stages? With complex task and search process management,
blending searching, browsing, and recommendations, and supporting exploratory search tosensemaking and analytics, UI and UX design pose an overconstrained challenge. How do
we know that our approach is any good? Supporting complex search tasks requires new
collaborations across the whole field of IR, and the proposed workshop brought together adiverse group of researchers to work together on one of the greatest challenges of our field.
The workshop featured three main elements. First, two keynotes, one on the complexityof meaningful interactive IR evaluation by Mark Hall and one on the types of search complexity encountered in real-world applications by Jussi Karlgren. Second, a lively boasterand poster session in which seven contributed papers were presented. Third, three breakout
groups discussed concrete ideas on: (1) search context and tasks, (2) search process, and (3)evaluation of complex search tasks. There was an general feeling that the discussion madeprogress, and built new connections between related strands of research in IR.

ACM SIGIR Forum

58

Vol. 51 No. 1 June 2017

1

Introduction

One of the current challenges in information access is supporting complex search tasks. A user’s
understanding of their information need and their overall task develops as they interact with thesystem. Supporting the various stages of the task involves many aspects of the system, e.g. interface features, presentation of information, retrieving and ranking. Many search systems treat
the search process as a series of identical steps of submitting a query and consulting documents.
Yet information seeking research has shown that users go through different phases in their searchsessions, from exploring and identifying vague information needs, to focusing and refining theirneeds and search strategies, to finalizing their search. To be able to support exploring and discovering strategies we need to understand the characteristics of different tasks including open-ended,leisure-focused sessions. This is a highly complex problem that touches upon and bridges areas of
information seeking, interactive information retrieval, system-centered (ranking, evaluation), anduser interface design.
The background for this workshop is derived from the Interactive Track of the CLEF/INEX
Social Book Search Lab [4, 5, 6], which investigates scenarios with complex book search tasks anddevelops systems and interfaces that support the user through the different stages of their search
process.
The overall goal of the workshop was to create and foster an interdisciplinary forum where
researchers can exchange and contribute to the development of alternative experiments and prototypes. The main aim was to better understand how to support complex search tasks, addressing
many open research questions to be explored, including:Context What are the obvious use cases and applications of complex search? In what sense arethese “complex”? What generic characteristic do they share? How can search become an
integral part of its context, and the context integral part of search?
Tasks What are essential features of work tasks and search tasks to take into account? And
how do these evolve over time? How do can complex tasks be decomposed into manageable
sub-tasks, and partial results composed into comprehensive answers? How can we monitor
and support task progress?
Heterogeneous sources With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of
information? When to show more or other types of information than directly requested by
the searcher? Do we know when the user has gotten enough?
Search process How does the information seeking process evolve and what are relevant differences between different stages? What search tactics and search strategies are effective? How
can we promote the use of effective search strategies? How does the information need evolveand what are relevant success criteria for the end result and intermediate steps? How can
we cast these as effective complex queries, and how to (interactively) construct such queries?
UI and UX Does the need of complex task and search process management, blending searching,browsing, and recommendations, and supporting exploratory search to sense-making andanalytics, make UI and UX design an overconstrained challenge? What affordances are

ACM SIGIR Forum

59

Vol. 51 No. 1 June 2017

required and in what stage of the search process? How can we make the search processtransparent to the user? How and when does the initiative shift between system and user?
Evaluation How do we know that our approach is any good? Can we carve out one or a range
of generic aspects testable on a suitable benchmarks? Is there enough empirical evidence
to ground simulated interactive search? What kind of novel retrieval models are needed to
combine topical, contextual and preferential aspects?
SCST 2017 was a half day workshop on supporting complex search tasks—a work shop proper
where discussion was central, and all attendees were active participants. The workshop brought
together a varied group of researchers with experience covering both user and system centeredapproaches, to work together on the problem and potential solutions, and identify the barriers to
success and work on ways of addressing them.
The rest of this report will follow the program of the workshop. The workshop started with around of introductions where each attendee introduced him- or herself, and explained their own
interest in the topic. Next, it featured two short keynotes (discussed in §2) which helped frame theproblems and reach a shared understanding among all workshop attendees of the issues involved.
Mark Hall (Edge Hill University) talked about interactive IR evaluation and Jussi Karlgren (Gavagai, Sweden) talked about the hidden complexity of seemingly simple business information needs.
This was followed by a boaster and poster session in which nine papers (discussed in §3) were
presented. In the next session, participants divided over three breakout groups (discussed in §4),
with in-depth discussion on three topics of importance in the area: (1) search context and tasks,(2) search process, and (3) evaluation of complex search tasks. In the final session the results andprogress of the workshop was discussed and preliminary conclusions were drawn (discussed in §5).

2

KeynoteThe workshop started with two short keynotes, one from an academic perspective and one froman industry perspective, to set the stage and ensure all attendees were on the same page.

2.1

Where does it end? Complex Search Tasks and Evaluation

Mark Hall (Edge Hill University, UK) presented the academic keynote in which he explored theblurring boundary between complex search tasks and the larger work tasks that motivate thesearch. This has important implications for what aspects of the process we should evaluate andhow we do the evaluation in a meaningful and measurable way.
The task is one of the core concepts around which Information Retrieval (IR) and particularly
its evaluation are structured. The traditional search task is a set of instructions that motivate
a query and which enable the assessment of the query results’ relevance. With the introductionof the user into the evaluation process the task evolved into a set of instructions that the user isprovided with when they start the evaluation process. As these tasks become ever more complex
the boundary between the IR task and the larger work task that motivates the IR task is starting toblur. This transition poses a major question for evaluation, as it also blurs the question of whether
the evaluation is judging the quality of the IR system, the quality of the data for completing the

ACM SIGIR Forum60

Vol. 51 No. 1 June 2017

work task, or the user’s ability to identify with and complete the larger work task. In his keynote,Mark explored some of these issues.

2.2

Complex Aspects of Seemingly Simple Information Needs

Jussi Karlgren (Gavagai, Sweden) presented the industrial perspective and discussed how thetypical information needs of corporate customers are often posed in short and basic questions but
are surprisingly hard to formulate in meaningful queries, and answering them requires complex
processes of curating and aggregating diverse and disparate data.
In commercial practice customers have seemingly simple information needs. What are customers and potential customers saying about my brand online? Are our customers happy? What
do our customers want from us? How could we make them happier? These questions are easy
to pose and intuitively will provide a basis for commercial action if information to meet them isgiven by a system. But at least three non-trivial challenges lie in the way of fulfilling informationneeds of this kind: (1) formulating an information need in terms a system can accommodate;(2) curating, aggregating, and refining the relevant information sources of reasonable quality; (3)understanding client strategies and sometimes internal unstated organisational mechanisms on
the client side. These are all in some sense gaps between technology offerings and business needs
which need to be addressed for information access technology to progress from ad-hoc search.

3

Accepted Papers

The workshop invited short paper contributions to be presented as posters. We received 11
submissions and accepted 9 (for an 81% acceptance rate). Each paper was reviewed by at least
three reviewers. Paper contributions are presented as a 1-minute boaster talk and as a poster
during the interactive poster session.
Rutter et al. [12] discuss a case study of a type of complex task that at face value is simple andstraightforward, but turns out to be complex to resolve: how do you make a phone safe for a child?
There are a lot of opinions online, many possibilities for actions, many variations in hardware andsoftware, but ultimately no single clear and correct answer for everyday phone users.
Bogers et al. [2] report on the experiences and challenges in organizing the CHIC and SBS
Interactive Tracks from 2013 to 2016 in the form of a list of important properties. These properties
could inform the design of new IIR evaluation campaigns and related researcher communities inways that expand our understanding of information (seeking) behavior.
Koesten and Singh [9] focus on how a large governmental data portal in the UK supports users
in conducting complex search tasks involving data, identify problems with the used interface, anddiscuss potential research directions to improve interfaces for complex, data-related search tasks.
Hoeber et al. [7] examine the use of exploratory search strategies for purposive sampling fromlarge text collections. The use of exploratory search strategies that leverage visual analytics enables
them to consider the relevance of the data in addition to more traditional sampling methods.
Egusa et al. [3] investigate the use of concept maps—graphical representations that allow people
to represent their knowledge explicitly—to evaluate the effects of interactive complex search. Theirstudy showed a significant change in the concepts maps produced before and after executing acomplex search.

ACM SIGIR Forum

61

Vol. 51 No. 1 June 2017

Huurdeman [8] proposes a framework for the design of search user interfaces for complex
search tasks. His framework covers three different types of features—personalizable features,informational features, and input & control features—and discusses the different stages of complex
information seeking where these features are relevant.
Ventocilla et al. [13] suggest a bottom-up approach to displaying and exploring relations andcorrelations in datasets. Using billiards as a metaphor, a graph-representation of (cor)relations
in a dataset are unfolded in directions based on the user’s choices. This provides an intuitive
exploratory faceted search interface with quantitative analyses calculated at run-time.
Novin [11] argues that studies on complex search tasks should make their designs more contextbased, which will make them more applicable to real-world scenarios, as well as more reproducible
and falsifiable. The paper reviews literature on cognitive experiments that stress the importanceif situation on actions and proposes a outside-in approach where the context is defined first, then
the work task, after which different experimental variables can be considered.
Arora and Jones [1] conducted a user study to investigate how users perceive relevance andimportance of highlighted document fragments related to specific search topics, to better understand how to generate effective summaries of documents. The results provide insights on whattypes of information are effective for satisfying information needs and why users find some parts
more relevant than others.

4

Breakout Sessions

The second half of the workshop consisted of 3 breakout groups, seeded from the open researchquestions (see §1) and the contributed papers (see §3). The goal of the breakout groups was todiscuss concrete activities to make progress on this topic, for instance in the form of a workshop
or a new Interactive IR task at e.g. TREC or CLEF. The three themes were (1) search context
and tasks, (2) search process, and (3) evaluation of complex search tasks.

4.1

Search Context and Tasks

The first break-out session focused on tasks and context and had 17 participants. Many participants agreed that while several taxonomies of task facets have been proposed, there is no true
methodical approach for how to construct authentic tasks yet. One proposal was to use Bloom’s
taxonomy of learning outcomes could be used to prompt different kind of behaviors, although notevery task has learning focus. Another actively debated question was the role that context plays
in tasks: is it even possible to talk about complex tasks without involving context? The context
that people are in influences the complexity of that task (mobile vs. desktop, time pressure vs.
relaxed, etc.) One potential barrier to progress on research on tasks is the lack of a common setof instruments that as a minimum could be used in interactive IR experiments with a possibility
for extension. Unfortunately, this has not been done yet and it makes experiments difficult tocompare. A proposed solution to this was a repository of interactive IR data to allow researchers
to share and compare their data.
At the end of the session, several actionable outcomes were identified to further the state ofresearch on task-based search:

ACM SIGIR Forum

62

Vol. 51 No. 1 June 2017• A repository for interactive track data. A sensible first step for this could be a workshop
where people discuss how such datasets would need to be described and shared.
• More resource papers that describe existing datasets for IIR. These could be submitted to
the SIGIR resource track, SIGIR Forum or possibly to CHIIR.
• A participatory workshop for organizing and clustering tasks, starting with the database oftasks initiated by Luanne Freund and Barbara Wildemuth. This workshop could also try toprioritize the types of tasks the community should focus on.
• Domain-specific workshops for generating sets of authentic tasks for a specific domain byinviting real-world domain experts and IIR experts

4.2

Search Process

The third break-out session focused on the search process of complex search tasks and had 24
participants. The majority of the discussion in the breakout group focused on how to create ashared evaluation task that would be of interest to a wide range of IIR researchers.
The basic idea that the group came up with was to organize a shared task or workshop to
determine what methods, tools, techniques are commonly used to investigate a given problem.
That would give an idea of what any future, shared IIR evaluation task would have to take into
account to generate interest from a broad spectrum of IIR researchers (both user and systemfocused researchers).
The shared task or workshop would set a specific, complex task. It was left open how this task
should be specified or obtained. Each participant would then be free to use whatever methods theywanted to use to collect data and address the task. At the workshop each participant reports andshares what methods they used, what data they collected, and what results they got. In addition
to the shared dataset that the task and workshop would produce, the increased understandingof what methods are in use would also allow for the development of new tools and systems to
support IIR evaluations and IIR shared evaluation tasks.

4.3

Evaluation of Complex Search Tasks

The third break-out session focused on evaluation and had 15 participants. It focused on thechallenge of reducing dependence on standard datasets for shared tasks. Reliance on shareddatasets risks over-fitting and takes away research efforts for contexts where data cannot (easily)be shared but deserve attention. Datasets or streams are to an increasing degree heterogenous,
comprising mixed media, very various text genres, levels of abstraction and editorial oversightand structured and unstructured data. Reproducibility of research cannot rely only on identical
data, but on using comparable data. As a concrete activity the group suggested a so-called
extract-a-thon:• Before the workshop, participants review and describe the methodology section of publishedpapers, including how the data used are described and characterized, and assess if theinformation is sufficient for reproducing research.

ACM SIGIR Forum

63

Vol. 51 No. 1 June 2017• At the workshop, participants go through these identified characteristics, to assess if a reported experiment is based on data which is described carefully enough to be useful forthe respective research and application interests. An example of the challenge would be astatement in a paper claiming that topic modeling works badly for social media, where infact the actual problem is that the dataset is heterogeneous.
• At end of workshop, participants suggest concise ways of characterizing data that can beused across publications to fit the respective (and diverse) research interests.
• The outcome of the workshop is a check list and quality certification to share with reviewers
in the field and make assessment easier and more uniform. This is somewhat similar to what
is currently considered a hygiene factor for quality assurance of research in the medical field(e.g. the Cochrane Training1 ).
The group noted that it is important to include ways of evaluating retrieval with mixed media
collections (video, text). Text collections are easier to describe as there are more standard waysof doing this so, but the categories used may not be sufficiently fine-grained to reproduce anexperiment.

5

Conclusions

Finally, the breakout groups reported to the audience with short pitches for their concrete plans
for addressing these themes. The workshop provided a comprehensive overview of current workon supporting complex tasks in a variety of settings, and fostered new collaboration within ourfield on one of the most important topics in the coming years. Although there is a clear sense of
direction emerging, it is less easy to pinpoint concrete insights or lessons learned. There was ageneral feeling that we made progress, and that the open discussion with participants across thefields of user-centered and system-centered information retrieval and human computer interaction
was useful. There was great support for holding another edition of the workshop at a future
CHIIR.
Last, but certainly not least, the workshop lived up to its proud reputation of social events,
leading to new papers, spin-off workshops, and new friendships. This tradition was continued
with an informal program at Restaurant Schrøder in Oslo, attended by workshop participants
and other CHIIR attendees interested in the workshops topic, combining great discussion withfood and drinks.

6

Acknowledgments

We thank the CHIIR workshop chair Preben Hansen and the local organizers for their support.
We also thank the program committee members for their timely and constructive reviews. Last,
but not least, we thank all presenters and attendees for their lively and engaged discussions. The
workshop proceedings are available at http://ceur-ws.org/Vol-1798. .
1

See http://training.cochrane.org/

ACM SIGIR Forum

64

Vol. 51 No. 1 June 2017

References[1] P. Arora and G. Jones. How do Users Perceive Information: Analyzing user feedback
while annotating textual units within Web Documents. In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017), pages 7–10, 2017. URL http://ceur-ws.org/Vol-1798/.
[2] T. Bogers, M. Gäde, M. Hall, V. Petras, and M. Skov. Lessons Learned from the CHiC and
SBS Interactive Tracks: A Wishlist for Interactive IR Evaluation. In CHIIR 2017 Second
Workshop on Supporting Complex Search Tasks (SCST 2017), pages 11–14, 2017. URL
http://ceur-ws.org/Vol-1798/.
[3] Y. Egusa, M. Takaku, and H. Saito. Evaluating Complex Interactive Searches Using Concept
Maps. In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017),
pages 15–17, 2017. URL http://ceur-ws.org/Vol-1798/.
[4] M. Gäde, M. M. Hall, H. C. Huurdeman, J. Kamps, M. Koolen, M. Skov, E. Toms, andD. Walsh. Overview of the SBS 2015 Interactive Track. In L. Cappellato, N. Ferro, G. J. F.
Jones, and E. SanJuan, editors, Working Notes of CLEF 2015 - Conference and Labs of the
Evaluation forum, Toulouse, France, September 8-11, 2015., volume 1391 of CEUR Workshop
Proceedings. CEUR-WS.org, 2015. URL http://ceur-ws.org/Vol-1391/78-CR.pdf.
[5] M. Gäde, M. M. Hall, H. C. Huurdeman, J. Kamps, M. Koolen, M. Skov, T. Bogers, andD. Walsh. Overview of the SBS 2016 interactive track. In Working Notes of CLEF 2016 Conference and Labs of the Evaluation forum, Évora, Portugal, 5-8 September, 2016., volume1609 of CEUR Workshop Proceedings, pages 1024–1038. CEUR-WS.org, 2016. URL http://ceur-ws.org/Vol-1609/16091024.pdf.
[6] M. M. Hall, H. C. Huurdeman, M. Koolen, M. Skov, and D. Walsh. Overview of the INEX
2014 Interactive Social Book Search Track. In L. Cappellato, N. Ferro, M. Halvey, and
W. Kraaij, editors, Working Notes for CLEF 2014 Conference, Sheffield, UK, September
15-18, 2014., volume 1180 of CEUR Workshop Proceedings, pages 480–493. CEUR-WS.org,
2014. URL http://ceur-ws.org/Vol-1180/CLEF2014wn-Inex-HallEt2014.pdf.
[7] O. Hoeber, L. Hoeber, R. Snelgrove, and L. Wood. Interactively Producing Purposive Samples
for Qualitative Research using Exploratory Search. In CHIIR 2017 Second Workshop on
Supporting Complex Search Tasks (SCST 2017), pages 18–20, 2017. URL http://ceur-ws.
org/Vol-1798/.
[8] H. C. Huurdeman. Dynamic Compositions: Recombining Search User Interface Features for
Supporting Complex Work Tasks. In CHIIR 2017 Second Workshop on Supporting Complex
Search Tasks (SCST 2017), pages 21–24, 2017. URL http://ceur-ws.org/Vol-1798/.
[9] L. Koesten and J. Singh. Searching Data Portals - More Complex Than We Thought? In
CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017), pages
25–28, 2017. URL http://ceur-ws.org/Vol-1798/.

ACM SIGIR Forum

65

Vol. 51 No. 1 June 2017[10] M. Koolen, J. Kamps, T. Bogers, N. Belkin, D. Kelly, and E. Yilmaz, editors. Proceedings of
the Second Workshop on Supporting Complex Search Tasks, volume 1798, 2017. CEUR-WS.
URL http://ceur-ws.org/Vol-1798.
[11] A. Novin. Contextualizing the Cognition Crisis. In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017), pages 29–32, 2017. URL http://ceur-ws.org/
Vol-1798/.
[12] S. Rutter, V. Blinzler, C. Ye, M. B. Twidale, and M. L. Wilson. Complex Search Task: How
to Make a Phone Safe for a Child. In CHIIR 2017 Second Workshop on Supporting Complex
Search Tasks (SCST 2017), pages 33–35, 2017. URL http://ceur-ws.org/Vol-1798/.
[13] E. Ventocilla, J. Bae, M. Riveiro, and A. Said. A Billiard Metaphor for Exploring Complex
Graphs. In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017),
pages 36–40, 2017. URL http://ceur-ws.org/Vol-1798/.

ACM SIGIR Forum

66

Vol. 51 No. 1 June 2017