15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Information Search Processes inComplex Tasks
Pertti Vakkari
University of Tampere
Finland

CHIIR'18, New Brunswick, 12.03.2018

Outline• Definitions of basic concepts• Task complexity, knowing as changes inknowledge structures, search process• Information search as a part of complex tasks• Search formulation, source selection, working
with sources and task outcome• Usefulness of information (Information gain) as acriterion for evaluation• The effect of search process on task outcome• Conclusions
21

15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Definitions• A larger task is an activity to be performed to accomplish agoal, that generates information searching (e.g. work tasks,hobbies) (Vakkari 2003)• Information search as a sub-task of a larger task
• A search task is a set of activities aiming at finding useful(contributing) information for a larger task

• Task complexity• Pre-determinability of task outcome, process and information
requirements (Byström & Järvelin 1995)• Complex tasks are vague and ill-structured typically generating
multiple search session• In this talk focus on complex, information intensive tasks withmultiple search sessions
3CHIIR'18, New Brunswick, 12.03.2018

Knowledge structures in human mind and texts• Knowledge represented as knowledge structures• Knowledge structures consists of concepts and theirrelations representing some phenomenon• “Searching”, “learning”• “Searching to learn” or “Learning to search”• Structures can be called as mental models or
schemas• Learning means changing or reinforcing existingknowledge structures42

15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Changes in knowledge structures
VagueGeneralSearching

Learning to search

Clear
Specific

Learning methods
• M1
• M2
• M3

Learning

Searching to learn

Carrying out procedures= Search skills• Explicating info need
• Selecting keywords• Formulating query• Assessing search results

5

CHIIR'18, New Brunswick, 12.03.2018

Changes in knowledge structures(Piaget 1976, Rumelhart & Norman 1981, Zhang & Soergel 2014)• Assimilation = addition of information into existingknowledge structures• Accretion = gradual addition of factual information intoexisting knowledge structures, i.e. no conceptual change• Accommodation = changes in existing knowledge
structures• Restructuring = changing and replacing concepts and theirrelations in the knowledge structure• Tuning does not include replacing concepts or theirrelations in the structure, but merely tuning the scope or
meaning of concepts
6315/03/18

CHIIR'18, New Brunswick, 12.03.2018

Information search process
Järvelin et al. 2015

Searchformulation!

SelectingsourcesWorking
withsourcesTaskoutcome7

CHIIR'18, New Brunswick, 12.03.2018

Knowledge structures and searching• My point of departure is the restructuring of knowledge
structures in a task process• A person is not familiar with the topic of task
• She is in an ASK (Belkin 1980), or in the beginning of ISP model(Kuhlthau 1993)• In the course of task performance process the revised
knowledge structures stabilize and learning changes fromrestructuring to tuning and assimilation• The process resembles Kuhlthau’s ISP model• Searching is a thinking and learning process• Searching changes from exploratory to lookup search over several
sessions (Marchionini 2006)84

15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Stabilization of knowledge structures
Task performance process

Restructuring

Tuning- Conceptual- The scope ofconcepts

changes

Topic selection

Focus
formulation

Collection

Assimilation- Instantiating

conceptualstructure

Presentation

Exploration
9CHIIR'18, New Brunswick, 12.03.2018

Articulating information need (Vakkari 2010)• In the beginning of task users seek to articulate informationneed, to make sense of task topic• Lack of sufficient concepts and their relations for representing thetopic• How to structure the topic-> search of conceptual structures forreplacing an insufficient structure (cf. Klein et al. 2006, Qu & Furnas
2008; Russell et al. 1993, Vakkari 1999)• Categorize and relate concepts• There are no ready-made structures available, but persons have tocreate them incrementally – working with sources• No single right solution, but many options available how to proceed
and to specify the structure (cf. Campbell 1988)• Initial queries are based on a vague conceptual structure• Poorly defined search goals
105

15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Query formulation as conceptualexploration• Users explore the conceptual space of retrieved documents
in relation to their own conceptual structure of the topic (cf.
Qu & Furnas 2008)• Recognizing potential extended conceptual structures• Identifying additional search terms for querying• Limiting the extension of topic by adding concepts (structuralchange) in a query• Specifying existing concepts (tuning) in the query• Vary tactics (replacing concepts) (Bates 1979; Xie & Joo2012)• A,B -> A,C -> A,D• Increase in the # and specificity of concepts (Vakkari 2010)
11CHIIR'18, New Brunswick, 12.03.2018

Intrinsically diverse search (IDS)• Vary tactics and exploration as means to articulating andstructuring task topic resembles IDS• IDS (Raman et al. 2013)• User requires information about multiple aspects of a topic• Longer and more complex, exploratory search sessions• Query suggestions representing new aspects of the topic, which theuser is likely to issue later in the session ≈ vary tactics• Query suggestions specifications of the original query• Query suggestions likely help searchers to explore theconceptual (aspect) space and to structure it• Help in categorizing a topic, not relating categories• Do ID Query Suggestions help to identify more aspects?
12615/03/18

CHIIR'18, New Brunswick, 12.03.2018

Search formulation by the changes in knowledge structure(Vakkari 2001, Vakkari 2002, Vakkari 2016)

Query formul. The modification of knowledge structures
Terms

Terms fromresults

Restructuring

TuningAssimilation•

Few general terms

à

•

•

A few synonyms

à••

Much new terms fromresults
Terms with associativerelations

àLess new terms fromresults•

Increase in # andspecificity of termsSynonyms

Specifying terms

Query length

A few terms

àMore terms

Queryreformulation••àLess reformulation

ID querysuggestions

Useful, new aspects

à

Not useful

Query similarityLow

àHighSession timeLong sessions

à

Short sessions

Vary tactics
Much reformulation

13CHIIR'18, New Brunswick, 12.03.2018

Source selection• In restructuring person’s conceptual structure is vague• Difficulties in recognizing useful sources (with potential newstructures) (Spink et al. 1998, Tang & Solomon 1998, Vakkari &Hakala 2000, Gwizdka 2014)• Vague selection (relevance) criteria• Many (potentially useful) documents selected•

In tuning the structure is stabilized• Easy to recognize useful sources matching with the stabilized
conceptual structure• Stricter selection criteria -> fewer documents selected147

15/03/18

CHIIR'18, New Brunswick, 12.03.2018

Source selection: SERP behaviour
Source selection

The modification of knowledge structure

SERP Behaviour

Restructuring

TuningAssimilation

Usefulness criteria

Vagueà

Clear

Orientation

RecallàPrecisionLevel of usefulness

Possible usefulà

Useful# Snippets viewedLarge

à

Small

Time on a snippet1)

Longà

Short

Click trough rate

Highà

Low

SERP browsing

Deepà

Shallow1) cf. Gwizdka 2014, Smucker & Jethani 2013
15

CHIIR'18, New Brunswick, 12.03.2018

Source selection: document behaviour
Source selection

The modification of knowledge structure

Document behaviour

Restructuring

TuningAssimilation# Documents viewed

Large

à

Small

Dwell time/document• Decision time
• Usage time

Varies by utility
Long
Varies by utilityààà

Varies by utility
Short
Varies by utility# Documents selectedLarge

à

Small

Selected/viewed

Large

à

Small

Used/selected

SmallàLarge

Information typeneeded•••••

BackgroundConc. & Theories
Procedural

ProceduralSpecific••

Specific
Factual16815/03/18

CHIIR'18, New Brunswick, 12.03.2018

The usefulness of documents• Task represented as a collection of terms T from thetext a user is working with (Budzig & Hammond 2000)
before and after information in a document is used
• Tt2 – Tt1 = Tnew = [(∑#td)/docwords] + e = Usefulness• td1,..,n = a new term in text identified in the document• # = the frequency of new term in the text• e = new terms not identified in the document• Influenced by the document > include all new terms = Tnew
• Similarity with the new terms identified in the document• Information about the new terms for task profile to rerank results or for query reformulation (diversification)
17

CHIIR'18, New Brunswick, 12.03.2018

The effect of information search on task outcome• Kuhlthau (1993), Hersh (1996), Wildemuth (1995) pioneers• Findings highly varied due to differences in tasks•

Precision or recall has no impact (Hersh 2003), has a negative effect(Vakkari & Huuskonen 2012) or positive effect (Wildemuth et al. 1995)
on task outcome• Effort in search process improves task outcome (Vakkari &Huuskonen 2012) or has no effect (Bron et al. 2012, Liu et al. 2012)• The more effort in result inspection instead of querying, and inworking with documents instead of result inspection, thebetter to the process of construction and task outcome (Liu &Belkin 2012, Butcher et al. 2011, Vakkari & Huuskonen 2012)18915/03/18

CHIIR'18, New Brunswick, 12.03.2018

Conclusions• In task performance information search processes patterned
due to changes in actors’ knowledge structures• In complex tasks, in addition to querying and result
examination, search process consists of working with sources
for creating a task outcome• The latter stages influence significantly the former ones
• Studying also information use in documents would deepen ourknowledge about querying and result examination, not to speak about
the whole search process -> e.g. personalization• In addition to accessing information, actors need help in making
sense of search results for progressing in the task (Kuhlthau 1993)• Tools for both accessing and making sense of information(Budzig & Hammond 2000, Ahn et al. 2008, Villa et al. 2009)19

CHIIR'18, New Brunswick, 12.03.2018

Empirical evidence of hypotheses proposed• Empirical ground insufficient• A few longitudinal studies (e.g. Kuhlthau, Wildemuth, Wang, Vakkari,
Liu & Belkin)• Supported by findings of some studies in pedagogics (e.g. Butcher etal. 2011, Cho et al. 2017) or on the role of topic knowledge insearching (e.g. White et al. 2009, Zhang et al. 2015)• Framed with ideas and findings mostly in cognitive psychology (e.g.
Gavin 1998, Robertson 2001) or organization science (e.g. Campbell1988)• More studies to test the validity of these hypotheses• Search patterns in successive sessions•

Personalization between and within sessions

201015/03/18

CHIIR'18, New Brunswick, 12.03.2018

Usefulness as information gain
• Complex search tasks require also other indictors ofsuccess than perceived usefulness of documents• Search tasks based on cognitive complexity (Kelly et al.
2015)• The contribution of information in documents for
progressing in the (search) task

• Information gain = (∑#td)/docwords + e• An indicator of information derived from a document into
the evolving text

• An objective measure, that can be generated from log
data without interfering with user activities
21

CHIIR'18, New Brunswick, 12.03.2018

Thank you!

Answers to Questions
2211