The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, JapanThe Inﬂuence of Expectation and System Performance on
User Satisfaction with Retrieval Systems
Katrin Lamm

University of Hildesheim
Marienburger Platz 22
31141 Hildesheim

katrin.lamm@unihildesheim.deThomas Mandl

Christa Womser-Hacker

mandl@unihildesheim.deWerner Greve

womser@unihildesheim.de

University of Hildesheim
Marienburger Platz 22
31141 Hildesheim

University of Hildesheim
Marienburger Platz 22
31141 Hildesheim

University of Hildesheim
Marienburger Platz 22
31141 Hildesheim

wgreve@unihildesheim.de

ABSTRACT

1.

Correlations between information retrieval system performance anduser satisfaction are an important research topic. The expectationof users is a factor in most models of customer satisfaction in marketing research; however, it has not been used in experiments withinformation retrieval systems so far. In an interdisciplinary effort
between information retrieval and psychology we developed an experimental design which uses the so-called conﬁrmation/disconﬁrmation paradigm (C/D-paradigm) as a theoretical framework. This
paradigm predicts that the satisfaction of users is strongly governedby their expectations towards a product or a system. We report a
study with 89 participants in which two levels of both system performance and user expectation were tested. The results show thatuser expectation has an effect on the satisfaction as predicted by theC/D-paradigm. In addition, we conﬁrmed previous studies whichhint that system performance correlates with user satisfaction. The
experiment also revealed that users signiﬁcantly relax their relevance criteria and compensate for low system performance.

Information retrieval systems provide their users with text documents for an information need that the user entered into the system.
Retrieval systems are mostly evaluated with the so-called batch
method. Independent jurors (assessors) judge whether documents
are relevant for a search topic or not. This method is an aspect of
the Cranﬁeld paradigm [6, 13]. It makes objective results possible because the juror does not know from which system a result
document comes. After jurors completed the relevance judgmentprocess, it can be calculated how well systems perform according
to measures like Mean Average Precision (MAP) or Binary Preference (BPref). Nevertheless, a problem of this approach is that therelevance assessment process cannot account for issues such as auser’s knowledge and experience, situational relevance or user saturation [10].
User based evaluations confront test users with the results ofsearch systems and let them solve information tasks given in theexperiment. In such a test setting, the performance of the users canbe measured by observing the number of relevant documents theyﬁnd. This measure can be compared to a gold standard of relevancefor the search topic to see if the perceived performance correlateswith an objective notion of relevance deﬁned by a juror. In addition, the user can be asked about his satisfaction with the searchsystem and its results.
In recent years, there has been a growing concern on whether
the results of batch and user experiments correlate. Do users havea better search experience, when systems improve in a batch comparison and contain more relevant documents in their result lists?
Are changes in MAP noticeable for users in tests or real life? Are
users more satisﬁed with better result lists and do better systems
enable them to ﬁnd more relevant documents? Some studies could
not conﬁrm this relation between system performance and user satisfaction, some did. Please note that, throughout this paper, wedeﬁne the performance of a system as its ability to retrieve relevant
documents according to a given query describing a users’ information need. Previous research is presented in the following section.
In psychology, research on customer satisfaction has developed
multi-dimensional models of satisfaction. These models take theexpectation of the user into consideration. The C/D-paradigm predicts that the satisfaction judgment depends on both the product orsystem quality and the expectations of the users. Low expectations

Categories and Subject Descriptors
H.3.3 [Information Storage and Retrieval]: Information Searchand Retrieval

General Terms
Measurement, Human Factors

Keywords
Information retrieval systems, user studies, user satisfaction, user
expectation, conﬁrmation/disconﬁrmation paradigm― 60 ―INTRODUCTION

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

lead to satisﬁed users even for low system quality because the expectations of the user or customer are met. On the other hand, users
with high expectations can only be satisﬁed if the system performs
well. Otherwise their expectations are not conﬁrmed and negative
disconﬁrmation is likely [15]. An example may be a car enthusiast and a person who has no passion for cars. If these two typesof customers are going to buy a new car, then the same car may
meet the buyer’s expectations in the one case, but fail in the othercase. This simple example shows that the same stimulus can evoke
completely different satisfaction responses depending on the underlying expectation. An interdisciplinary approach has great potential
for user-based retrieval evaluation. The exchange of ideas between
psychology and information retrieval has led to the study design
described in this paper.
The study presented in this paper introduces expectation as one
factor in a user based experiment. In addition to manipulate thesystem results to guarantee ﬁxed performance measures, the test
scenario manipulates the users to assume a certain expectation. The
relation between expectations before using the system, the performance and the satisfaction after using the system are explored in
the result section.

2.

STATE OF THE ART

User-based retrieval evaluation focuses on the potential users of
information retrieval systems. Järvelin and Ingwersen put the reason why the involvement of the user is so important for experimental retrieval evaluation in a nutshell: "The real issue in information
retrieval systems design is not whether its recall-precision performance goes up by a statistically signiﬁcant percentage. Rather, it
is whether it helps the actor solve the search task more effectively
or efﬁciently." [12] Recent studies can be classiﬁed into two types.
On the one hand there are the satisfaction-oriented studies which
concentrate on the relationship between system performance anduser satisfaction. On the other hand there are the user performanceoriented studies which primarily analyze the inﬂuence of the system performance on the user performance.

2.1

Research on User Satisfaction

Compared to studies regarding the performance of users withretrieval systems, so far the satisfaction of users has been less frequently investigated in the context of interactive information retrieval. As can be seen from the following literature, due to thecomplexity of the evaluation task, a generalized method for assessing user satisfaction has not been established yet.
An investigation reported by Tagliacozzo [22] shows the difﬁculty of measuring user satisfaction. Therein Tagliacozzo analyzes
the responses to a questionnaire sent to users of the MEDLINE bibliographic service. From the observation of inconsistencies in theusers’ judgment of the helpfulness and the usefulness of the MEDLINE search system, Tagliacozzo draws the conclusion that it isessential to tap several aspects of satisfaction in order to provide amore complete and accurate picture of the users satisfaction withthe evaluated system [22].
Among other things, the issue of user satisfaction has been considered by Su [21], who proposed a user-oriented evaluation modelfor the information retrieval contexts. Besides various performancemeasures, subdivided into relevance, efﬁciency and connectivity
measures, this model also includes several subjective measures ofeffectiveness, subdivided into utility and user satisfaction measures.
Earlier research by Su [20] showed that the value or usefulness ofsearch results as a whole seems to be a good predictor of information retrieval performance.
In 2006, Al-Maskari et al. [1] conducted a study in association

with a submission to iCLEF2006. iCLEF is the interactive trackof the Cross Language Evaluation Forum (CLEF1 ). In 2006, image retrieval was selected as the central theme of this track2 . Pictures from the photo sharing community FLICKR3 were used asdata collection. Several effectiveness measures were used to evaluate the system performance (P@50norm; P@100norm; Q-measure;BPref-10; 10-Precision4 ). In addition, user performance was measured via task-speciﬁc modiﬁcations of recall and precision. Furthermore, the authors also asked participants for their satisfaction
with the usefulness, accuracy and coverage of the search results.
Participants rated their satisfaction on a 3-point scale (1 = very satisﬁed; 0.5 = partially satisﬁed; 0 = not satisﬁed). No direct relationship between system performance and user performance respectively satisfaction could be conﬁrmed within the scope of thisuser study [1].
In another study, Al-Maskari et al. [2] conducted a similar experiment on the satisfaction of users. This time, participants directly
searched the internet via the Google5 search engine. The purposeof this study was to determine whether a correlation between theeffectiveness of Google results quantiﬁed by Precision and Cumulative Gain measures (such as Cumulative Gain (CG), Discounted
Cumulative Gain (DCG) and Normalized Discounted Cumulative
Gain (NDCG)) and user satisfaction could be established. The latter was again assessed using a 3-point scale. Instead of the satisfaction with the usefulness this time the users’ satisfaction withthe ranking of results was requested. This user study could demonstrate a signiﬁcant relationship between system performance anduser satisfaction [2].
Huffman and Hochster [9] pursued a slightly different approach.
The experiment was also based upon the Google search engine.
The test users were given real queries already submitted to Google
as information needs. Within this user satisfaction study, one groupof test users rated the results in terms of their relevancy and anothergroup in terms of their satisfaction. Subsequently, the latter group
was asked to submit the queries again and then act as if they really had this information need. In order to investigate the relationship between system performance and user satisfaction Huffman
and Hochster contrasted the relevance judgments for the top three
search results of the ﬁrst query of each session with the user’s ﬁnal satisfaction. Thus, Huffman and Hochster introduced a simple
Cumulative Discounted Gain Measure which correlated with user
satisfaction [9].
Also in 2007 Jansen et al. [11] studied the effect of brandingon evaluation of system performance. In this experiment the same
result documents were integrated in the search engine results pagesof Google, MSN Live Search6 , Yahoo7 and one unknown, in-housesearch engine. Participants completed four different search topics
with supposedly four different search engines. In order to evaluate the users’ perception of the system performance participants
were asked to evaluate the result documents. The ﬁndings of thisstudy show that branding has an effect on user’s evaluation of system performance [11], which further implies that user expectations
inﬂuence user satisfaction.
1

http://www.clef-campaign.org
http://nlp.uned.es/iCLEF/2006/guidelines.
htm
3
http://www.flickr.com/
4
For a brief description of each measure see [1]5
http://www.google.co.uk
6
Since Bing the new search engine by Microsoft started, the domain http://www.live.com forwards to the homepage from
Bing http://www.bing.com/
7
http://m.www.yahoo.com/
2― 61 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

This short overview demonstrates that current research on user
satisfaction is quite heterogeneous according to experimental setups and results. To obtain valid results, real-world search situations have to be simulated as closely as possible. For comparison
between the results of different studies a standard measure for user
satisfaction would be necessary. Furthermore, it can be noticed thatuser expectation as a variable has been almost neglected so far inuser satisfaction research.

2.2

Research on User Performance

Evaluations regarding the performance of the user have become
more common over the last years.
In the years 2000 and 2001 Turpin and Hersh [24] carried out
two user studies on the question whether batch and user evaluations lead to comparable results. The ﬁrst study was performed
within the framework of the TREC-8 interactive track and consisted in an instance recall task. The second study was carried out
within the TREC-9 interactive track and consisted of a questionanswering task. For both experiments, the authors used two searchsystems with different MAP performance (TREC-8: 0.27 vs. 0.32
MAP; TREC-9: 0.27 vs. 0.35 MAP). An inﬂuence of the system
performance on the user performance could neither be observed forthe instance recall nor for the question-answering task. Merely forone measure, a relation could be observed [24].
The following two studies investigated the inﬂuence of different
levels of system quality on the user performance by using artiﬁcially constructed result lists. That way, the system performancecan be better controlled. Whereas Allan et al. [4] adopted BPref tomeasure the system performance, Turpin and Scholer [23] adopted
MAP. The experiment of Allan et al. involved eight ﬁxed systemlevels ranging from 0.5 to 0.98 BPref and the experiment of Turpin
and Scholer involved ﬁve levels between 0.55 and 0.95 MAP. In addition, these two studies differ with respect to the underlying searchtasks. Allan et al. chose a passage retrieval task in which the test
subjects had to ﬁnd, highlight and label all facets of the answer
to a given information need. According to the BPref level theyalso differentiated between hard and easy topics. Turpin and Scholer found it especially important to use simple search tasks. Their
precision-oriented task consisted in ﬁnding a document relevant toan information need (time was measured). The recall-oriented task
consisted in ﬁnding as many relevant documents as possible withina given time limit of ﬁve minutes. Whereas the results of Allan etal. show statistically signiﬁcant effects on the user performance at
certain levels of BPref [4], the results of Turpin and Scholer only
show a weak effect for the recall-oriented task [23].
Another interesting approach is reported by Al-Maskari et al. in2008 [3]. They investigate the relationship between system performance and user performance as well as the system’s inﬂuence on
the user’s satisfaction and perception of topic easiness. By using
an application that facilitates the access to three different searchengines it was possible to select the worst and the best system forevery single topic. Thus system performance (measured in Average Precision (AP)) was speciﬁed on a per-topic basis. During
the test, participants used both the superior and the inferior system without being aware of it. The user task was recall-oriented
and consisted in identifying as many relevant documents as possible within a seven minute period of time. Interestingly, unlike inour present study, Al-Maskari et al. discovered signiﬁcant correlations for a recall-oriented user performance measure. Besides thefact that participants of the superior system were able to ﬁnd more
relevant documents, results also show that they took less time andfelt more satisﬁed [3].
Smith and Kantor also provide some interesting results on thisissue [19]. Using a 3 × 3 factorial design they examined whether
users alter their behavior in response to the performance of a searchsystem. In the context of this study, subjects were told that theywould be searching with the Google search engine. Similar tothe study described just above, the result pages were selected on
the ﬂy. The test system would submit the original user queries toGoogle and than return result lists starting from different ranking
positions depending on the actual system performance condition.
Three different conditions were tested. The ﬁrst condition was thestandard system, which displayed result documents in the same order returned by Google. The remaining two conditions related tothe two experimental systems, one condition with consistently-lowrankings (CLR) and one with inconsistently-low-rankings (ILR).
In the CLR-condition the displayed lists constantly started at the300th ranking position in the original Google result list. In the ILRcondition the starting positions within the original Google ranking
were varied across one session for the ﬁrst twenty documents of thedisplayed lists. The results obtained by Smith and Kantor strongly
support our ﬁndings regarding the relaxation of the user-deﬁned
relevance criteria. As in the study presented in this paper, users of
the bad system (the CLR-system) seemed to relax their relevancecriteria and in the consequence thereof also accept documents ofminor relevance [19].
Scholer and Turpin [17] analyzed the concept of an individual’s
relevance threshold in relation to the system performance. The aimof this study was to further understand the mismatch in the results
between batch and user evaluations. The diversity of the user’s
relevance criteria constituted the starting point for establishing a
study design that opens up the possibility to investigate their relevance judgments more precisely. Therefore, similar to their previous study from 2006, Scholer and Turpin artiﬁcially constructedresult lists to simulate different search systems based on a fourlevel relevance scale of the documents. Findings suggest that different users adopt different relevance criteria. As a consequence,variances in system performance which can be observed in batch
experiments are not necessarily visible in user experiments due toinhomogeneous relevance judgments of different users[17].
Most recently, Dostert and Kelly [8] conducted a study to investigate how and when users decide to end their search. The interactive information retrieval experiment was based on the TREC
2005 Robust Track collection. All subjects used the same test system and were asked to ﬁnd relevant documents for four different
search tasks. Besides recording the participants’ actual recall, they
were asked to estimate their achieved recall value and time spentsearching. Results revealed that subjects recall estimates were notaccurate, but positively correlated with actual recall. Furthermore,
the time spent searching was also positively correlated with actual
recall. A post-test interview showed that the main reason for terminating a search seems to be their assessment of enough information[8].
In summary, we may conclude that research results especiallyregarding the satisfaction of the user are not consistent and further
study is needed to address the subject in a more detailed manner.

3.

STUDY DESIGNThe objective of the present study was to develop a controlled
study design which makes it possible to investigate the predictionsof the C/D-paradigm in the information retrieval context. The research questions addressed were:― 62 ―• What inﬂuence does the system performance have on user
satisfaction and performance?

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan• What inﬂuence does the user expectation have on user satisfaction and performance?
To answer these questions our study adopts a between subjects
design, in which each subject participates only in one treatment.
That means that each participant is confronted with only one expectation manipulation. We assume that such a setup is more realistic
and less prone to fail in achieving the objective.
In order to investigate the impact of system performance and expectations on user performance and satisfaction both of these independent variables had to be manipulated.
To control the system performance we used artiﬁcially constructedresult lists for three different search topics, as it has been done inprevious studies. For each of these topics two different result lists
were created - one to simulate a low and one to simulate a highquality information retrieval system (subsequently also referred toas the inferior and the superior system or low and high systemlevel). More details on the underlying system performance are presented in the subsection on the test system.
In order to test whether the users’satisfaction depends on their
expectation, two different introduction sheets were designed. Therein
the system was introduced as an expensive professional search system being about to be implemented in the university library in one
case and as a student project with unknown quality in the othercase. Test users read this description before the test and we expected that their expectation toward the system was affected. The
introduction sheet also contained further information on the user
test.
Since a between subjects design was adopted we randomly assigned the users to four treatment groups. In order to reach significant statements, we decided to test at least 20 subjects per group.
Table 1 shows the two-by-two factorial design of our experiment.
Our participants were not aware of the different experimental conditions.

expectation

low
high

Topics and Documents• C187: Transports of nuclear material in Germany• C190: Child labor in Asia

3.3

Test System

Design and functions of the application system resemble presently
popular internet search engines in order to facilitate intuitive interaction. This was necessary to assure that interaction problems
would not become a confounding variable. This was also necessary because no training session was given to the participants inorder to shorten the duration of the test. A screenshot of the userinterface can be seen in ﬁgure 1.

Figure 1: Screenshot of test application.

For the experiment, a standard IR evaluation collection was usedin order to obtain a collection with topics and relevance assessments. We decided to use the German language corpus from the
Cross Language Evaluation Forum (CLEF) which includes documents from the Swiss news agency Schweizerische Depeschenagentur (SDA), the German daily newspaper Frankfurter Rundschau(FR) and the German weekly news magazine Der Spiegel [14]. The
collection was used for a monolingual ad hoc retrieval setting.
From the large pool of topics available at CLEF, three were selected. Several criteria had to be met during the selection process.
Finally, there needed to be a sufﬁcient number of relevant documents available. Secondly, the topic should be about an issue whichis still relevant and interesting for the test persons. This might notalways be the case because the CLEF topics are developed for a testcollection from the years 1993 and 1994.
The selected topics were the following ones:• C86: Renewable energy

Subjects

Test users were recruited from our university. We selected only
students who did not study any information technology related ﬁeld.
In order to avoid a gender effect, only female students were selected. Thus the results obtained from the study can only be generalized to the population of female searchers, but considering agender-mixed sample would have either demanded a larger sample
size or we had to live with the fact that a gender bias might affectthe results. Users were motivated to participate in the study by an
incentive. All users took part in a lottery for three cash prizes witha value of 100 Euro in total. Altogether, 89 participants completed
the test which makes this experiment larger than most studies reported in the literature.

system performancelowhighgroup1group2group3group4

Table 1: Two factor study design.

3.1

3.2

We adopted the algorithm presented by Turpin and Scholer forthe construction of the result lists [23]. The algorithm starts witha random list and randomly swaps pairs of documents until the desired Average Precision is reached. The top positions are not treatedspeciﬁcally. It can be observed that result lists with the same Average Precision may look quite different. To avoid any issues wemade sure that the top of the list mirrored the system level. In theﬁrst ﬁve documents, one or three irrelevant documents were placed
for the high respectively low system level. Average Precision is deﬁned as follows: "The mean of the precision scores obtained after
each relevant document is retrieved, using zero as the precision for
relevant documents that are not retrieved." [7]Both Average Precision and Precision scores were selected within
the range that has been used in previous studies (e.g. [23]).
For the superior system performance an Average Precision of
0.75 and for the inferior system performance an Average Precisionof 0.55 was used.
Furthermore we varied the ratio of relevant to irrelevant documents (i.e. precision) for the high respectively low system level.
For the superior system performance a precision of 0.6 and for theinferior system performance a precision of 0.5 was chosen. These
values are within the range of previous experiments and represent
the potential of modern search systems. For future experiments,
these values should be empirically derived. Table 2 shows the ra-― 63 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan%
documents
available
inferior
systemsuperior
systemrelevant
irrelevant
total
relevant
irrelevant
total
relevant
irrelevant
total

50%
50%
60%
40%

C86
60
50
110
50
50
100
60
40
100

C187
57
48
105
48
48
96
57
39
96

C190
50
42
92
42
42
84
50
34
84

Table 2: Ratio of documents per topic.

tio of relevant and irrelevant documents for the three topics in our
experiment.
In table 3 we provide the rankings of the six result lists usedduring the experiment. Each 0 represents an irrelevant document
and each 1 an relevant document.
Each results page consisted of ten result documents. Results included the title of the document, snippet (ﬁrst sentence of document) and document source. Our application system did not support keyword highlighting.

3.4

Experiment

The users came to a lab to conduct the test. After a welcome,they were given the written scenario for the test. The scenario wasformulated to direct the test users toward the desired recall-oriented
search process. The scenario was also used to create high and low
expectations. Therefore, as already described in the context of thestudy design, one test group was told they would be using a professional search system, whereas the other group was told they would
be using a search system, developed within a student project.
The search was motivated by the following scenario. All subjects were asked to imagine they were journalists and would use
the presented search system to look for articles that concerned the
topic of their next article. This scenario also ﬁts the documents
available within the CLEF collection. As previous work has shownsuch task-based scenarios help to minimize the artiﬁciality of thetest situation and at the same time motivate the participants [5, 18].
In the search tasks participants were required to ﬁnd relevant
documents according to the task-based scenario. To avoid confounding order effects the information needs were given to the users
in different orders. Participants were given a maximum of ten minutes to complete each task, although they could terminate the search
early if they felt they had completed the task. As in the case of thetask-based scenario the opportunity to end the search early should
help minimize the artiﬁciality of the test situation and promote
more natural behavior. Nevertheless, due to the study design, thetypical real-world iterative search behavior was disabled. Queries
were predetermined and users could not reformulate them in orderto provide each subject with the same list of results. The explanation given to the participants to justify why the queries had to bepredetermined was that we wanted to assure comparable conditions
across participants.
The users could perform a limited number of actions in the userinterface. They could browse through a list of result documents
and evaluate them based on a representation by title and a snippet
as in most commercial systems today. This approach was used toprovide users with a somewhat familiar system design and at thesame time maintain a high degree of control over the test situation
for better comparison between users. Users could then select documents by clicking. The full text of the documents was shown ina separate window and users could then read them and decide if
they were relevant or not. We want to point out that the subjects
did not have to read through all documents in the result lists but
only those that seemed relevant to them based on title and snippet.
Once selected by a user, the user had to decide whether or not the
document was relevant. The judgments as well as timestamps for
all interactions with the application system were recorded in a log
ﬁle. Oral comments of the users were not systematically recordedby the test conductor.
Finally, after ﬁnishing the search tasks for all three topics, eachsubject completed a questionnaire regarding the overall satisfaction, that is to say, there was not a single questionnaire after eachsearch task. The questionnaire contained 28 question items anddealt with both ease of use of the application system and satisfaction with the presented result lists. The questionnaire also included
a section on personal data, which inquired about age, occupation,
years of computer and internet use. In terms of content the satisfaction part of the questionnaire includes questions regarding satisfaction with ease of use, efﬁciency, output display, precision, rankingof results, result quality and reuse probability. Therefore, the questionnaire covers similar content areas to the studies described in
section 2.1. Most questions were rated on a 7-point scale whichranged from 1 (completely correct) to 7 (not at all correct). We also
tried to indirectly determine the satisfaction of the test users withthe search results. Therefore we asked them at the end of the test if
they were willing to participate in another user test. The underlying
assumption was that it is more likely that satisﬁed subjects would
agree to assist in a second test.

4.

RESULTS AND DISCUSSION

The results of the experiments were analyzed according to thevalidity of the test design and subsequently the (subjective) user
perception and the (objective) user performance were analyzed.

4.1

Test Design

Prior to user based analysis, several validation checks were performed to ensure that there were no confounding variables that had
an additional effect on the collected data. We brieﬂy mention two
effects that are connected with the search tasks.
Similar to the results of Turpin and Scholer [23] the one-way
ANOVA analysis of our data indicates that there is a statisticallysigniﬁcant topic effect. The recorded user performance for the three
search tasks was included as repeated measurement factor. Our observations suggest that comparatively topic C86 was more difﬁcult
and C187 more easy to deal with. We assume that the existence ofsuch a topic effect can be looked upon favorably because different
information needs in realistic search situations also vary in theirlevel of difﬁculty.
As already mentioned the information needs were given to thetest users in different orders. In this manner we intended to control
possible order effects such as learning and fatigue. The relationshipbetween user performance and topic order were examined using a
one-way ANOVA, with the order as independent factor. Only forthe topics C86 and C190 signiﬁcant order effects can be observed.
This supports the above observation that topic C187 was in generaleasier to deal with and therefore no training was necessary for thetest subjects. This check also shows that the variation of the topic
order was appropriate.
In most cases, subjects did not terminate their search early (onaverage 10 participants (SD = 1.7) searched less than 10 minutes
per task). In contrast to the results obtained by Dostert and Kelly[8], in our study spending more time searching did not lead to better
recall. However, early search termination did correlate with the― 64 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan

topic

system
level
low

C86
high
low
C187
high
low
C190
high

ranking
110001010011100011011111000011100111011010110110101011000100
0011101001001100100001101110001001110100
110111100111110110111111111111011110110101000001110100111101
1 1 1 1 0 0 1 1 0 1 0 1 1 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 1 1 1 10 0 0 0 0 0 1 0011000111011110101110010001001110010010001110111110011001110
011100111000100000010101101000110010
011111110111011111101101110001111101011100011110111011111111
1 1 10 0 1 0 0 0 1 1 0 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0110001011101001101011101100001100010000100110110111011100110
110011101000001100111000
111101110111111110101011111010101010010101001011100010101101
111001101100010001100011
Table 3: Ranking of result lists.

time taken to ﬁnd the ﬁrst relevant document for two of the three
topics (C86/C187 p = 0.0). Thus it seems that some participants
searched slightly faster than others. As this behavior reﬂects reallife searching and those participants are equally distributed acrossthe four treatment groups, we assume that it adds to the potential
generalization of our ﬁndings.
The inﬂuence of expectations and system performance on user
satisfaction and user performance was for the most part examinedusing a two-way ANOVA. No signiﬁcant inﬂuence could be detected for the expectation of the users. One possible interpretation
could be that expectations do not have an inﬂuence on the perception of retrieval results. Far more likely however is the interpretation that the manipulation was not sufﬁcient. This assumption was
conﬁrmed by some subjects within informal conversations at theend of the test. These observations suggest that further research isneeded to clarify the validity of the conﬁrmation/disconﬁrmation
paradigm in the ﬁeld of information retrieval.
With respect to the system performance on the contrary significant inﬂuences could be identiﬁed for both user satisfaction anduser performance. These group differences are described in the following sections.

way ANOVA. Corresponding to the underlying study design, thesystem performance and the user expectation form the independent
variables. The individual responses were included as dependent
variable. Signiﬁcant group differences in relation to the systemlevel only occurred for the following precision-oriented items:• Item 1: The ﬁltering of articles could have been better. (p =0.008)• Item 2: Most articles have been relevant with respect of thequeries. (p = 0.025)
Figure 2 shows the results for these two questions. It must be
pointed out that the ﬁrst question is formulated negatively. For
better comparison, we inverted the scale for item 2, so that now
for both questions higher values correspond to higher satisfaction.
In ﬁgure 2 it can be seen that in both cases subjects that used thesuperior system were also more satisﬁed with the presented performance of the system. This shows that users are indeed able toperceive improvements in system performance. Nevertheless, eventhough almost no signiﬁcant group differences have occurred inthis ﬁrst analysis, in the next step we tried to combine several items
to one scale. Therefore we only used those items that dealt withthe user’s satisfaction with the presented result lists. An analysis ofreliability has been adopted to test the quality of this scale and tochoose the most appropriate items to be included. We applied Cronbach’s Alpha as a measure of internal test reliability to identify avalid group of items from our questionnaire. The best Cronbach’s
Alpha of 0.69 (which is close to the as sufﬁcient regarded value of0.7) is achieved by combining the following items:• Item 1: The ﬁltering of articles could have been better.
• Item 2: Most articles have been relevant with respect of thequeries.
• Item 3: I am satisﬁed with the quality of the search results.
• Item 4: The presentation of the results was clearly structured.
• Item 5: The order of the search results reﬂected the relevanceof the articles.

Figure 2: Satisfaction precision-oriented question items.

• Item 6: The articles I selected were helpful for the search.

4.2

User Satisfaction

At ﬁrst, we separately analyzed the question items using a two-

The two-way ANOVA of the combined scale reinforces the results from the individual analysis, as can be seen in ﬁgure 3.

― 65 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japan• t1.Doc : Following Turpin and Scholer we compared the timethat users took to ﬁnd their ﬁrst relevant document [23].

The predictions of the C/D-paradigm can be observed in our experimental data. Nevertheless, we did not ﬁnd statistically signiﬁcant differences for the user expectations (p = 0.50). However, as
already shown for question items 1 and 2 there is a signiﬁcant difference (p = 0.01) between the two system levels. There occurred
no interaction effects (p = 0.78).
Figure 3 plots the mean values for the four test groups based on
the combined satisfaction scale speciﬁed above. Of the two groups
that used the inferior system the group with the treatment - high expectation - is less satisﬁed (4.15 vs. 4.32) because its expectations
were not fulﬁlled (negative disconﬁrmation). Furthermore, of thetwo groups that used the superior system the group with the treatment - low expectation - is more satisﬁed (4.65 vs. 4.72) because
its expectations were exceeded (positive disconﬁrmation).

• User Precision (UP): According to Al-Maskari et al. the UP
was calculated as number of documents correctly identiﬁed
as relevant divided by total number of documents judged as
relevant by the user [1].
• Pre-Click-Precision (PCP): Number of documents correctly
identiﬁed as relevant divided by total number of documents
selected by the user to read the full text. Thus PCP catches
the surface impression that the users had from the result lists.
This performance measure was derived from the idea of Resnick and Lergier to ask participants about their "[. . . ] expectation of how well his/her selection would match his/her
expectations." [16]. In this context they introduced the term
Pre-click conﬁdence.
In the focus of the analysis of the user performance was the mean
performance of the users over all three tasks and not per individualtask. Thus we are able to make more general and more widely
applicable statements.
Again a two-way ANOVA was used to determine whether signiﬁcant differences existed between the four test groups. The ﬁveuser performance measures were one after another included as dependent variables. Table 4 summarizes the signiﬁcance values regarding the main and interaction effects of the independent factors
with one degree of freedom.
expectation
measureF1
p2
Doc@10 2.868 0.094
UR
2.562 0.113
t1.Doc0.041 0.84
UP
0.486 0.488
PCP
0.939 0.335
1F-value2
Signiﬁcance value

Figure 3: Conﬁrmation/disconﬁrmation paradigm.
The indirect satisfaction test with the additional user test did notlead to conclusive results. Altogether 76 of the 89 subjects offeredto participate in another test. Such a high percentage (85%) suggests that this result does not represent the indirect satisfaction of
the users. Instead, it seems that they felt obliged to offer their willingness to attend a further test.
The fact that correlations between system performance and user
satisfaction are rarely observed also reﬂects the fact that experimental variables for such studies are hard to evaluate. As Al-Maskari
et al. [3] already pointed out the satisfaction of the user "[. . . ] isalways likely to exhibit discrepancies amongst individuals."

4.3

User Performance

In addition to the satisfaction as expressed by the test users, we
also intended to record and analyze their performance. Five user
performance measures were used to investigate the experimental
data:• Doc@10: Number of relevant documents that users wereable to identify within the given time of ten minutes per task.
In the experiment of Turpin and Scholer subjects had a ﬁveminute time limit per topic [23].
• User Recall (UR): Number of documents correctly identiﬁed
as relevant divided by total number of relevant documentsin the result list. Al-Maskari et al. pursued a similar strategy within the framework of their user study. But since thisstudy dealt with image retrieval the focus was on identifying
unique documents respectively images [1].

systemperformanceFp
0.465 0.497
0.519 0.473
0.101 0.751
13.045 0.001
4.424 0.038

interaction
F
2.731
2.316
0.289
4.823
1.013

p
0.102
0.132
0.592
0.031
0.317

Table 4: Summary ANOVA user performance.
These results show that the system performance has a signiﬁcant
impact solely on the precision-oriented measures, UP and PCP. For
the remaining measures none of the test conditions has a signiﬁcant
inﬂuence (p > 0.05). The fact that there are no signiﬁcant main
effects for the recall-oriented measures, Doc@10 and UR, indicatesthat users are able to compensate the difference between the twosystem levels for these performance measures.
The signiﬁcant average value difference for PCP between thehigh and low system performance group can be explained as a reinforcement effect. Although the results for the Doc@10 are notsigniﬁcant (cf. Table 4) users of the inferior system by trend foundfewer relevant documents (8.75) than users of the superior system(9.55). In addition a weak, but also not signiﬁcant trend could
be observed that users with low system level selected more documents to read the full text (13.87) than users with high systemlevel (13.49). In themselves the differences are not signiﬁcant but
for the calculation of PCP these two quantities are divided by one
another. Thus the non signiﬁcant differences are reinforced by theuser performance measure.
The most interesting result concerning the user performance isthe signiﬁcant difference in the User Precision for the two systemlevels. Figure 4 illustrates this difference. In the case of using the― 66 ―

The Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japansuperior system subjects on average achieved a higher Precision
value (0.93) than in the case of using the inferior system (0.86).
The percentage difference is 8%. At ﬁrst it might be surprising to

Figure 5: Assimilation effect.

Figure 4: User precision.
see that the test users were not able to compensate the differencein system performance for the User Precision as well. Figure 5
shows why the detected differences occurred. The y-axis is sortedby the number of documents. As the ﬁgure shows, subjects who
were using the inferior system erroneously judged more documents
as relevant than users of the superior system did. In addition, thesame subjects judged fewer documents erroneously as irrelevant
compared to users of the superior system. Erroneously refers to
the relevance judgment provided by the CLEF jurors which wasconsidered as ground truth for our experiment. At this point wealso want to remark that non-relevant documents in our result lists
correspond to explicitly CLEF-judged non-relevant documents.
These ﬁndings suggest that users assimilate their relevance criteria to some extent to the quality of the information retrieval system.
For result lists with a wide range of relevant documents, users seemto be stricter in their relevance judgment than for result lists withfew relevant documents. This type of adaptive search behavior has
also been observed in previous user studies with different experimental setups [19, 17].
The difference for the User Precision is 8% and cannot be considered as large. Nevertheless, the percentage differences for thenumber of incorrectly judged documents are larger. For the documents incorrectly judged as relevant a difference of 57% (inferior
system: 1.68 vs. superior system: 0.73) was perceived. For the
documents incorrectly judged as irrelevant, 28% (inferior system:
1.64 vs. superior system: 2.29) were measured. These numbers
can be compared to interrater agreement experiments carried out
with TREC data. When more than one juror judged the documentsfor one topic, they agreed only on 30% to 40% of the relevant documents [25]. To draw the conclusion, the perception of relevanceseems to be inﬂuenced by the context, in the case of our experiment, the system performance level.

4.4

Analysis of Covariance

In order to account for different levels of experience in using information retrieval systems we ran an analysis of covariance withthe user’s search experience as covariate. For this purpose the questionnaire contained items that recorded how many hours per week
the test users spent on average with computer work, online etc.

These items were used to classify users as superior versus average experienced users. The search engine experience reported by
the test users was normally distributed. However, the search experience neither had a signiﬁcant impact on the user satisfaction nor
the user performance.
Our experiment shows that user reported relevance as it is usedin many interactive IR studies is subject to many contextual factors
which may not be controlled in each experiment. We show that thesystem performance is one context factor which has an inﬂuence.
As a consequence, user studies should not solely rely on user reported relevance but introduce some sort of objective assessment.
The same is true for click-through data.

4.5

Conclusions

The primary aim of our experiment was to investigate the inﬂuence that the users’ expectations about the performance of a searchsystem have on their satisfaction and performance with retrieval
systems. The experimental setup is motivated by the conﬁrmation/disconﬁrmation paradigm, a widely used model to describe
customer satisfaction in marketing research. Besides studying theinﬂuence of user expectations, we also investigated the inﬂuencethat the system performance has on both dependent variables. Although this study only constitutes a ﬁrst attempt to apply the concept of expectation conﬁrmation to interactive retrieval evaluation,
we already found some indication that the users’expectation of system performance may inﬂuence their satisfaction as predicted by
the conﬁrmation/disconﬁrmation paradigm. Nevertheless, the differences were small and not signiﬁcant according to the statisticalanalysis undertaken. The same is true for relation between the subjects’ expectation and performance, the expectation manipulation
revealed no signiﬁcant differences in the user performance measures between the four experimental groups. With respect to thesystem performance, however, we found that self-reported relevance in user studies is highly context dependent. This can be seenby the fact that users signiﬁcantly seem to relax their relevance criteria as soon as they begin to use the lower quality search engine.
We also found that the subjects’ statements of satisfaction correlated positively with the actual system performance level.

5.

― 67 ―OUTLOOKIn future studies, we intend to further elaborate the concept ofThe Third International Workshop on Evaluating Information Access (EVIA), June 15, 2010, Tokyo, Japancustomer expectation in the context of information retrieval. The
expectation can be manipulated stronger to see if there is a larger
and signiﬁcant effect on the satisfaction and performance of theuser. Another strategy to overcome the weaknesses attributed tothe operationalisation of the expectation manipulation may be tolet each participant compare two treatments.
It is also interesting and important for future research to allow
participants to use individual search strategies and in this context
further investigate the observed assimilation effect. For user studies, between subjects test design should be preferred or at least
combined with within subjects setups. Another important aspect
for future experiments is the degree to which users relax their relevance criteria. It should be measured in respect to the system performance and other factors.
The results of our study show that self reported relevance in user
studies is highly context dependent. The relevance criteria are affected by the system performance level. As a consequence, pastexperiments without a ground truth for relevance need to be interpreted carefully.

6.

REFERENCES[1] A. Al-Maskari, P. Clough, and M. Sanderson. Users’
effectiveness and satisfaction for image retrieval. In Proc.
LWA ’06, pages 84–88, 2006.
http://web1.bib.uni-hildesheim.de/edocs/2007/
522070116/doc/522070116.pdf.

[2] A. Al-Maskari, M. Sanderson, and P. Clough. The
relationship between IR effectiveness measures and user
satisfaction. In Proc. SIGIR ’07, pages 773–774. ACM Press,
2007.
[3] A. Al-Maskari, M. Sanderson, P. Clough, and E. Airio. The
good and the bad system: does the test collection predict
users’ effectiveness? In Proc. SIGIR ’08, pages 59–66. ACM
Press, 2008.
[4] J. Allan, B. Carterette, and J. Lewis. When will information
retrieval be "good enough"? In Proc. SIGIR ’05, pages
433–440. ACM Press, 2005.
[5] P. Borlund. Experimental components for the evaluation ofinteractive information retrieval systems. Journal ofDocumentation, 56(1):71–90, 2000.
[6] C. Buckley and E. Voorhees. Retrieval system evaluation. In
TREC: Experiment and Evaluation in Information Retrieval,
pages 53–75. Cambridge & London: MIT Press, 2005.
[7] C. Buckley and E. M. Voorhees. Evaluating evaluation
measure stability. In Proc. SIGIR ’00, pages 33–40. ACM
Press, 2000.
[8] M. Dostert and D. Kelly. Users’ stopping behaviors andestimates of recall. In Proc. SIGIR ’09, pages 820–821.
ACM Press, 2009.
[9] S. B. Huffman and M. Hochster. How well does result
relevance predict session satisfaction? In Proc. SIGIR ’07,

pages 567–574. ACM Press, 2007.
[10] P. Ingwersen and K. Järvelin. The turn: integration of
information seeking and retrieval in context. Springer,
Dordrecht, 2005.
[11] B. J. Jansen, M. Zhang, and Y. Zhang. The effect of brandawareness on the evaluation of search engine results. In Proc.
CHI ’07, pages 2471–2476. ACM Press, 2007.
[12] K. Järvelin and P. Ingwersen. Information seeking researchneeds extension toward tasks and technology. Information
Research, 10(1), 2004.
http://informationr.net/ir/10-1/paper212.html.
[13] T. Mandl. Recent developments in the evaluation of
information retrieval systems: Moving towards diversity andpractical relevance. Informatica, 32:27Ű–38, 2008.
[14] G. M. Nunzio, N. Ferro, T. Mandl, and C. Peters. Clef 2007:
Ad hoc track overview. In Proc. CLEF ’07, pages 13–32.
Springer, 2008.
[15] P. G. Patterson. Expectations and product performance as
determinants of satisfaction for a high-involvement purchase.
Psychology and Marketing, 10(5):449–465, 1993.
[16] M. L. Resnick and R. Lergier. Task speciﬁc user strategies in
on-line search. Journal of E-Business, 3(1):1–22, 2003.
[17] F. Scholer and A. Turpin. Relevance thresholds in systemevaluations. In Proc. SIGIR ’08, pages 693–694. ACM Press,
2008.
[18] F. Scholer, A. H. Turpin, and M. Wu. Measuring user
relevance criteria. In Proc. EVIA ’08, pages 47–56, 2008.
http://research.nii.ac.jp/ntcir/workshop/
OnlineProceedings7/pdf/EVIA2008/
09-EVIA2008-ScholerF.pdf.

[19] C. L. Smith and P. B. Kantor. User adaptation: Good resultsfrom poor systems. In Proc. SIGIR ’08, pages 147–154.
ACM Press, 2008.
[20] L. T. Su. Value of search results as a whole as the best single
measure of information retrieval performance. Information
Processing & Management, 34(5):557–579, 1998.
[21] L. T. Su. A comprehensive and systematic model of user
evaluation of web search engines: I. theory and background.
JASIST, 54(13):1175–1192, 2003.
[22] R. Tagliacozzo. Estimating the satisfaction of information
users. Bull. Med. Libr. Assoc., 65(2):243–249, 1977.
[23] A. Turpin and F. Scholer. User performance versus precision
measures for simple search tasks. In Proc. SIGIR ’06, pages
11–18. ACM Press, 2006.
[24] A. H. Turpin and W. Hersh. Why batch and user evaluations
do not give the same results. In Proc. SIGIR ’01, pages
225–231. ACM Press, 2001.
[25] E. Voorhees. Variations in relevance judgments and themeasurement of retrieval effectiveness. Information
Processing and Management, 36(5):697–716, 2000.

― 68 ―