ABSTRACT

KEYWORDS

INTRODUCTION
1
Search evaluation is one of the central concerns in information retrieval (IR) studies. Most existing evaluation metrics ( e.g., RBP [8], ERR [1], etc. ) are based on result lists of a single query. However, session-level evaluation has also received more and more attention in recent years. Most of the session-level evaluation metrics (e.g. Session-based DCG (sDCG) [4] and Expected Utility (EU) [10]) are designed in a manner similar to query-level evaluation metrics. They are built on the cascade hypothesis [2] which assumes the user views search results from top to bottom and the user's attention will gradually de- cay during the browsing process. Although this assumption makes sense for single query evaluation, whether it is valid at session-level has not been verified. The metrics that have a decaying weighting function emphasize the primacy effect that the first documents ex- amined by users will be more influential for the overall session-level satisfaction. On the other hand, the recency effect , that users tend to begin recall with the end of search session, suggests the documents at the end of the session are more important. In other words, while the assumption behind a decaying weighting function is that the primacy effect is the major effect, an increasing weighting function favors the hypothesis that the recency effect is more important. In order to investigate whether the primacy effect or the recency effect is more important for session-level evaluation, we conducted a laboratory user study to collect a dataset that contains search logs for 675 sessions and the corresponding satisfaction feedback from users. Using this dataset, we try to answer the following research questions: ● RQ1 Does the primacy effect or recency effect has a stronger influence on user's session-level satisfaction? ● RQ2 What is the contribution difference of query satisfaction at different positions to session satisfaction? The remainder of this paper is organized as follows. Section 2 reviews some related work. Section 3 describes the user study and experimental settings. In Section 4, we present data analysis to address RQ1 and RQ2. We found the flaws of existing session eval- uation metrics and propose a number of criteria for designing better session search evaluation metrics. Finally, we give our conclusions and future work in Section 5.
2
RELATED WORK
Some recent studies concentrated on session-level evaluation meth- ods. sDCG [4] is a extended version of the Discounted Cumulative Gain (DCG) [3], it assumes the documents at lower position and retrieved by later query are less likely to be read by users, and there- fore, have weaker influence on session-level satisfaction. EU [10] take the novelty of results into account, the gain of a result will be discounted if the same content has been encountered in previous results. There are many existing studies focusing on the estimation of user satisfaction. Wang et al. [9] proposed a model in which user's action-level satisfaction was considered as a latent factor that af- fects the session-level satisfaction. Jiang et al. [5] compared user's feedback in two experimental settings, which include an in situ one and a context-independent one. We also use two similar ex- perimental settings for comparison in our study, the experimental details will be introduced in Section 3. Mao et al. [7] found that users' usefulness feedback reflects users' satisfaction better than relevance. They compared a series of evaluation metrics based on user's click sequence, but they did not investigate whether these metrics were suitable for session evaluation. In this study, we mainly focus on investigating how fine-grained perceptions contribute to session-level satisfaction so that we can know how to design a session evaluation metric.
DATA COLLECTION
3
To investigate the relationship between users' session satisfaction and query satisfaction, we conducted a laboratory user study which consists of two parts (see Figure 1) : I. Main User Study and II. Comparison User Study. We collect the three kinds of feedback from participants in these two parts: (1) Document-level usefulness feedback. (2) Query-level satisfaction feedback. (3) Session-level satisfaction feedback.
3.1
Main user study
In our main user study, we recruited 50 participants aged from 18 to 27. 24 participants were female, and other 26 participants were male. All the participants were familiar with basic usage of web search engines. Each participant needed to complete 9 tasks. These search tasks were designed based on the following criteria. Firstly, the task should be easily interpreted by all participants. Secondly, the task should not be a trivial one, since we mainly focus on search sessions with multiple interactions. An experimental search engine system is developed for the user study. When users submit queries to this system, it crawls corre- sponding results from a major commercial search engine. In the crawled SERPs, all query suggestions, ads, and sponsor search re- sults are removed to reduce the potential impacts on users' behavior. When performing tasks, participants can freely formulate queries. We made sure that each participant understood the experimental process through a pre-experiment training task. After the training stage, each participant was asked to perform 9 tasks in a random order. As shown in Figure 1-(I), the experimental procedure con- tains: (I-1) In the first stage, the participant should read and memorize the task description in an initial page, and she is asked to repeat the task description without viewing it to ensure she has remembered it. (I-2) Next, the participant can submit a query and click on the results to collect information as they usually do with commercial search engines. (I-3) After finishing the current query, she is asked to mark whether each document was useful for her at an evaluation page (0: not at all, 1: somewhat, 2: fairly, 3: very useful). She is also asked to give a 5-level graded satisfaction feedback on this query. If she wants to find more information, she can go back to step (I-2) and submit a new query. She can end the search whenever she thinks enough information has been found, or she can find no more useful information. (I-4) Finally, she is required to give a search answer and an overall 5-level graded satisfaction feedback of the whole search session of the task.
3.2
Comparison user study
The usefulness judgement and query satisfaction feedback stage in User-Study-I may affect participants' search behavior and subse- quent session satisfaction feedback, therefore, we designed another comparison user study. As shown in Figure 1-(II), only the second and third step was changed. The participants were asked to give usefulness feedback for all documents at once when they finish a task instead of after each query. We recruited another 25 participants aged from 18 to 26 to take part in this experiment. 13 participants were female, and other 12 participants were male. This part of the data will only be used for comparative analysis in Section 4.4.
DATA ANALYSIS
4
In this section, we first examine the relationship between users' query-level satisfaction and the corresponding usefulness judg- ments of the clicked documents. Then we analyze whether the primacy effect or recency effect has a stronger influence on user's session-level satisfaction. Furthermore, we investigate how users' query-level satisfaction contribute to the corresponding session- level satisfaction.
Data distribution
4.1
As mentioned in Section 3, we collected 1,548 users' 5-level graded query satisfaction feedback and 3,276 4-level graded usefulness feedback. We show the distributions of the number of queries in a session, usefulness feedback, query-level and session-level satisfac- tion feedback in Figure 2. From figure 2(a), we can see that sessions with three queries have the largest frequency, and sessions with no more than five queries accounts for 85.1%. An almost evenly distribution of usefulness judgments and query satisfaction can be seen in figure 2(b) and figure 2(c). As for session satisfaction shown in figure 2(d), only 6.6% sessions have a satisfaction score no more than 2. The results
indicate that most users have found some satisfying information within several queries.
Query-level satisfaction
4.2
To investigate the relationship between users' query-level satis- faction and the usefulness judgments. We calculated five different metrics ( cCG, cDCG, cERR, cMin, cMax ) based on users' useful- ness judgments of click sequence. As shown in Equation(1)-(4), CS = (d 1 , d 2 , ...,d 1 CS 1 ) is the click sequence in which each element d r is a clicked document, u r is the usefulness judgment of d r and we use (2 x r  – 1) to represent the gain of it.
query satisfaction for all queries and queries with at least one click are shown in Table 1. The results show that all metrics have signifi- cant correlations with query satisfaction. cMax has the strongest correlation ( r = 0.831) with query satisfaction, which suggests that the maximum usefulness of clicked documents can best reflect user's query-level satisfaction.
Table 3: Correlation of different metrics with session-level satisfaction. (The two columns represent metrics based on query satisfaction or cMax of query respectively).

Comparison of different weighting
4.3
function
Since existing session evaluation metrics often include a weighting function, we investigate whether the primacy effect or recency effect has a stronger influence on user's session-level satisfaction. As shown in Table 2, we used five types of weighting functions to weight user's query-level satisfaction. Each metric can be calcu- lated with Equation(5) in which s i represents the user's satisfaction feedback on i -th query. To let the metrics comparable across ses- sions with different length, we use a normalized query weight w , in this Equation.
The w * can be calculated with Equation(6). Here, N is the query number, wr is the r-th query's original weight which has been shown in Table 2.
Table 3 shows the Pearson's correlation coefficient between these five metrics and session satisfaction. We can see that the metric with decreasing weights has the lowest correlation coefficient with session satisfaction while the metric with increasing weight has the strongest correlation. We also use the cMax of all queries instead of query satisfaction feedback from users to calculate the five metrics and show their correlation with session satisfaction in Table 3. Similarly, user's session-level satisfaction has a higher correlation with the metric that has an increasing weight. These results suggests that it is the recency effect but not the primacy effect that has a stronger influence on user's session-level satisfaction.
Fitting weight analysis
4.4
As shown in section 4.2, metrics with different weighting functions have different performance in estimating session satisfaction. There- fore, the contribution of query satisfaction to session satisfaction may vary at different positions.
Table 5: Fitting weight at different query positions (Compar- To analyze the weight difference between queries, we perform a linear regressions analysis. We standardize query satisfaction and session satisfaction with a z-score transformation respectively and fit different models for sessions with different length. Only the sessions with no more than five queries are considered because there are only a few sessions with more than five queries. As shown in Table 4, we can see that the weight of the last query is higher than those of other query positions in each group of sessions. This proves again that the recency effect has a stronger influence on user's session-level satisfaction than the primacy effect. In our main user study, the users are asked to give their use- fulness judgments and satisfaction feedback while finishing each query. So the satisfaction feedback of the last query is collected just before the session satisfaction. To investigate whether the experi- mental settings have impacts on the results, we conduct a parallel analysis on the data collected in the comparison user study (Section 3.2). The results of the comparison analysis are shown in Table 5. We find that the weight of the last query is still higher than oth- ers. The observed recency effect is not caused by the adjacency of satisfaction feedback of the last query and the whole session. These results imply that a user is more concerned about the qual- ity of recent queries. If a user experiences a series of unsatisfying queries and finally finds a good one, she may also feel satisfied with her search session.
Discussion
4.5
In this section, we analyze the user study data to address the two RQs. A comparative experiment is designed to ensure our findings are not due to the experimental settings. From our results, we can see that traditional evaluation metrics may not be suitable for session satisfaction evaluation based on browsing sequence because they ignore the recency effect in users' perception of session-level satisfaction. Furthermore, the experi- ment results suggest that a session satisfaction evaluation metric should meet the following criteria: (1) The most useful document in a query is the most important; (2) The weighting function be- tween queries should be normalized; (3) The primacy effect is not suitable for session evaluation; (4) The recency effect has a stronger influence on user's session satisfaction. 5 In this study, we conducted two laboratory studies in which users need to give their usefulness judgments and satisfaction feedback. We investigated how users' query-level satisfaction contribute to their session-level satisfaction. Based on users' click sequence, we calculated a series of query-level evaluation metrics and found that users' query-level satisfaction mainly decided by the clicked document that is most useful from users' perspective. We tried different weighting function to fit session satisfaction with query satisfaction and found that a decaying function is not appropriate for session evaluation. Furthermore, through a linear regression analysis, we found that users' perceptions of the last query have the greatest impact on their session satisfaction. Finally, we proposed some criteria for designing session evaluation metrics based on the experiment results. As for future work, we would like to design click-sequence-based evaluation metrics for multi-query sessions. Beyond usefulness and satisfaction, more kinds of measures ( e.g. search success [5, 6]) will be investigated in our future work. 6
ACKNOWLEDGMENTS
This work is supported by Natural Science Foundation of China (Grant No. 61622208, 61732008, 61532011) and National Key Basic Research Program (2015CB358700).