OpenRaiser/Intern-Atlas huggingface dataset:
FARA: Future- Aware Ranking Algorithm for Fairness Optimization
Authors: Tao Yang, Zhichao Xu, Zhenduo Wang, Qingyao Ai
Venue: Conference CIKM 2023
Year: 2023

Ranking systems are the key components of modern Information Retrieval (IR) applications, such as search engines and recommender systems. Besides the ranking relevance to users, the exposure fairness to item providers has also been considered an important factor in ranking optimization. Many fair ranking algorithms have been proposed to jointly optimize both ranking relevance and fairness. However, we find that most existing fair ranking methods adopt greedy algorithms that only optimize rankings for the next immediate session or request. As shown in this paper, such a myopic paradigm could limit the upper bound of ranking optimization and lead to suboptimal performance in the long term. To this end, we propose \textbf{FARA}, a novel \textbf{F}uture-\textbf{A}ware \textbf{R}anking \textbf{A}lgorithm for ranking relevance and fairness optimization. Instead of greedily optimizing rankings for the next immediate session, FARA plans ahead by jointly optimizing multiple ranklists together and saving them for future sessions. Specifically, FARA first uses the Taylor expansion to investigate how future ranklists will influence the overall fairness of the system. Then, based on the analysis of the Taylor expansion, FARA adopts a two-phase optimization algorithm where we first solve an optimal future exposure planning problem and then construct the optimal ranklists according to the optimal future exposure planning. Theoretically, we show that FARA is optimal for ranking relevance and fairness joint optimization. Empirically, our extensive experiments on three semi-synthesized datasets show that FARA is efficient, effective, and can deliver significantly better ranking performance compared to state-of-the-art fair ranking methods. We make our implementation public at \href{https://github.com/Taosheng-ty/QP_fairness/}{https://github.com/Taosheng-ty/QP\_fairness/}.
Marginal-Certainty-Aware Fair Ranking Algorithm
Authors: Tao Yang, Zhichao Xu, Zhenduo Wang, Anh Tran, Qingyao Ai
Venue: Conference WSDM 2023
Year: 2023

Ranking systems are ubiquitous in modern Internet services, including online marketplaces, social media, and search engines. Traditionally, ranking systems only focus on how to get better relevance estimation. When relevance estimation is available, they usually adopt a user-centric optimization strategy where ranked lists are generated by sorting items according to their estimated relevance. However, such user-centric optimization ignores the fact that item providers also draw utility from ranking systems. It has been shown in existing research that such user-centric optimization will cause much unfairness to item providers, followed by unfair opportunities and unfair economic gains for item providers. To address ranking fairness, many fair ranking methods have been proposed. However, as we show in this paper, these methods could be suboptimal as they directly rely on the relevance estimation without being aware of the uncertainty (i.e., variance of the estimated relevance). To address this uncertainty, we propose a novel Marginal-Certainty-aware Fair algorithm named MCFair. MCFair jointly optimizes fairness and user utility, while relevance estimation is constantly updated in an online manner. In MCFair, we first develop a ranking objective that includes uncertainty, fairness, and user utility. Then we directly use the gradient of the ranking objective as the ranking score. We theoretically prove that MCFair based on gradients is optimal for the aforementioned ranking objective. Empirically, we find that on semi-synthesized datasets, MCFair is effective and practical and can deliver superior performance compared to state-of-the-art fair ranking methods. To facilitate reproducibility, we release our code.
User Consumption Intention Prediction in Meituan
Authors: Yukun Ping, Chen Gao, Taichi Liu, Xiaoyi Du, Hengliang Luo, Depeng Jin, Yong Li
Venue: Conference KDD 2021
Year: 2021

For online life service platforms, such as Meituan, user consumption intention, as the internal driving force of consumption behaviors, plays a significant role in understanding and predicting users' demand and purchase. However, user consumption intention prediction is quite challenging. Different from consumption behaviors, consumption intention is implicit and always not reflected by behavioral data. Moreover, it is affected by both user intrinsic preference and spatio-temporal context. To overcome these challenges, in Meituan, we design a real-world system consisting of two stages, intention detection and prediction. Specifically, at the intention-detection stage, we combine the knowledge of human experts and consumption information to obtain explicit intentions and match consumption with intentions based on user review data. At the intention-prediction stage, to collectively exploit the rich heterogeneous influencing factors, we design a graph neural network-based intention prediction model GRIP, which can capture user intrinsic preference and spatio-temporal context. Extensive offline evaluations demonstrate that our prediction model outperforms the best baseline by 10.26% and 33.28% for two metrics and online A/B tests on millions of users validate the effectiveness of our system.
Engaged or Frustrated? Disambiguating Emotional State in Search
Authors: Ashlee Edwards, Diane Kelly
Venue: Conference SIGIR 2017
Year: 2017

One of the primary ways researchers have characterized engagement during information search is by increases in search behaviors, such as queries and clicks. However, studies have shown that frustration is also characterized by increases in these same behaviors. This research examines the differences in the search behaviors and physiologies of people who are engaged or frustrated during search. A 2x2 within-subject laboratory experiment was conducted with 40 participants. Engagement was induced by manipulating task interest and frustration was induced by manipulating the quality of the search results. Participants' interactions and physiological responses were recorded, and after they searched, they evaluated their levels of engagement, frustration and stress. Participants reported significantly greater levels of engagement when completing tasks that interested them and significantly less engagement during searches with poor results quality. For all search behaviors measured, only two significant differences were found according to task interest: participants had more scrolls and longer query intervals when searching for interesting tasks, suggesting greater interaction with content. Significant differences were found for nine behaviors according to results quality, including queries issued, number of SERPs displayed and number of SERP clicks, suggesting these are potentially better indicators of frustration rather than engagement. When presented with poor quality results, participants had significantly higher heart rates than when presented with normal quality results. Finally, participants had lower heart rates and greater skin conductance responses when conducting interesting tasks than when conducting uninteresting tasks. This research provides insight into the differences in search behaviors and physiologies of participants when they are engaged versus frustrated and presents techniques that can be used by those wishing to induce engagement and frustration during laboratory IIR studies.
Cognitive Activity during Web Search
Authors: Md. Hedayetul Islam Shovon, Nanda Nandagopal, Jia Tina Du, Ramasamy Vijayalakshmi, Bernadine Cocks
Venue: Conference SIGIR 2015
Year: 2015

Searching on the Web or Net-surfing is a part of everyday life for many people, but little is known about the brain activity during Web searching. Such knowledge is essential for better understanding of the cognitive demands imposed by the search system and search tasks. The current study contributes to this understanding by constructing brain networks from EEG data using normalized transfer entropy (NTE) during three Web search task stages: query formulation, viewing of a search result list and reading each individual content page. This study further contributes to the connectivity analysis of the constructed brain networks, since it is an advanced quantitative technique which enables the exploration of brain function by distinct and varied brain areas. By using this approach, we identified that the cognitive activities during the three stages of Web searching are different, with various brain areas becoming more active during the three Web search task stages. Of note, query formulation generated higher interaction between cortical regions than viewing a result list or reading a content page. These findings will have implications for the improvement of Web search engines and search interfaces.
Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system
Authors: Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, Aldo Lipani
Venue: Conference CIKM 2023
Year: 2023

Dialogue systems and large language models (LLMs) have gained considerable attention. However, the direct utilization of LLMs as task-oriented dialogue (TOD) models has been found to underperform compared to smaller task-specific models. Nonetheless, it is crucial to acknowledge the significant potential of LLMs and explore improved approaches for leveraging their impressive abilities. Motivated by the goal of leveraging LLMs, we propose an alternative approach called User-Guided Response Optimization (UGRO) to combine it with a smaller TOD model. This approach uses LLM as an annotation-free user simulator to assess dialogue responses, combining them with smaller fine-tuned end-to-end TOD models. By utilizing the satisfaction feedback generated by LLMs, UGRO further optimizes the supervised fine-tuned TOD model. Specifically, the TOD model takes the dialogue history as input and, with the assistance of the user simulator's feedback, generates high-satisfaction responses that meet the user's requirements. Through empirical experiments on two TOD benchmarks, we validate the effectiveness of our method. The results demonstrate that our approach outperforms previous state-of-the-art (SOTA) results.
System Initiative Prediction for Multi-turn Conversational Information Seeking
Authors: Chuan Meng, Mohammad Aliannejadi, Maarten de Rijke
Venue: Conference CIKM 2023
Year: 2023

Identifying the right moment for a system to take the initiative is essential to conversational information seeking (CIS). Existing studies have extensively studied the clarification need prediction task, i.e., predicting when to ask a clarifying question, however, it only covers one specific system-initiative action. We define the system initiative prediction (SIP) task as predicting whether a CIS system should take the initiative at the next turn. Our analysis reveals that for effective modeling of SIP, it is crucial to capture dependencies between adjacent user?system initiative-taking decisions. We propose to model SIP by CRFs. Due to their graphical nature, CRFs are effective in capturing such dependencies and have greater transparency than more complex methods, e.g., LLMs. Applying CRFs to SIP comes with two challenges: (i) CRFs need to be given the unobservable system utterance at the next turn, and (ii) they do not explicitly model multi-turn features. We model SIP as an input-incomplete sequence labeling problem and propose a multi-turn system initiative predictor (MuSIc) that has (i) prior-posterior inter-utterance encoders to eliminate the need to be given the unobservable system utterance, and (ii) a multi-turn feature-aware CRF layer to incorporate multi-turn features into the dependencies between adjacent initiative-taking decisions. Experiments show that MuSIc outperforms LLM-based baselines including LLaMA, achieving state-of-the-art results on SIP. We also show the benefits of SIP on clarification need prediction and action prediction.
Recommendation with Causality enhanced Natural Language Explanations
Authors: Jingsen Zhang, Xu Chen, Jiakai Tang, Weiqi Shao, Quanyu Dai, Zhenhua Dong, Rui Zhang
Venue: Conference WWW 2023
Year: 2023

Explainable recommendation has recently attracted increasing attention from both academic and industry communities. Among different explainable strategies, generating natural language explanations is an important method, which can deliver more informative, flexible and readable explanations to facilitate better user decisions. Despite the effectiveness, existing models are mostly optimized based on the observed datasets, which can be skewed due to the selection or exposure bias. To alleviate this problem, in this paper, we formulate the task of explainable recommendation with a causal graph, and design a causality enhanced framework to generate unbiased explanations. More specifically, we firstly define an ideal unbiased learning objective, and then derive a tractable loss for the observational data based on the inverse propensity score (IPS), where the key is a sample re-weighting strategy for equalizing the loss and ideal objective in expectation. Considering that the IPS estimated from the sparse and noisy recommendation datasets can be inaccurate, we introduce a fault tolerant mechanism by minimizing the maximum loss induced by the sample weights near the IPS. For more comprehensive modeling, we further analyze and infer the potential latent confounders induced by the complex and diverse user personalities. We conduct extensive experiments by comparing with the state-of-the-art methods based on three real-world datasets to demonstrate the effectiveness of our method.
Evaluating Generative Ad Hoc Information Retrieval
Authors: Lukas Gienapp, Harrisen Scells, Niklas Deckers, Janek Bevendorff, Shuai Wang, Johannes Kiesel, Shahbaz Syed, Maik Fröbe, Guido Zuccon, Benno Stein, Matthias Hagen, Martin Potthast
Venue: Conference SIGIR 2024
Year: 2024

Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.
Crowdsourcing a wikipedia vandalism corpus
Authors: Martin Potthast
Venue: Conference SIGIR 2010
Year: 2010

We report on the construction of the PAN Wikipedia vandalism corpus, PAN-WVC-10, using Amazon's Mechanical Turk. The corpus compiles 32452 edits on 28468 Wikipedia articles, among which 2391 vandalism edits have been identified. 753 human annotators cast a total of 193022 votes on the edits, so that each edit was reviewed by at least 3 annotators, whereas the achieved level of agreement was analyzed in order to label an edit as "regular" or "vandalism." The corpus is available free of charge.
Improving web browsing performance on wireless pdas using thin-client computing
Authors: Albert M. Lai, Jason Nieh, Bhagyashree Bohra, Vijayarka Nandikonda, Abhishek P. Surana, Suchita Varshneya
Venue: Conference WWW 2004
Year: 2004

Web applications are becoming increasingly popular for mobile wireless PDAs. However, web browsing on these systems can be quite slow. An alternative approach is handheld thin-client computing, in which the web browser and associated application logic run on a server, which then sends simple screen updates to thePDA for display. To assess the viability of this thin-client approach, we compare the web browsing performance of thin clients against fat clients that run the web browser locally on a PDA. Our results show that thin clients can provide better web browsing performance compared to fat clients, both in terms of speed and ability to correctly display web content. Surprisingly, thin clients are faster even when having to send more data over the network. We characterize and analyze different design choices in various thin-client systems and explain why these approaches can yield superior web browsing performance on mobile wireless PDAs.
pTHINC: a thin-client architecture for mobile wireless web
Authors: Joeng Kim, Ricardo A. Baratto, Jason Nieh
Venue: Conference WWW 2006
Year: 2006

Although web applications are gaining popularity on mobile wireless PDAs, web browsers on these systems can be quite slow and often lack adequate functionality to access many web sites. We have developed pTHINC, a PDA thin-client solution that leverages more powerful servers to run full-function web browsers and other application logic, then sends simple screen updates to the PDA for display. pTHINC uses server-side screen scaling to provide high-fidelity display and seamless mobility across a broad range of different clients and screen sizes, including both portrait and landscape viewing modes. pTHINC also leverages existing PDA control buttons to improve system usability and maximize available screen resolution for application display. We have implemented pTHINC on Windows Mobile and evaluated its performance on mobile wireless devices. Our results compared to local PDA web browsers and other thin-client approaches demonstrate that pTHINC provides superior web browsing performance and is the only PDA thin client that effectively supports crucial browser helper applications such as video playback.
Session level techniques for improving web browsing performance on wireless links
Authors: Pablo Rodriguez, Sarit Mukherjee, Sampath Rangarajan
Venue: Conference WWW 2004
Year: 2004

Recent observations through experiments that we have performed in current third generation wireless networks have revealed that the achieved throughput over wireless links varies widely depending on the application. In particular, the throughput achieved by file transfer application (FTP) and web browsing application (HTTP) are quite different. The throughput achieved over a HTTP session is much lower than that achieved over an FTP session. The reason for the lower HTTP throughput is that the HTTP protocol is affected by the large Round-Trip Time (RTT) across Wireless links. HTTP transfers require multiple TCP connections and DNS lookups before a HTTP page can be displayed. Each TCP connection requires several RTTs to fully open the TCP send window and each DNS lookup requires several RTTs before resolving the domain name to IP mapping. These TCP/DNS RTTs significantly degrade the performance of HTTP over wireless links. To overcome these problems, we have developed session level optimization techniques to enhance HTTP download mechanisms. These techniques (a) minimize the number of DNS lookups over the wireless link and (b) minimize the number of TCP connections opened by the browser. These optimizations bridge the mismatch caused by wireless links between application-level protocols (such as HTTP) and transportlevel protocols (such as TCP). Our solutions do not require any client-side software and can be deployed transparently on a service provider network to provide 30 50% decrease in end-to-end user perceived latency and 50-100% increase in data throughput across wireless links for HTTP sessions.
Fast and parallel webpage layout
Authors: Leo A. Meyerovich, Rastislav Bodík
Venue: Conference WWW 2010
Year: 2010

The web browser is a CPU-intensive program. Especially on mobile devices, webpages load too slowly, expending significant time in processing a document's appearance. Due to power constraints, most hardware-driven speedups will come in the form of parallel architectures. This is also true of mobile devices such as phones and e-books. In this paper, we introduce new algorithms for CSS selector matching, layout solving, and font rendering, which represent key components for a fast layout engine. Evaluation on popular sites shows speedups as high as 80x. We also formulate the layout problem with attribute grammars, enabling us to not only parallelize our algorithm but prove that it computes in O(log) time and without reflow.
Smart caching for web browsers
Authors: Kaimin Zhang, Lu Wang 0002, Aimin Pan, Bin Benjamin Zhu
Venue: Conference WWW 2010
Year: 2010

This paper presents smart caching schemes for Web browsers. For modern Web applications, the style formatting and layout calculation often account for substantial amounts of the local computation in order to render a Web page. In this paper, we propose two caching schemes to reduce the computation of style formatting and layout calculation, named smart style caching and layout caching, respectively. The stable style data and layout data for DOM (Document Object Model) elements are recorded to construct the caches when a Web page is browsed. The cached data is checked in the granularity of DOM elements and applied directly if the identified DOM element is not changed in the sequent visits to the same page.
Nonintrusive-sensing and reinforcement-learning based adaptive personalized music recommendation
Authors: Daocheng Hong, Yang Li, Qiwen Dong
Venue: Conference SIGIR 2020
Year: 2020

As a particularly prominent application of recommender systems on automated personalized service, the music recommendation has been widely used in various music network platforms, music education and music therapy. Importantly, the individual music preference for a certain moment is closely related to personal experience of the music and music literacy, as well as temporal scenario without any interruption. Therefore, this paper proposes a novel policy for music recommendation NRRS (Nonintrusive-Sensing and Reinforcement-Learning based Recommender Systems) by integrating prior research streams. Specifically, we develop a novel recommendation framework for sensing, learning and adaptation to user's current preference based on wireless sensing and reinforcement learning in real time during a listening session. The established music recommendation prototype monitors individual vital signals for listening music, and captures song characters, individual dynamic preferences, and that it can yield better listening experience for users.
Wsdm cup 2018: Music recommendation and churn prediction
Authors: Yian Chen, Xing Xie, Shou-De Lin, Arden Chiu
Venue: Conference WSDM 2018
Year: 2018

Excellent recommendation system facilitates users retrieving contents they like and, what»s much more important - the contents they might like but they are not aware of yet. It will further increase the satisfaction of users and increase the retention rate and conversion rate indirectly. While the public's now listening to all kinds of music, recommendation algorithms still struggle in key areas. Without enough historical data, how would an algorithm know if listeners will like a new song or a new artist? And, how would it know what songs to recommend brand new users? In WSDM Cup 2018, the first task is to solve the abovementioned challenges to build a better music recommendation system. The 2nd task in the Cup focuses on churn prediction. For a subscription business, accurately predicting churn is critical to long-term success. Even slight variations in churn can drastically affect profits. In this task, participants are asked to build an algorithm that predicts whether a user will churn after their subscription expires. The competition data and award are provided by KKBOX, a leading music streaming service in Taiwan.