OpenRaiser/Intern-Atlas huggingface dataset:
Scope-aware Re-ranking with Gated Attention in Feed
Authors: Hao Qian, Qintong Wu, Kai Zhang, Zhiqiang Zhang, Lihong Gu, Xiaodong Zeng, Jun Zhou, Jinjie Gu
Venue: Conference WSDM 2022
Year: 2022

Modern recommendation systems introduce the re-ranking stage to optimize the entire list directly. This paper focuses on the design of re-ranking framework in feed to optimally model the mutual influence between items and further promote user engagement. On mobile devices, users browse the feed almost in a top-down manner and rarely compare items back and forth. Besides, users often compare item with its adjacency based on their partial observations. Given the distinct user behavior patterns, the modeling of mutual influence between items should be carefully designed. Existing re-ranking models encode the mutual influence between items with sequential encoding methods. However, previous works may be dissatisfactory due to the ignorance of connections between items on different scopes. In this paper, we first discuss Unidirectivity and Locality on the impacts and consequences, then report corresponding solutions in industrial applications. We propose a novel framework based on the empirical evidence from user analysis. To address the above problems, we design a \underlineS cope-aware \underlineR e-ranking with \underlineG ated \underlineA ttention model (SRGA ) to emulate the user behavior patterns from two aspects: 1) we emphasize the influence along the user's common browsing direction; 2) we strength the impacts of pivotal adjacent items within the user visual window. Specifically, we design a global scope attention to encode inter-item patterns unidirectionally from top to bottom. Besides, we devise a local scope attention sliding over the recommendation list to underline interactions among neighboring items. Furthermore, we design a learned gate mechanism to aggregating the information dynamically from local and global scope attention. Extensive offline experiments and online A/B testing demonstrate the benefits of our novel framework. The proposed SRGA model achieves the best performance in offline metrics compared with the state-of-the-art re-ranking methods. Further, empirical results on live traffic validate that our recommender system, equipped with SRGA in the re-ranking stage, improves significantly in user engagement.
Multi-Level Interaction Reranking with User Behavior History
Authors: Yunjia Xi, Weiwen Liu, Jieming Zhu, Xilong Zhao, Xinyi Dai, Ruiming Tang, Weinan Zhang, Rui Zhang, Yong Yu
Venue: Conference SIGIR 2022
Year: 2022

As the final stage of the multi-stage recommender system (MRS), reranking directly affects users' experience and satisfaction, thus playing a critical role in MRS. Despite the improvement achieved in the existing work, three issues are yet to be solved. First, users' historical behaviors contain rich preference information, such as users' long and short-term interests, but are not fully exploited in reranking. Previous work typically treats items in history equally important, neglecting the dynamic interaction between the history and candidate items. Second, existing reranking models focus on learning interactions at the item level while ignoring the fine-grained feature-level interactions. Lastly, estimating the reranking score on the ordered initial list before reranking may lead to the early scoring problem, thereby yielding suboptimal reranking performance. To address the above issues, we propose a framework named Multi-level Interaction Reranking (MIR). MIR combines low-level cross-item interaction and high-level set-to-list interaction, where we view the candidate items to be reranked as a set and the users' behavior history in chronological order as a list. We design a novel SLAttention structure for modeling the set-to-list interactions with personalized long-short term interests. Moreover, feature-level interactions are incorporated to capture the fine-grained influence among items. We design MIR in such a way that any permutation of the input items would not change the output ranking, and we theoretically prove it. Extensive experiments on three public and proprietary datasets show that MIR significantly outperforms the state-of-the-art models using various ranking and utility metrics.
Relevance judgments: Preferences, scores and ties
Authors: Ziying Yang
Venue: Conference SIGIR 2017
Year: 2017

Conventionally, relevance judgments were assessed using ordinal relevance scales such as binary and Sormunen categories [9]. Such judgments record how much overlap there is between the document and the topic. However they have been argued as unreliable and not objective [3, 5, 10] because: (1) documents are usually assessed by limited numbers of experts, with different viewpoints of relevance because of individual factors such as gender, age and background [1]; (2) the distinctions of relevance levels expected by users disparate types may be diverse [7]; (3) assessors' examining criteria drift in varying degrees as more documents are judged [8]; (4) many judgment ties are generated using ordinal scales. In order to have a better understanding of users' perceptions of relevance and collect data with high fidelity, we propose to use the Pairwise Preference technique [2] to collect relevance judgments from a crowdsourcing platform. With the collected judgments, a computed rank list containing all judged documents for each topic will be generated with the goal of having fewer relevance ties.
Relevance and Effort: An Analysis of Document Utility
Authors: Emine Yilmaz, Manisha Verma, Nick Craswell, Filip Radlinski, Peter Bailey
Venue: Conference CIKM 2014
Year: 2014

In this paper, we study one important source of the mis-match between user data and relevance judgments, those due to the high degree of effort required by users to identify and consume the information in a document. Information retrieval relevance judges are trained to search for evidence of relevance when assessing documents. For complex documents, this can lead to judges' spending substantial time considering each document. However, in practice, search users are often much more impatient: if they do not see evidence of relevance quickly, they tend to give up.
Counterfactual Evaluation and Learning for Interactive Systems: Foundations, Implementations, and Recent Advances
Authors: Yuta Saito, Thorsten Joachims
Venue: Conference KDD 2022
Year: 2022

Counterfactual estimators enable the use of existing log data to estimate how some new target policy would have performed, if it had been used instead of the policy that logged the data. We say that those estimators work "off-policy", since the policy that logged the data is different from the target policy. In this way, counterfactual estimators enable Off-policy Evaluation (OPE) akin to an unbiased offline A/B test, as well as learning new decision-making policies through Off-policy Learning (OPL). The goal of this tutorial is to summarize Foundations, Implementations, and Recent Advances of OPE and OPL (OPE/OPL), with applications in recommendation, search, and an ever growing range of interactive systems. Specifically, we will introduce the fundamentals of OPE/OPL and provide theoretical and empirical comparisons of conventional methods. Then, we will cover emerging practical challenges such as how to handle large action spaces, distributional shift, and hyper-parameter tuning. We will then present Open Bandit Pipeline, an open-source Python software for OPE/OPL to better enable new research and applications. We will conclude the tutorial with future directions.
Multimodal Pre-Training with Self-Distillation for Product Understanding in E-Commerce
Authors: Shilei Liu, Lin Li, Jun Song, Yonghua Yang, Xiaoyi Zeng
Venue: Conference WSDM 2023
Year: 2023

Product understanding refers to a series of product-centric tasks, such as classification, alignment and attribute values prediction, which requires fine-grained fusion of various modalities of products. Excellent product modeling ability will enhance the user experience and benefit search and recommendation systems. In this paper, we propose MBSD, a pre-trained vision-and-language model which can integrate the heterogeneous information of product in a single stream BERT-style architecture. Compared with current approaches, MBSD uses a lightweight convolutional neural network instead of a heavy feature extractor for image encoding, which has lower latency. Besides, we cleverly utilize user behavior data to design a two-stage pre-training task to understand products from different perspectives. In addition, there is an underlying imbalanced problem in multimodal pre-training, which will impairs downstream tasks. To this end, we propose a novel self-distillation strategy to transfer the knowledge in dominated modality to weaker modality, so that each modality can be fully tapped during pre-training. Experimental results on several product understanding tasks demonstrate that the performance of MBSD outperforms the competitive baselines.
Deliberate Exposure to Opposing Views and its Association with Behavior and Rewards on Political Communities
Authors: Alexandros Efstratiou
Venue: Conference WWW 2024
Year: 2024

Engaging with diverse political views is important for reaching better collective decisions, however, users online tend to remain confined within ideologically homogeneous spaces. In this work, we study users who are members of these spaces but who also show a willingness to engage with diverse views, as they have the potential to introduce more informational diversity into their communities. Across four Reddit communities (r/Conservative, r/The_Donald, r/ChapoTrapHouse, r/SandersForPresident), we find that these users tend to use less hostile and more advanced and personable language, but receive fewer social rewards from their peers compared to others. We also find that social sanctions on the discussion community r/changemyview are insufficient to drive them out in the short term, though they may play a role over the longer term.
SDFVAE: Static and Dynamic Factorized VAE for Anomaly Detection of Multivariate CDN KPIs
Authors: Liang Dai, Tao Lin, Chang Liu, Bo Jiang, Yanwei Liu, Zhen Xu, Zhi-Li Zhang
Venue: Conference WWW 2021
Year: 2021

Content Delivery Networks (CDNs) are critical for providing good user experience of cloud services. CDN providers typically collect various multivariate Key Performance Indicators (KPIs) time series to monitor and diagnose system performance. State-of-the-art anomaly detection methods mostly use deep learning to extract the normal patterns of data, due to its superior performance. However, KPI data usually exhibit non-additive Gaussian noise, which makes it difficult for deep learning models to learn the normal patterns, resulting in degraded performance in anomaly detection. In this paper, we propose a robust and noise-resilient anomaly detection mechanism using multivariate KPIs. Our key insight is that different KPIs are constrained by certain time-invariant characteristics of the underlying system, and that explicitly modelling such invariance may help resist noise in the data. We thus propose a novel anomaly detection method called SDFVAE, short for Static and Dynamic Factorized VAE, that learns the representations of KPIs by explicitly factorizing the latent variables into dynamic and static parts. Extensive experiments using real-world data show that SDFVAE achieves a F1-score ranging from 0.92 to 0.99 on both regular and noisy dataset, outperforming state-of-the-art methods by a large margin.
CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting
Authors: Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodríguez, Chao Zhang, B. Aditya Prakash
Venue: Conference WWW 2022
Year: 2022

Probabilistic time-series forecasting enables reliable decision making across many domains. Most forecasting problems have diverse sources of data containing multiple modalities and structures. Leveraging information as well as uncertainty from these data sources for well-calibrated and accurate forecasts is an important challenging problem. Most previous work on multi-modal learning and forecasting simply aggregate intermediate representations from each data view by simple methods of summation or concatenation and do not explicitly model uncertainty for each data-view. We propose a general probabilistic multi-view forecasting framework CAMul, that can learn representations and uncertainty from diverse data sources. It integrates the knowledge and uncertainty from each data view in a dynamic context-specific manner assigning more importance to useful views to model a well-calibrated forecast distribution. We use CAMul for multiple domains with varied sources and modalities and show that CAMul outperforms other state-of-art probabilistic forecasting models by over 25\% in accuracy and calibration.
Detection of Infectious Disease Outbreaks in Search Engine Time Series Using Non-Specific Syndromic Surveillance with Effect-Size Filtering
Authors: Oded Ovadia, Oren Elisha, Elad Yom-Tov
Venue: Conference WWW 2022
Year: 2022

Novel infectious disease outbreaks, including most recently that of the COVID-19 pandemic, could be detected by non-specific syndromic surveillance systems. Such systems, utilizing a variety of data sources ranging from Electronic Health Records to internet data such as aggregated search engine queries, create alerts when unusually high rates of symptom reports occur. This is especially important for the detection of novel diseases, where their manifested symptoms are unknown. Here we improve upon a set of previously-proposed non-specific syndromic surveillance methods by taking into account both how unusual a preponderance of symptoms is and their effect size. We demonstrate that our method is as accurate as previously-proposed methods for low dimensional data and show its effectiveness for high-dimensional aggregated data by applying it to aggregated time-series health-related search engine queries. We find that in 2019 the method would have raised alerts related to several disease outbreaks earlier than health authorities did. During the COVID-19 pandemic the system identified the beginning of pandemic waves quickly, through combinations of symptoms which varied from wave to wave. Thus, the proposed method could be used as a practical tool for decision makers to detect new disease outbreaks using time series derived from search engine data even in the absence of specific information on the diseases of interest and their symptoms.
Review Spam Detection via Time Series Pattern Discovery
Authors: Sihong Xie, Guan Wang, Shuyang Lin, Philip S. Yu
Venue: Conference WWW 2012
Year: 2012

Online reviews play a crucial role in today's electronic commerce. Due to the pervasive spam reviews, customers can be misled to buy low-quality products, while decent stores can be defamed by malicious reviews. We observe that, in reality, a great portion (> 90% in the data we study) of the reviewers write only one review (singleton review). These reviews are so enormous in number that they can almost determine a store's rating and impression. However, existing methods ignore these reviewers. To address this problem, we observe that the normal reviewers' arrival pattern is stable and uncorrelated to their rating pattern temporally. In contrast, spam attacks are usually bursty and either positively or negatively correlated to the rating. Thus, we propose to detect such attacks via unusually correlated temporal patterns. We identify and construct multidimensional time series based on aggregate statistics, in order to depict and mine such correlation. Experimental results show that the proposed method is effective in detecting singleton review attacks. We discover that singleton review is a significant source of spam reviews and largely affects the ratings of online stores.
Causal understanding of fake news dissemination on social media
Authors: Lu Cheng, Ruocheng Guo, Kai Shu, Huan Liu
Venue: Conference KDD 2021
Year: 2021

Recent years have witnessed remarkable progress towards computational fake news detection. To mitigate its negative impact, we argue that it is critical to understand what user attributes potentially cause users to share fake news. The key to this causal-inference problem is to identify confounders -- variables that cause spurious associations between treatments (e.g., user attributes) and outcome (e.g., user susceptibility). In fake news dissemination, confounders can be characterized by fake news sharing behavior that inherently relates to user attributes and online activities. Learning such user behavior is typically subject to selection bias in users who are susceptible to share news on social media. Drawing on causal inference theories, we first propose a principled approach to alleviating selection bias in fake news dissemination. We then consider the learned unbiased fake news sharing behavior as the surrogate confounder that can fully capture the causal links between user attributes and user susceptibility. We theoretically and empirically characterize the effectiveness of the proposed approach and find that it could be useful in protecting society from the perils of fake news.
Detecting large reshare cascades in social networks
Authors: Karthik Subbian, B. Aditya Prakash, Lada A. Adamic
Venue: Conference WWW 2017
Year: 2017

Detecting large reshare cascades is an important problem in online social networks. There are a variety of attempts to model this problem, from using time series analysis methods to stochastic processes. Most of these approaches heavily depend on the underlying network features and use network information to detect the virality of cascades. In most cases, however, getting such detailed network information can be hard or even impossible.
Modelling dynamics in semantic web knowledge graphs with formal concept analysis
Authors: Larry González, Aidan Hogan
Venue: Conference WWW 2018
Year: 2018

In this paper, we propose a novel data-driven schema for large-scale heterogeneous knowledge graphs inspired by Formal Concept Analysis (FCA). We first extract the sets of properties associated with individual entities; these property sets (aka. characteristic sets) are annotated with cardinalities and used to induce a lattice based on set-containment relations, forming a natural hierarchical structure describing the knowledge graph. We then propose an algebra over such schema lattices, which allows to compute diffs between lattices (for example, to summarise the changes from one version of a knowledge graph to another), to add lattices (for example, to project future changes), and so forth. While we argue that this lattice structure (and associated algebra) may have various applications, we currently focus on the use-case of modelling and predicting the dynamic behaviour of knowledge graphs. Along those lines, we instantiate and evaluate our methods for analysing how versions of the Wikidata knowledge graph have changed over a period of 11 weeks. We propose algorithms for constructing the lattice-based schema from Wikidata, and evaluate their efficiency and scalability. We then evaluate use of the resulting schema(ta) for predicting how the knowledge graph will evolve in future versions.
TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds
Authors: Karan Vombatkere, Sepehr Mousavi, Savvas Zannettou, Franziska Roesner, Krishna P. Gummadi
Venue: Conference WWW 2024
Year: 2024

Recommendation algorithms for social media feeds often function as black boxes from the perspective of users. We aim to detect whether social media feed recommendations are personalized to users, and to characterize the factors contributing to personalization in these feeds. We introduce a general framework to examine a set of social media feed recommendations for a user as a timeline. We label items in the timeline as the result of exploration vs. exploitation of the user's interests on the part of the recommendation algorithm and introduce a set of metrics to capture the extent of personalization across user timelines. We apply our framework to a real TikTok dataset and validate our results using a baseline generated from automated TikTok bots, as well as a randomized baseline. We also investigate the extent to which factors such as video viewing duration, liking, and following drive the personalization of content on TikTok. Our results demonstrate that our framework produces intuitive and explainable results, and can be used to audit and understand personalization in social media feeds.
Towards controllable explanation generation for recommender systems via neural template
Authors: Lei Li, Li Chen, Yongfeng Zhang
Venue: Conference WWW 2020
Year: 2020

It has been commonly agreed that the explanation associated with recommendation can be effective in increasing the recommender systems (RS)’s transparency and thus users’ satisfaction and acceptance. Among the various types of explanation in RS, the commonly used textual explanation can be roughly classified into two categories, i.e., template-based and generation-based. As for the former, the fixed template may lose flexibility, while, though the latter may enrich the explanation, it may produce less useful content due to the lack of controllability. In this work, we combine the advantages of the two types of method by developing a neural generation approach named Neural Template (NETE) whose explanations are not only flexible but also controllable and useful. Our human evaluation results confirm that the explanations from our model are perceived helpful by users. Furthermore, our case study illustrates that the explanation generation process is controllable. To demonstrate the controllability of our model, we present a demo that can be easily viewed on a Web browser.
Dual learning for explainable recommendation: Towards unifying user preference prediction and review generation
Authors: Peijie Sun, Le Wu, Kun Zhang, Yanjie Fu, Richang Hong, Meng Wang
Venue: Conference WWW 2020
Year: 2020

In many recommender systems, users express item opinions through two kinds of behaviors: giving preferences and writing detailed reviews. As both kinds of behaviors reflect users’ assessment of items, review enhanced recommender systems leverage these two kinds of user behaviors to boost recommendation performance. On the one hand, researchers proposed to better model the user and item embeddings with additional review information for enhancing preference prediction accuracy. On the other hand, some recent works focused on automatically generating item reviews for recommendation explanations with related user and item embeddings. We argue that, while the task of preference prediction with the accuracy goal is well recognized in the community, the task of generating reviews for explainable recommendation is also important to gain user trust and increase conversion rate. Some preliminary attempts have considered jointly modeling these two tasks, with the user and item embeddings are shared. These studies empirically showed that these two tasks are correlated, and jointly modeling them would benefit the performance of both tasks. In this paper, we make a further study of unifying these two tasks for explainable recommendation. Instead of simply correlating these two tasks with shared user and item embeddings, we argue that these two tasks are presented in dual forms. In other words, the input of the primal preference prediction task is exactly the output of the dual review generation task , with and denote the preference value space and review space. Therefore, we could explicitly model the probabilistic correlation between these two dual tasks with . We design a unified dual framework of how to inject the probabilistic duality of the two tasks in the training stage. Furthermore, as the detailed preference and review information are not available for each user-item pair in the test stage, we propose a transfer learning based model for preference prediction and review generation. Finally, extensive experimental results on two real-world datasets clearly show the effectiveness of our proposed model for both user preference prediction and review generation.