OpenRaiser/Intern-Atlas huggingface dataset:
Structure Aware Experience Replay for Incremental Learning in Graph-based Recommender Systems
Authors: Kian Ahrabian, Yishi Xu, Yingxue Zhang, Jiapeng Wu, Yuening Wang, Mark Coates
Venue: CIKM
Year: 2021

Large-scale recommender systems are integral parts of many services. With the recent rapid growth of accessible data, the need for efficient training methods has arisen. Given the high computational cost of training state-of-the-art graph neural network (GNN) based models, it is infeasible to train them from scratch with every new set of interactions. In this work, we present a novel framework for incrementally training GNN-based models. Our framework takes advantage of an experience reply technique built on top of a structurally aware reservoir sampling method tailored for this setting. This framework addresses catastrophic forgetting, allowing the model to preserve its understanding of users' long-term behavioral patterns while adapting to new trends. Our experiments demonstrate the superior performance of our framework on numerous datasets when combined with state-of-the-art GNN-based models.
Model-agnostic vs. Model-intrinsic Interpretability for Explainable Product Search
Authors: Qingyao Ai, Lakshmi Narayanan Ramasamy
Venue: CIKM
Year: 2021

Product retrieval systems have served as the main entry for customers to discover and purchase products online. With increasing concerns on the transparency and accountability of AI systems, studies on explainable information retrieval has received more and more attention in the research community. Interestingly, in the domain of e-commerce, despite the extensive studies on explainable product recommendation, the studies of explainable product search is still in an early stage. In this paper, we study how to construct effective explainable product search by comparing model-agnostic explanation paradigms with model-intrinsic paradigms and analyzing the important factors that determine the performance of product search explanations. We propose an explainable product search model with model-intrinsic interpretability and conduct crowdsourcing to compare it with the state-of-the-art explainable product search model with model-agnostic interpretability. We observe that both paradigms have their own advantages and the effectiveness of search explanations on different properties are affected by different factors. For example, explanation fidelity is more important for user's overall satisfaction on the system while explanation novelty may be more useful in attracting user purchases. These findings could have important implications for the future studies and design of explainable product search engines.
Anomaly Mining: Past, Present and Future
Authors: Leman Akoglu
Venue: CIKM
Year: 2021

Anomaly mining finds high-stakes applications in various real-world domains such as cybersecurity, finance, environmental monitoring, to name a few. Therefore, it has been studied widely and a large body of detection techniques exists [1]. Today, many real-world settings necessitate detection at speed for streaming/evolving data, and/or detection at scale for massive datasets stored in a distributed environment [2]. Despite the plethora of detection algorithms, selecting an algorithm to use on a new task as well as setting the values for its hyperparameter(s), known as the model selection problem, is an open challenge for unsupervised anomaly detection. This issue is only to be exacerbated with the recent advent of detectors based on deep neural networks that exhibit a long list of hyperparameters. The challenge stems from two main factors: the lack of labeled data and the lack of a widely accepted anomaly loss function. Toward automation, one can explore internal evaluation strategies [3], or capitalize on the experience from historical detection tasks through meta-learning [4]. However, the problem remains far from solved. In deployment, many real-world use cases of anomaly detection require the flagged anomalies from a detector to be screened or audited by a human expert, typically for vetting purposes, where taking automatic actions can be costly (e.g. directly charging a flagged medical provider with fraud). While a vast majority of the literature focuses on novel detection algorithms, as humans are often involved with(in) the process, anomaly mining also concerns various human-centric problems that are beyond mere detection, namely explanation [5, 6], human interaction [7], and fairness [8]. These aspects of the field are under-studied and pose many open challenges.
XPL-CF: Explainable Embeddings for Feature-based Collaborative Filtering
Authors: Faisal M. Almutairi, Nicholas D. Sidiropoulos, Bo Yang
Venue: CIKM
Year: 2021

Collaborative filtering (CF) methods are making an impact on our daily lives in a wide range of applications, including recommender systems and personalization. Latent factor methods, e.g., matrix factorization (MF), have been the state-of-the-art in CF, however they lack interpretability and do not provide a straightforward explanation for their predictions. Explainability is gaining momentum in recommender systems for accountability, and because a good explanation can swing an undecided user. Most recent explainable recommendation methods require auxiliary data such as review text or item content on top of item ratings. In this paper, we address the case where no additional data are available and propose augmenting the classical MF framework for CF with a prior that encodes each user's embedding as a sparse linear combination of item embeddings, and vice versa for each item embedding. Our XPL-CF approach automatically reveals these user-item relationships, which underpin the latent factors and explain how the resulting recommendations are formed. We showcase the effectiveness of XPL-CF on real data from various application domains. We also evaluate the explainability of the user-item relationship obtained from XPL-CF through numeric evaluation and case study examples.
Automated Selection of Multiple Datasets for Extension by Integration
Authors: Yael Amsterdamer, Moran Cohen
Venue: CIKM
Year: 2021

Organizations often seek to extend their data by integration with available datasets originating from external sources. While there are many tools that recommend how to perform the integration for given datasets, the selection of what datasets to integrate is often challenging in itself. First, the relevant candidates must be efficiently identified among irrelevant ones. Next, relevant datasets need to be evaluated according to issues such as low quality or poor matching to the target data and schema. Last, jointly integrating multiple datasets may have significant benefits such as increasing completeness and information gain, but may also greatly complicate the task due to dependencies in the integration process. To assist administrators in this task, we quantify to what extent an integration of multiple datasets is valuable as an extension of an initial dataset and formalize the computational problem of finding the most valuable subset to integrate by this measure. We formally analyze the problem, showing that it is NP-hard; we nevertheless introduce heuristic efficient algorithms, which our experiments show to be near-optimal in practice and highly effective in finding the most valuable integration.
Discovering Conflicts of Interest across Heterogeneous Data Sources with ConnectionLens
Authors: Angelos-Christos G. Anadiotis, Oana Balalau, Théo Bouganim, Francesco Chimienti, Helena Galhardas, Mhd Yamen Haddad, Stephane Horel, Ioana Manolescu, Youssr Youssef
Venue: CIKM
Year: 2021

Investigative Journalism (IJ, in short) requires combining highly heterogeneous digital datasets coming from a wide variety of sources. We have developed ConnectionLens, a system that integrates such sources into a single heterogeneous graph and enables users to query the graph using keywords. The first iteration of the system [7] followed a mediator architecture which severely constrained its query scalability. Thus, we fully re-engineered the system, moving it to a warehouse architecture, and replacing its core components (information extraction, data querying, and interactive interfaces), which allowed us to handle uses cases orders of magnitude larger than the previous platform. In a consortium of computer scientists and investigative journalists, we propose to demonstrate ConnectionLens' capability to integrate arbitrary heterogeneous datasets and query them flexibly by means of keywords. Among several scenarios, our main focus will be on a real-world journalistic use case about situations which may lead to Conflicts of Interest between biomedical experts and various organizations, such as corporations, lobbies, etc. The demonstration will showcase the end-to-end data analysis pipeline, illustrate each system component, and the different parameters governing graph creation and querying.
Skyline in Crowdsourcing with Imprecise Comparisons
Authors: Aris Anagnostopoulos, Adriano Fazzone, Giacomo Vettraino
Venue: CIKM
Year: 2021

Given an input of a set of objects each one represented as a vector of features in a feature space, the problem of finding the skyline is the problem of determining the subset of objects that are not dominated by any other input object. An example of an application is to find the best hotel(s) with respect to some features (location, price, cleanliness, etc.) The use of the crowd for solving this problem is useful when a score of items according to their features is not available. Yet the crowd can give inconsistent answers. In this paper we study the computation of the skyline when the comparisons between objects are performed by humans. We model the problem using the threshold model [Ajtai et al, TALG 2015] in which the comparison of two objects may create errors/inconsistencies if the objects are close to each other. We provide algorithms for the problem and we analyze the required number of human comparisons and lower bounds. We also evaluate the effectiveness and efficiency of our algorithms using synthetic and real-world data.
A Formal Analysis of Recommendation Quality of Adversarially-trained Recommenders
Authors: Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Felice Antonio Merra
Venue: CIKM
Year: 2021

Recommender systems (RSs) employ user-item feedback, e.g., ratings, to match customers to personalized lists of products. Approaches to top-k recommendation mainly rely on Learning-To-Rank algorithms and, among them, the most widely adopted is Bayesian Personalized Ranking (BPR), which bases on a pair-wise optimization approach. Recently, BPR has been found vulnerable against adversarial perturbations of its model parameters. Adversarial Personalized Ranking (APR) mitigates this issue by robustifying BPR via an adversarial training procedure. The empirical improvements of APR's accuracy performance on BPR have led to its wide use in several recommender models. However, a key overlooked aspect has been the beyond-accuracy performance of APR, i.e., novelty, coverage, and amplification of popularity bias, considering that recent results suggest that BPR, the building block of APR, is sensitive to the intensification of biases and reduction of recommendation novelty. In this work, we model the learning characteristics of the BPR and APR optimization frameworks to give mathematical evidence that, when the feedback data have a tailed distribution, APR amplifies the popularity bias more than BPR due to an unbalanced number of received positive updates from short-head items. Using matrix factorization (MF), we empirically validate the theoretical results by performing preliminary experiments on two public datasets to compare BPR-MF and APR-MF performance on accuracy and beyond-accuracy metrics. The experimental results consistently show the degradation of novelty and coverage measures and a worrying amplification of bias.
Understanding Multi-channel Customer Behavior in Retail
Authors: Mozhdeh Ariannezhad, Sami Jullien, Pim Nauts, Min Fang, Sebastian Schelter, Maarten de Rijke
Venue: CIKM
Year: 2021

Online shopping is gaining popularity. Traditional retailers with physical stores adjust to this trend by allowing their customers to shop online as well as offline, in-store. Increasingly, customers can browse and purchase products across multiple shopping channels. Understanding how customer behavior relates to the availability of multiple shopping channels is an important prerequisite for many downstream machine learning tasks, such as recommendation and purchase prediction. However, previous work in this domain is limited to analyzing single-channel behavior only. In this paper, we provide the first insights into multi-channel customer behavior in retail based on a large sample of 2.8 million transactions originating from 300,000 customers of a food retailer in Europe. Our analysis reveals significant differences in customer behavior across online and offline channels, for example with respect to the repeat ratio of item purchases and basket size. Based on these findings, we investigate the performance of a next basket recommendation model under multi-channel settings. We find that the recommendation performance differs significantly for customers based on their choice of shopping channel, which strongly indicates that future research on recommenders in this area should take into account the particular characteristics of multi-channel retail shopping.
Non-Visual Accessibility Assessment of Videos
Authors: Ali Selman Aydin, Yu-Jung Ko, Utku Uckun, I. V. Ramakrishnan, Vikas Ashok
Venue: CIKM
Year: 2021

Video accessibility is crucial for blind screen-reader users as online videos are increasingly playing an essential role in education, employment, and entertainment. While there exist quite a few techniques and guidelines that focus on creating accessible videos, there is a dearth of research that attempts to characterize the accessibility of existing videos. Therefore in this paper, we define and investigate a diverse set of video and audio-based accessibility features in an effort to characterize accessible and inaccessible videos. As a ground truth for our investigation, we built a custom dataset of 600 videos, in which each video was assigned an accessibility score based on the number of its wins in a Swiss-system tournament, where human annotators performed pairwise accessibility comparisons of videos. In contrast to existing accessibility research where the assessments are typically done by blind users, we recruited sighted users for our effort, since videos comprise a special case where sight could be required to better judge if any particular scene in a video is presently accessible or not. Subsequently, by examining the extent of association between the accessibility features and the accessibility scores, we could determine the features that significantly (positively or negatively) impact video accessibility and therefore serve as good indicators for assessing the accessibility of videos. Using the custom dataset, we also trained machine learning models that leveraged our handcrafted features to either classify an arbitrary video as accessible/inaccessible or predict an accessibility score for the video. Evaluation of our models yielded an F1 score of 0.675 for binary classification and a mean absolute error of 0.53 for score prediction, thereby demonstrating their potential in video accessibility assessment while also illuminating their current limitations and the need for further research in this area.
Enabling Efficiency-Precision Trade-offs for Label Trees in Extreme Classification
Authors: Tavor Z. Baharav, Daniel L. Jiang, Kedarnath Kolluri, Sujay Sanghavi, Inderjit S. Dhillon
Venue: CIKM
Year: 2021

Extreme multi-label classification (XMC) aims to learn a model that can tag data points with a subset of relevant labels from an extremely large label set. Real world e-commerce applications like personalized recommendations and product advertising can be formulated as XMC problems, where the objective is to predict for a user a small subset of items from a catalog of several million products. For such applications, a common approach is to organize these labels into a tree, enabling training and inference times that are logarithmic in the number of labels [23]. While training a model once a label tree is available is well studied, designing the structure of the tree is a difficult task that is not yet well understood, and can dramatically impact both model latency and statistical performance. Existing approaches to tree construction either optimize exclusively for statistical performance or optimize exclusively for latency. We propose an efficient information theory inspired algorithm to construct intermediate operating points that trade off between the benefits of both, which was not previously possible. We corroborate our theoretical analysis with numerical results, showing that on the Wiki-500K [4] benchmark dataset our method can reduce a proxy for expected latency by up to 28% while maintaining the same accuracy as Parabel [23]. On several datasets derived from e-commerce customer logs, our modified label tree is able to improve this expected latency metric by up to 20% while maintaining the same accuracy. Finally, we discuss challenges in realizing these latency improvements in deployed models.
MVQAS: A Medical Visual Question Answering System
Authors: Haoyue Bai, Xiaoyan Shan, Yefan Huang, Xiaoli Wang
Venue: CIKM
Year: 2021

This paper demonstrates a medical visual question answering (VQA) system to address three challenges: 1) medical VQA often lacks large-scale labeled training data which requires huge efforts to build; 2) it is costly to implement and thoroughly compare medical VQA models on self-created datasets; 3) applying general VQA models to the medical domain by transfer learning is challenging due to various visual concepts between general images and medical images. Our system has three main components: data generation, model library, and model practice. To address the first challenge, we first allow users to upload self-collected clinical data such as electronic medical records (EMRs) to the data generation component and provides an annotating tool for labeling the data. Then, the system semi-automatically generates medical VQAs for users. Second, we develop a model library by implementing VQA models for users to evaluate their datasets. Users can do simple configurations by selecting self-interested models. The system then automatically trains the models, conducts extensive experimental evaluation, and reports comprehensive findings. The reports provide new insights into the strengths and weaknesses of selected models. Third, we provide an online chat module for users to communicate with an AI robots for further evaluating the models. The source codes are shared on https://github.com/shyanneshan/VQA-Demo.
Time-Aware Recommender System via Continuous-Time Modeling
Authors: Jianghan Bao, Yu Zhang
Venue: CIKM
Year: 2021

The overload of information on the Internet becomes ubiquitous nowadays, which makes the role of recommender systems more important. In recommender systems, the interest of users and popularity of items are not static, but can change drastically. Thus modeling the temporal dynamic of user-item interactions is crucial in recommender systems. The newly proposed Neural Ordinary Differential Equation (NODE) method is able to modeling the temporal mechanism of a system with neural networks. By using the ODE-LSTM method, which unites the ability of NODE to handle continuous time and that of LSTM to address sequential data, in this paper we achieve significant improvements for the recommendation task on several real-world datasets with the time irregularity. To handle sessions with different timestamps in ODE-LSTM, we propose a collective timeline technique that contributes a lot to the performance improvement. Moreover, we find that reducing the scale of time intervals in sessions significantly improves the recommendation performance.
Online Advertising Incrementality Testing: Practical Lessons And Emerging Challenges
Authors: Joel Barajas, Narayan Bhamidipati, James G. Shanahan
Venue: CIKM
Year: 2021

Online advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad-tech industries have progressed, advertisers have increasingly emphasized the causal effect estimation of their ads (incrementality) using controlled experiments (A/B testing). With low lift effects and sparse conversion, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in measurement precision. Similarly, the correct interpretation of results addressing a business goal requires significant data science and experimentation research expertise. We propose a practical tutorial in the incrementality testing landscape, including: The business need; Literature solutions and industry practices; Designs in the development of testing platforms; The testing cycle, case studies, and recommendations. We provide first-hand lessons based on the development of such a platform in a major combined DSP and ad network, and after running several tests for up to two months each over recent years.
Landmark Explanation: An Explainer for Entity Matching Models
Authors: Andrea Baraldi, Francesco Del Buono, Matteo Paganelli, Francesco Guerra
Venue: CIKM
Year: 2021

State-of-the-art approaches model Entity Matching (EM) as a binary classification problem, where Machine (ML) or Deep Learning (DL) based techniques are applied to evaluate if descriptions of pairs of entities refer to the same real-world instance. Despite these approaches have experimentally demonstrated to achieve high effectiveness, their adoption in real scenarios is limited by the lack of interpretability of their behavior. This paper showcases Landmark Explanation1, a tool that makes generic post-hoc (model-agnostic) perturbation-based explanation systems able to explain the behavior of EM models. In particular, Landmark Explanation computes local interpretations, i.e., given a description of a pair of entities and an EM model, it computes the contribution of each term in generating the prediction. The demonstration shows that the explanations generated by Landmark Explanation are effective even for non-matching pairs of entities, a challenge for explanation systems.
HopfE: Knowledge Graph Representation Learning using Inverse Hopf Fibrations
Authors: Anson Bastos, Kuldeep Singh, Abhishek Nadgeri, Saeedeh Shekarpour, Isaiah Onando Mulang&apos, Johannes Hoffart
Venue: CIKM
Year: 2021

Recently, several Knowledge Graph Embedding (KGE) approaches have been devised to represent entities and relations in a dense vector space and employed in downstream tasks such as link prediction. A few KGE techniques address interpretability, i.e., mapping the connectivity patterns of the relations (symmetric/asymmetric, inverse, and composition) to a geometric interpretation such as rotation. Other approaches model the representations in higher dimensional space such as four-dimensional space (4D) to enhance the ability to infer the connectivity patterns (i.e., expressiveness). However, modeling relation and entity in a 4D space often comes at the cost of interpretability. We propose HopfE, a novel KGE approach aiming to achieve the interpretability of inferred relations in the four-dimensional space. HopfE models the structural embeddings in 3D Euclidean space. Next, we map the entity embedding vector from a 3D Euclidean space to a 4D hypersphere using the inverse Hopf Fibration, in which we embed the semantic information from the KG ontology. Thus, HopfE considers the structural and semantic properties of the entities without losing expressivity and interpretability. Our empirical results on four well-known benchmarks achieve state-of-the-art performance for KG completion.