OpenRaiser/Intern-Atlas huggingface dataset:
Rethinking Conversational Recommendations: Is Decision Tree All You Need?
Authors: A. S. M. Ahsan-Ul-Haque, Hongning Wang
Venue: CIKM
Year: 2022

Conversational recommender systems (CRS) dynamically obtain the users' preferences via multi-turn questions and answers. The existing CRS solutions are widely dominated by deep reinforcement learning algorithms. However, deep reinforcement learning methods are often criticized for lacking interpretability and requiring a large amount of training data to perform. In this paper, we explore a simpler alternative and propose a decision tree based solution to CRS. The underlying challenge in CRS is that the same item can be described differently by different users. We show that decision trees are sufficient to characterize the interactions between users and items, and solve the key challenges in multi-turn CRS: namely which questions to ask, how to rank the candidate items, when to recommend, and how to handle user's negative feedback on the recommendations. Firstly, the training of decision trees enables us to find questions which effectively narrow down the search space. Secondly, by learning embeddings for each item and tree nodes, the candidate items can be ranked based on their similarity to the conversation context encoded by the tree nodes. Thirdly, the diversity of items associated with each tree node allows us to develop an early stopping strategy to decide when to make recommendations. Fourthly, when the user rejects a recommendation, we adaptively choose the next decision tree to improve subsequent questions and recommendations. Extensive experiments on three publicly available benchmark CRS datasets show that our approach provides significant improvement to the state of the art CRS methods.
A Case Study in Educational Recommenders: Recommending Music Partitures at Tomplay
Authors: Ahmad Ajalloeian, Michalis Vlachos, Johannes Schneider, Alexis Steinmann
Venue: CIKM
Year: 2022

Recommendation technologies have been playing an instrumental role for promoting both physical and digital content across several global platforms (Amazon, Apple, Netflix). Here we provide a study on the benefits of recommendation technologies in an educational platform with a focus on music learning. There are several characteristics present in this educational platform that make this recommendation problem particularly interesting, namely: a) the few but highly repetitive interactions, b) the existence of multiple versions of the same content across many difficulty levels, orchestrations, and musical instruments, and c) the user's expertise in a musical instrument which is essential for making appropriate recommendations. We highlight the unique dataset characteristics and compare them to those of other widely-used recommendation datasets. To alleviate the very high data sparsity due to the multi-instantiation of songs, we use entity resolution principles to embed songs in a new space. Using this lightweight entity resolution step on song data, in combination with neural recommendation architectures, we can double the predictive accuracy compared to techniques based on matrix factorization.
Rank-Aware Gain-Based Evaluation of Extractive Summarization
Authors: Mousumi Akter
Venue: CIKM
Year: 2022

ROUGE has long been a popular metric for evaluating text summarization tasks as it eliminates time-consuming and costly human evaluations. However, ROUGE is not a fair evaluation metric for extractive summarization task as it is entirely based on lexical overlap. Additionally, ROUGE ignores the quality of the ranker for extractive summarization which performs the actual sentence/phrase extraction job. The main focus of the thesis is to design a nCG (normalized cumulative gain)-based evaluation metric for extractive summarization that is both rank-aware and semantic-aware (called Sem-nCG). One fundamental contribution of the work is that it demonstrates how we can generate more reliable semantic-aware ground truths for evaluating extractive summarization tasks without any additional human intervention. To the best of our knowledge, this work is the first of its kind. Preliminary experimental results demonstrate that the new Sem-nCG metric is indeed semantic-aware and also exhibits higher correlation with human judgement for single document summarization when single reference is considered.
A Different VIM: Visualizing Incremental Machine Learning
Authors: Sikder Tahsin Al-Amin, Mohammad Imtiaz Nur, Aisha Farooque, Guoning Chen, Robin Varghese, Carlos Ordonez
Venue: CIKM
Year: 2022

Incremental learning is used to continuously update and tune an existing model, as more data points come in. However, there is a tradeoff between speed and accuracy as the model becomes stable. From a big data angle, computing machine learning models is challenging when data sets cannot fit in main memory or when they exceed CPU capacity. On the other hand, data summarization is a fundamental technique that has promise to accelerate data science computations and compress a data set. Keeping these motivations in mind, we present an innovative system, VIM, that computes machine learning models in an incremental manner, visualizing continuous learning of model parameters as the data set is scanned. Our system is fast, it works for a wide spectrum of machine learning models and it can handle data sets larger than main memory. We developed an intuitive GUI which: (1) guides the user to upload data sets and choose a machine learning model, (2) offers interactive visualization of model parameters, and (3) helps getting an approximate model, stopping early, without reading the whole data set.
SERF: Interpretable Sleep Staging using Embeddings, Rules, and Features
Authors: Irfan Al-Hussaini, Cassie S. Mitchell
Venue: CIKM
Year: 2022

The accuracy of recent deep learning based clinical decision support systems is promising. However, lack of model interpretability remains an obstacle to widespread adoption of artificial intelligence in healthcare. Using sleep as a case study, we propose a generalizable method to combine clinical interpretability with high accuracy derived from black-box deep learning. Clinician-determined sleep stages from polysomnogram (PSG) remain the gold standard for evaluating sleep quality. However, PSG manual annotation by experts is expensive and time-prohibitive. We propose SERF, interpretable Sleep staging using Embeddings, Rules, and Features to read PSG. SERF provides interpretation of classified sleep stages through meaningful features derived from the AASM Manual for the Scoring of Sleep and Associated Events. In SERF, the embeddings obtained from a hybrid of convolutional and recurrent neural networks are transposed to the interpretable feature space. These representative interpretable features are used to train simple models like a shallow decision tree for classification. Model results are validated on two publicly available datasets. SERF surpasses the current state-of-the-art for interpretable sleep staging by 2%. Using Gradient Boosted Trees as the classifier, SERF obtains 0.766 κ and 0.870 AUC-ROC, within 2% of the current state-of-the-art black-box models.
Intent Disambiguation for Task-oriented Dialogue Systems
Authors: Andrea Alfieri, Ralf Wolter, Seyyed Hadi Hashemi
Venue: CIKM
Year: 2022

Task-Oriented Dialogue Systems (TODS) have been widely deployed for domain specific virtual assistants at contact centres to route customers' calls or deliver information needs of the customer in a conversational interaction. TODS employ natural language understanding components in order to map user commands to a set of pre-defined intents. However, Contact Centre users often fail to formulate their complex information needs in a single utterance which leads to formulating ambiguous user commands. This can negatively impact intent classification, and consequently customer satisfaction. To avoid feeding ambiguous user commands to the intent classifier of virtual assistants and help users in formulating their commands, we have implemented a solution that (1) identifies when a user is ambiguous and the virtual assistant should ask a clarification question, (2) disambiguates the user command and provides top-N most likely intents in a form of a clarification question. Our experimental result shows that our proposed intent disambiguation solution has a statistically significant improvement over a popularity based intent disambiguation model and an Intent Ranking Model of the Natural Language Understanding engine for a virtual assistant of a contact centre in terms of intent disambiguation accuracy and Mean Reciprocal Rank.
Improving Imitation Learning by Merging Experts Trajectories
Authors: Pegah Alizadeh, Aomar Osmani, Sammy Taleb
Venue: CIKM
Year: 2022

This paper proposes an original approach based on expert trajectories combination and Deep Reinforcement Learning to provide a better MineCraft player. The combination is based on the idea that the problem is naturally decomposable and the search space presents large plateaus. We use two steps approach to build a better trajectory from all existed expert trajectories and consequently to extract an optimal policy. The first step uses Birch clustering approach and images cosine similarity to obtain compact representation and substantial state and action space reduction. To reduce the overall complexity, the image distances are computed in images latent space trained by an encoder-decoder model. In the second step, we first eliminate plateaus to keep only the nodes with non-zero rewards then we compare trajectories using the Bellman equation and an appropriate value function. By checking the incremental compatibility of the trajectory of compact representations, we build the solution combining the best compatible sub-trajectories of the experts. The experimental results on NeurIPS MineRL 2020 challenge show that training the actors model on the most rewarding extracted subset of trajectories leads to achieve state-of-the-art performances on the MineCraft environment. The paper's source code is available here: https://github.com/thomJeffDoe/CompareTrajectories.
Generative Adversarial Zero-Shot Learning for Cold-Start News Recommendation
Authors: Manal A. Alshehri, Xiangliang Zhang
Venue: CIKM
Year: 2022

News recommendation models extremely rely on the interactive information between users and news articles to personalize the recommendation. Therefore, one of their most serious challenges is the cold-start problem (CSP). Their performance is dropped intensely for new users or new news. Zero-shot learning helps in synthesizing a virtual representation of the missing data in a variety of application tasks. Therefore, it can be a promising solution for CSP to generate virtual interaction behaviors for new users or new news articles. In this paper, we utilize the generative adversarial zero-shot learning in building a framework, namely, GAZRec, which is able to address the CSP caused by purely new users or new news. GAZRec can be flexibly applied to any neural news recommendation model. According to the experimental evaluations, applying the proposed framework to various news recommendation baselines attains a significant AUC improvement of 1% - 21% in different cold start scenarios and 1.2% - 6.6% in the regular situation when both users and news have a few interactions.
TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval
Authors: Sophia Althammer, Sebastian Hofstätter, Suzan Verberne, Allan Hanbury
Venue: CIKM
Year: 2022

Robust test collections are crucial for Information Retrieval research. Recently there is a growing interest in evaluating retrieval systems for domain-specific retrieval tasks, however these tasks often lack a reliable test collection with human-annotated relevance assessments following the Cranfield paradigm. In the medical domain, the TripClick collection was recently proposed, which contains click log data from the Trip search engine and includes two click-based test sets. However the clicks are biased to the retrieval model used, which remains unknown, and a previous study shows that the test sets have a low judgement coverage for the Top-10 results of lexical and neural retrieval models. In this paper we present the novel, relevance judgement test collection TripJudge for TripClick health retrieval. We collect relevance judgements in an annotation campaign and ensure the quality and reusability of TripJudge by a variety of ranking methods for pool creation, by multiple judgements per query-document pair and by an at least moderate inter-annotator agreement. We compare system evaluation with TripJudge and TripClick and find that that click and judgement-based evaluation can lead to substantially different system rankings.
LearnShapley: Learning to Predict Rankings of Facts Contribution Based on Query Logs
Authors: Dana Arad, Daniel Deutch, Nave Frost
Venue: CIKM
Year: 2022

To explain query results, a recent line of work has proposed to leverage the game-theoretic notion of Shapley values to quantify the contribution of each input fact to each result. Despite significant recent breakthroughs improving the complexity of computing Shapley values in query answering, the computation remains quite costly. To this end, we propose an approach that aims at ranking input facts based on their (hidden) Shapley values. Our method utilizes a repository of queries over the same database for which we do store exact Shapley values. Intuitively, some queries bear similarity in the ways they transform data, and consequently in the contribution of database facts to their outputs. In this manner, given a new query and a query result, we can learn and predict the ranking of contributing facts. Our contributions are three-fold. First, we introduce DBShap, a curated dataset of queries and query results, along with the contributing facts and respective Shapley values. Second, we define the task of predicting the ranking of facts contribution w.r.t a query and query result. Finally, we propose a solution for the prediction task based on BERT.
IEEE13-AdvAttack A Novel Dataset for Benchmarking the Power of Adversarial Attacks against Fault Prediction Systems in Smart Electrical Grid
Authors: Carmelo Ardito, Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, Fatemeh Nazary
Venue: CIKM
Year: 2022

Due to their economic and significant importance, fault detection tasks in intelligent electrical grids are vital. Although numerous smart grid (SG) applications, such as fault detection and load forecasting, have adopted data-driven approaches, the robustness and security of these data-driven algorithms have not been widely examined. One of the greatest obstacles in the research of the security of smart grids is the lack of publicly accessible datasets that permit testing the system's resilience against various types of assault. In this paper, we present IEEE13-AdvAttack, a large-scaled simulated dataset based on the IEEE-13 test node feeder suitable for supervised tasks under SG. The dataset includes both conventional and renewable energy resources. We examine the robustness of fault type classification and fault zone classification systems to adversarial attacks. Through the release of datasets, benchmarking, and assessment of smart grid failure prediction systems against adversarial assaults, we seek to encourage the implementation of machine-learned security models in the context of smart grids. The benchmarking data and code for fault prediction are made publicly available on https://bit.ly/3NT5jxG.
KRAF: A Flexible Advertising Framework using Knowledge Graph-Enriched Multi-Agent Reinforcement Learning
Authors: Jose A. Ayala-Romero, Péter Mernyei, Bichen Shi, Diego Mazón
Venue: CIKM
Year: 2022

Bidding optimization is one of the most important problems in online advertising. Auto-bidding tools are designed to address this problem and are offered by most advertising platforms for advertisers to allocate their budgets. In this work, we present a Knowledge Graph-enriched Multi-Agent Reinforcement Learning Advertising Framework (KRAF). It combines Knowledge Graph (KG) techniques with a Multi-Agent Reinforcement Learning (MARL) algorithm for bidding optimization with the goal of maximizing advertisers' return on ad spend (ROAS) and user-ad interactions, which correlates to the ad platform revenue. In addition, this proposal is flexible enough to support different levels of user privacy and the advent of new advertising markets with more heterogeneous data. In contrast to most of the current advertising platforms that are based on click-through rate models using a fixed input format and rely on user tracking, KRAF integrates the heterogeneous available data (e.g., contextual features, interest-based attributes, information about ads) as graph nodes to generate their dense representation (embeddings). Then, our MARL algorithm leverages the embeddings of the entities to learn efficient budget allocation strategies. To that end, we propose a novel coordination strategy based on a mean-field style to coordinate the learning agents and avoid the curse of dimensionality when the number of agents grows. Our proposal is evaluated on three real-world datasets to assess its performance and the contribution of each of its components, outperforming several baseline methods in terms of ROAS and number of ad clicks.
SoCRATe: A Recommendation System with Limited-Availability Items
Authors: Davide Azzalini, Fabio Azzalini, Chiara Criscuolo, Tommaso Dolci, Davide Martinenghi, Sihem Amer-Yahia
Venue: CIKM
Year: 2022

We demonstrate SoCRATe, an online system dedicated to providing adaptive recommendations to users when items have limited availability. SoCRATe is relevant to several real-world applications, among which movie and task recommendations. SoCRATe has several appealing features: (i) watching users as they consume recommendations and accounting for user feedback in refining recommendations in the next round; (ii) implementing loss compensation strategies to make up for sub-optimal recommendations, in terms of accuracy, when items have limited availability; (iii) deciding when to re-generate recommendations on a need-based fashion. SoCRATe accommodates real users as well as simulated users to enable testing multiple recommendation choice models. To frame evaluation, SoCRATe introduces a new set of measures that capture recommendation accuracy, user satisfaction and item consumption over time. All these features make SoCRATe unique and able to adapt recommendations to user preferences in a resource-limited setting. A video of SoCRATe is available at https://youtu.be/4wlaScc_rUo.
Improving Text-based Similar Product Recommendation for Dynamic Product Advertising at Yahoo
Authors: Xiao Bai, Lei Duan, Richard Tang, Gaurav Batra, Ritesh Agrawal
Venue: CIKM
Year: 2022

Retrieving similar products is a critical functionality required by many e-commerce websites as well as dynamic product advertising systems. Retargeting and Prospecting are two major forms of dynamic product advertising. Typically, after a user interacts with a product on an advertiser website (e.g., Macy's), when the user later visits a website (e.g., yahoo.com) supported by a dynamic product advertising system, the same product may be shown to the user as a Retargeting product ad, while some similar products may be shown to the user as Prospecting product ads on the web page. Similar products can enrich users' ad experience based on users' intent on the Prospecting product ads through which the users interacted. These product ads can also serve as substitutes when Retargeting ad candidates are out of stock. However, it is challenging to retrieve similar products among billions of products in a product catalog efficiently. Deep Siamese models allow efficient retrieval but do not put enough emphasize on key product attributes. To improve the quality of the similar products, we propose to first use a Siamese Transformer-based model to retrieve similar products and then refine them with the attribute "product name" that indicates the type of a product (e.g., running shoes, engagement ring, etc.) for post filtering. We propose a novel product name generation model that fine tunes a pre-trained Transformer-based language model with a sequence to sequence objective. To the best of our knowledge, this is the first work using a generative approach for identifying product attributes. We introduce two applications of the proposed approach for the dynamic product advertising system of Yahoo for Retargeting and Prospecting respectively. Offline evaluation and online A/B testing shows that the proposed approach retrieves high quality similar products, leading to an increase of ad clicks and ad revenue.
Statistical Claim Checking: StatCheck in Action
Authors: Oana Balalau, Simon Ebel, Théo Galizzi, Ioana Manolescu, Quentin Massonnat, Antoine Deiana, Emilie Gautreau, Antoine Krempf, Thomas Pontillon, Gérald Roux, Joanna Yakin
Venue: CIKM
Year: 2022

To strengthen public trust and counter disinformation, computational fact-checking, leveraging digital data sources, attracts interest from the journalists and the computer science community. A particular class of interesting data sources is statistics, that is, numerical data compiled mostly by governments, administrations, and international organizations. Statistics typically are multidimensional datasets, where multiple dimensions characterize one value, and the dimensions may be organized in a hierarchy. We developed StatCheck, a fact-checking system specialized in French. The technical novelty of StatCheck is twofold: (i) we focus on multidimensional, complex-structure statistics, which have received little attention so far, despite their practical importance; and (ii) novel statistical claim extraction modules for French, an area where few resources exist. We will demonstrate our system on large statistic datasets (hundreds of millions of facts), including the complete INSEE (French) and Eurostat (European Union) datasets. More information about StatCheckis available online at: https://team.inria.fr/cedar/projects/statcheck/.
Abstra: Toward Generic Abstractions for Data of Any Model
Authors: Nelly Barret, Ioana Manolescu, Prajna Upadhyay
Venue: CIKM
Year: 2022

Digital data sharing leads to unprecedented opportunities to develop data-driven systems for supporting economic activities, the social and political life, and science. Many open-access datasets are RDF (Linked Data) graphs, but others are JSON or XML documents, CSV files, Neo4J property graphs, etc. Potential users need to understand a dataset in order to decide if it is useful for their goal. While some published datasets come with a schema and/or documentation, this is not always the case. We demonstrate Abstra, a dataset abstraction system, which applies on a large variety of data models. Abstra computes a description meant for humans, and integrates Information Extraction to classify dataset content among a set of categories of interest to the user. Our abstractions are conceptually close to Entity-Relationship diagrams, but our entities can have deeply nested structure.