OpenRaiser/Intern-Atlas huggingface dataset:
Harnessing GPT for Topic-Based Call Segmentation in Microsoft Dynamics 365 Sales
Authors: Itzik Malkiel, Uri Alon, Yakir Yehuda, Shahar Keren, Oren Barkan, Royi Ronen, Noam Koenigstein
Venue: CIKM
Year: 2023

Transcriptions of phone calls hold significant value in sales, customer service, healthcare, law enforcement, and more. However, analyzing recorded conversations can be a time-consuming process, especially for complex dialogues. In Microsoft Dynamics 365 Sales, a novel system, named GPT-Calls, is applied for efficient and accurate topic-based call segmentation. GPT-Calls comprises offline and online phases. In the offline phase, the system leverages a GPT model to generate synthetic sentences and extract anchor vectors for predefined topics. This phase, performed once on a given topic list, significantly reduces the computational burden. The online phase scores the similarity between the transcribed conversation and the topic anchors from the offline phase, followed by time domain analysis to group utterances into segments and tag them with topics. The GPT-Calls scheme offers an accurate and efficient approach to call segmentation and topic extraction, eliminating the need for labeled data. It is a versatile solution applicable to various industry domains. GPT-Calls operates in production under Dynamics 365 Sales Conversation Intelligence, applied to real sales conversations from diverse Dynamics 365 Sales tenants, streamlining call analysis, and saving time and resources while ensuring accuracy and effectiveness.
Learning Visibility Attention Graph Representation for Time Series Forecasting
Authors: Shengzhong Mao, Xiao-Jun Zeng
Venue: CIKM
Year: 2023

Visibility algorithm acts as a mapping that bridges graph representation learning with time series analysis, which has been broadly investigated for forecasting tasks. However, the intrinsic nature of visibility encoding yields graphs structured exclusively by binary adjacency matrix, leading to inevitable information loss of temporal sequence during the mapping. To this end, we introduce Angular Visibility Graph Networks (AVGNets), designed with two core features: (i) The framework reconstructs weighted graphs to encode time series by leveraging topological insights derived from visual angles of visibility networks, which capture sequential and structural information within weighted angular matrix. (ii) ProbAttention module is proposed for evaluating probabilistic attention of weighted networks, with remarkable capabilities to extract intrinsic and extrinsic temporal dependencies across multi-layer graphs. Extensive experiments and ablation studies on real-world datasets covering diverse ranges demonstrate that AVGNets achieve state-of-the-art performance, offering an innovative perspective on graph representation for sequence modeling.
LambdaRank Gradients are Incoherent
Authors: Federico Marcuzzi, Claudio Lucchese, Salvatore Orlando
Venue: CIKM
Year: 2023

In Information Retrieval (IR), the Learning-to-Rank (LTR) task requires building a ranking model that optimises a specific IR metric. One of the most effective approaches to do so is the well-known LambdaRank algorithm. LambdaRank uses gradient descent optimisation, and at its core, it defines approximate gradients, the so-called lambdas, for a non-differentiable IR metric. Intuitively, each lambda describes how much a document's score should be "pushed" up/down to reduce the ranking error. In this work, we show that lambdas may be incoherent w.r.t. the metric being optimised: e.g., a document with high relevance in the ground truth may receive a smaller gradient push than a document with lower relevance. This behaviour goes far beyond the expected degree of approximation. We analyse such behaviour of LambdaRank gradients and we introduce some strategies to reduce their incoherencies. We demonstrate through extensive experiments, conducted using publicly available datasets, that the proposed approach reduces the frequency of the incoherencies in LambdaRank and derivatives, and leads to models that achieve statistically significant improvements in the NDCG metric, without compromising the training efficiency.
On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents
Authors: Jordan Massiah, Emine Yilmaz, Yunlong Jiao, Gabriella Kazai
Venue: CIKM
Year: 2023

We analyse the reliability of users' explicit feedback for evaluating the quality of conversational agents. Using data from a commercial conversational system, we analyse how user feedback compares with human annotations; how well it aligns with implicit user satisfaction signals, such as retention; and how much user feedback is needed to reliably evaluate the quality of a conversational system.
A Robust Backward Compatibility Metric for Model Retraining
Authors: Ryuta Matsuno, Keita Sakuma
Venue: CIKM
Year: 2023

Model retraining and updating are essential processes in AI applications. However, during updates, there is a potential for performance degradation, in which the overall performance improves, but local performance deteriorates. This study proposes a backward compatibility metric that focuses on the compatibility of local predictive performance. The score of the proposed metric increases if the accuracy over the conditional distribution for each input is higher than before. Furthermore, we propose a model retraining method based on the proposed metric. Due to the use of the conditional distribution, our metric and retraining method are robust against label noises, while existing sample-based backward compatibility metrics are often affected by noise. We perform a theoretical analysis of our method and derive an upper bound for the generalization error. Numerical experiments demonstrate that our retraining method enhances compatibility while achieving equal or better trade-offs in overall performance compared to existing methods.
A Graph Neural Network Model for Concept Prerequisite Relation Extraction
Authors: Debjani Mazumder, Jiaul H. Paik, Anupam Basu
Venue: CIKM
Year: 2023

In recent years, with the emergence of online learning platforms and e-learning resources, many documents are available for a particular topic. For a better learning experience, the learner often needs to know and learn first the prerequisite concepts for a given concept. Traditionally, the identification of such prerequisite concepts is done manually by subject experts, which in turn, often limits self-paced learning. Recently, machine learning models have found encouraging success for the task, obviating manual effort. In this paper, we propose a graph neural network based approach that leverages node attention over a heterogeneous graph to extract the prerequisite concepts for a given concept. Experiments on a set of benchmark data show that the proposed model outperforms the existing models by large margins almost always, making the model a new state-of-the-art for the task.
Graph Contrastive Learning with Graph Info-Min
Authors: En Meng, Yong Liu
Venue: CIKM
Year: 2023

The complexity of the graph structure poses a challenge for graph representation learning. Contrastive learning offers a straightforward and efficient unsupervised framework for graph representation learning. It achieves unsupervised learning by augmenting the original views and comparing them with the augmented views. Several methods based on this framework have achieved significant progress in the field of graph representation learning. Despite its success, the factors contributing to good augmented views in graph contrast learning have received less attention. In order to address this issue, we introduce the graph info-min principle. We investigate the relationship between mutual information (MI) and good augmented views through experimental and theoretical analysis. Additionally, we present a new contrastive learning method called Info-min Contrastive Learning (IMCL). Specifically, The method comprises an adaptive graph augmentation generator and a pseudo-label generator. The graph augmentation generator ensures sufficient differentiation between the augmented and original views. The pseudo-label generator generates pseudo-labels as supervision signals, ensuring consistency between the classification results of augmented views and original views. Our method demonstrates excellent performance through extensive experimental results on various datasets.
Disparity, Inequality, and Accuracy Tradeoffs in Graph Neural Networks for Node Classification
Authors: Arpit Merchant, Carlos Castillo
Venue: CIKM
Year: 2023

Graph neural networks (GNNs) are increasingly used in critical human applications for predicting node labels in attributed graphs. Their ability to aggregate features from nodes' neighbors for accurate classification also has the capacity to exacerbate existing biases in data or to introduce new ones towards members from protected demographic groups. Thus, it is imperative to quantify how GNNs may be biased and to what extent their harmful effects may be mitigated. To this end, we propose two new GNN-agnostic interventions namely, (i) PFR-AX which decreases the separability between nodes in protected and non-protected groups, and (ii) PostProcess which updates model predictions based on a blackbox policy to minimize differences between error rates across demographic groups. Through a large set of experiments on four datasets, we frame the efficacies of our approaches (and three variants) in terms of their algorithmic fairness-accuracy tradeoff and bench- mark our results against three strong baseline interventions on three state-of-the-art GNN models. Our results show that no single intervention offers a universally optimal tradeoff, but PFR-AX and PostProcess provide granular control and improve model confidence when correctly predicting positive outcomes for nodes in protected groups.
Astrolabe: Visual Graph Database Queries with Tabular Output
Authors: Michael Miller
Venue: CIKM
Year: 2023

Graph databases are an established solution for large, highly connected datasets. One challenge associated with deploying graph databases in industrial settings is usability. Typically, developers interact with graph databases through queries in languages such as Cypher or GraphQL. Many end-users, analysts, and administrators are not familiar with these specialized languages. Additionally, these queries return hierarchical data in formats such as JSON (JavaScript Object Notation) or XML (Extensible Markup Language). Additional scripts and interfaces are needed to convert hierarchical data into more easily digested tables. To overcome these challenges, each graph database use-case typically involves significant custom software to explore, view, and export data. We introduce Astrolabe, a generalized interface that addresses the challenges of querying graph databases. In Astrolabe, queries are constructed visually, so users do not need to learn new graph query languages. Results are returned as tables, which can be easily digested by end users or down-stream applications. Astrolabe was designed to function with arbitrary graph databases, so schema definition is not required. Astrolabe revolutionizes graph exploration and querying by allowing graph databases to be viewed as tables, without the need for custom software adapters.
SeqGen: A Sequence Generator via User Side Information for Behavior Sparsity in Recommendation
Authors: Xu Min, Xiaolu Zhang, Bin Shen, Shuhan Wang, Yong He, Changsheng Li, Jun Zhou
Venue: CIKM
Year: 2023

In real-world industrial advertising systems, user behavior sparsity is a key issue that affects online recommendation performance. We observe that users with rich behaviors can obtain better recommendation results than those with sparse behaviors in a conversion-rate (CVR) prediction model. Inspired by this phenomenon, we propose a new method SeqGen, in an effort to exploit user side information to bridge the gap between rich and sparse behaviors. SeqGen is a learnable and pluggable module, which can be easily integrated into any CVR model and no longer requires two-stage training as in previous works. In particular, SeqGen learns a mapping relationship between the user side information and behavior sequences, only on the basis of the users with long behavior sequences. After that, SeqGen can generate rich sequence features for users with sparse behaviors based on their side information, so as to alleviate the issue of user behavior sparsity. The generated sequence features will then be fed into the classifier tower of an arbitrary CVR model together with the original sequence features. To the best of our knowledge, our approach constitutes the first attempt to exploit user side information for addressing the user behavior sparsity issue. We validate the effectiveness of SeqGen on the publicly available dataset MovieLens-1M, and our method receives an improvement of up to 0.5% in terms of the AUC score. More importantly, we successfully deploy SeqGen in the commercial advertising system Xlight of Alipay, which improves the grouped AUC of the CVR model by 0.6% and brings a boost of 0.49% in terms of the conversion rate on A/B testing.
Info-Wild: Knowledge Extraction and Management for Wildlife Conservation
Authors: Prasenjit Mitra, Shreya Ghosh, Bistra Dilkina, Thomas Müller
Venue: CIKM
Year: 2023

Our primary objective is to explore and enhance AI's role for wildlife conservation, in brief, Nature Through the Lens of AI. It seeks to address crucial challenges related to data heterogeneity, scale integration, data privacy, mitigating biases, and decision-making under uncertainty. This workshop is centred around leveraging AI's prowess in deciphering complex spatio-temporal data patterns for wildlife conservation, thereby contributing significantly to the broader canvas of AI for social good. The workshop intends to create an interdisciplinary platform bringing together computer scientists, data scientists, geospatial experts, ecologists, and conservation practitioners, fostering collaboration and driving real-world impact. The program will include keynote speeches, panel discussions, and interactive sessions focusing on efficient knowledge extraction and management, remote sensing technologies, predictive modeling, species distribution modeling, habitat quality assessment, and human-wildlife conflict mitigation. With an em- phasis on CIKM's primary interests, our aim is not only to enrich understanding of AI's symbiotic potential with ecology but also to utilize it to address pressing societal and environmental challenges.
Unified Generative & Dense Retrieval for Query Rewriting in Sponsored Search
Authors: Akash Kumar Mohankumar, Bhargav Dodla, Gururaj K, Amit Singh
Venue: CIKM
Year: 2023

Sponsored search is a key revenue source for search engines, where advertisers bid on keywords to target users or search queries of interest. However, finding relevant keywords for a given query is challenging due to the large and dynamic keyword space, ambiguous user/advertiser intents, and diverse possible topics and languages. In this work, we present a comprehensive comparison between two paradigms for online query rewriting: Generative (NLG) and Dense Retrieval (DR) methods. We observe that both methods offer complementary benefits that are additive. As a result, we show that around 40% of the high-quality keywords retrieved by the two approaches are unique and not retrieved by the other. To leverage the strengths of both methods, we propose CLOVER-Unity, a novel approach that unifies generative and dense retrieval methods in one single model. Through offline experiments, we show that the NLG and DR components of CLOVER-Unity consistently outperform individually trained NLG and DR models on public and internal benchmarks. Furthermore, we show that CLOVER-Unity achieves 9.8% higher good keyword density than the ensemble of two separate DR and NLG models while reducing computational costs by almost half. We conduct extensive online A/B experiments on Microsoft Bing in 140+ countries and achieve improved user engagement, with an average increase in total clicks by 0.89% and increased revenue by 1.27%. We also share our practical lessons and optimization tricks for deploying such unified models in production.
Exploiting Sequential Music Preferences via Optimisation-Based Sequencing
Authors: Dmitrii Moor, Yi Yuan, Rishabh Mehrotra, Zhenwen Dai, Mounia Lalmas
Venue: CIKM
Year: 2023

Users in music streaming platforms typically consume tracks sequentially in sessions by interacting with personalised playlists. To satisfy users, music platforms usually rely on recommender systems that learn users' preferences over individual tracks and rank the tracks within each playlist according to the learned preferences. However, such rankings often do not fully exploit the sequential nature of the users' consumption, which may result in a lower within-a-session consumption. In this paper, we model the sequential within-a-session preferences of users and propose an optimisation-based sequencing approach that allows for optimally incorporating such preferences into the rankings. To this end, we rely on interaction data of a major music streaming service to identify two most common aspects of the users' sequential preferences: (1) Position-Aware preferences, and (2) Local-Sequential preferences. We propose a sequencing model that can leverage each of these aspects optimally to maximise the expected total consumption from the session. We further perform an extensive offline and off-policy evaluation of our model, and carry out a large scale online randomised control trial with 7M users across 80 countries. Our findings confirm that we can effectively incorporate sequential preferences of users into our sequencer to make users complete more and skip less tracks within their listening sessions.
ThyExp: An explainable AI-assisted Decision Making Toolkit for Thyroid Nodule Diagnosis based on Ultra-sound Images
Authors: Jamie Morris, Zehao Liu, Huizhi Liang, Sidhartha Nagala, Xia Hong
Venue: CIKM
Year: 2023

Radiologists have an important task of diagnosing thyroid nodules present in ultra sound images. Although reporting systems exist to aid in the diagnosis process, these systems do not provide explanations about the diagnosis results. We present ThyExp -- a web based toolkit for it use by medical professionals, allowing for accurate diagnosis with explanations of thyroid nodules present in ultrasound images utilising artificial intelligence models. The proposed web-based toolkit can be easily incorporated into current medical workflows, and allows medical professionals to have the confidence of a highly accurate machine learning model with explanations to provide supplementary diagnosis data. The solution provides classification results with their probability accuracy, as well as the explanations in the form of presenting the key features or characteristics that contribute to the classification results. The experiments conducted on a real-world UK NHS hospital patient dataset demonstrate the effectiveness of the proposed approach. This toolkit can improve the trust of medical professional to understand the confidence of the model in its predictions. This toolkit can improve the trust of medical professionals in understanding the models reasoning behind its predictions.
A Flash Attention Transformer for Multi-Behaviour Recommendation
Authors: Tendai Mukande, Esraa Ali, Annalina Caputo, Ruihai Dong, Noel E. O&apos, Connor
Venue: CIKM
Year: 2023

\beginabstract Recently, modelling heterogeneous interactions in recommender systems has attracted research interest. Real-world scenarios involve sequential multi-type user-item interactions such as ''shape view'', ''shape add-to-favourites'', ''shape add-to-cart'' and ''shape purchase''. Graph Neural Network (GNN) methods have been widely adopted in Representation Learning of similar sequential user-item interactions. Promising results have been achieved by the integration of GNNs and transformers for self-attention. However, GNN based methods suffer from limited capability in handling global user-item interaction dependencies, particularly for long sequences. Moreover, these models require high computational cost of transformers, due to the quadratic memory and time complexity with respect to sequence length. This results in memory bottlenecks and slow training especially in computational resource-constrained environments. To address these challenges, we propose the FATH model which employs Flash Attention mechanism to reduce the high-bandwidth memory usage over higher-order user-item interaction sequences. Experimental results show that our model improves the training speed and reduces the memory usage with better recommendation performance in comparison with the state-of the art baselines.
Unlocking the Potential of Non-PSD Kernel Matrices: A Polar Decomposition-based Transformation for Improved Prediction Models
Authors: Maximilian Münch, Manuel Röder, Frank-Michael Schleif
Venue: CIKM
Year: 2023

Kernel functions are a key element in many machine learning methods to capture the similarity between data points. However, a considerable number of these functions do not meet all mathematical requirements to be a valid positive semi-definite kernel, a crucial precondition for kernel-based classifiers such as Support Vector Machines or Kernel Fisher Discriminant classifiers. In this paper, we propose a novel strategy employing a polar decomposition to effectively transform invalid kernel matrices to positive semi-definite matrices, while preserving the topological structure inherent to the data points. Utilizing polar decomposition allows the effective transformation of indefinite kernel matrices from Krein space to positive semi-definite matrices in Hilbert space, thereby providing an efficient out-of-sample extension for new unseen data and enhancing kernel method applicability across diverse classification tasks. We evaluate our approach on a variety of benchmark datasets and demonstrate its superiority over competitive methods.
Joint Link Prediction Via Inference from a Model
Authors: Parmis Naddaf, Erfaneh Mahmoudzaheh Ahmadi Nejad, Kiarash Zahirnia, Manfred Jaeger, Oliver Schulte
Venue: CIKM
Year: 2023

A Joint Link Prediction Query (JLPQ) specifies a set of links to be predicted, given another set of links as well as node attributes as evidence. While single link prediction has been well studied in literature on deep graph learning, predicting multiple links together has gained little attention. This paper presents a novel framework for computing JLPQs using a probabilistic deep Graph Generative Model. Specifically, we develop inference procedures for an inductively trained Variational Graph Auto-Encoder (VGAE) that estimates the joint link probability for any input JLPQ, without retraining. For evaluation, we apply inference to a range of joint link prediction queries on six benchmark datasets. We find that for most datasets and query types, joint link prediction via inference from a model achieves good predictive performance, better than the independent link prediction baselines (by 0.02-0.4 AUC points depending on the dataset).
Efficient Differencing of System-level Provenance Graphs
Authors: Yuta Nakamura, Iyad Kanj, Tanu Malik
Venue: CIKM
Year: 2023

Data provenance, when audited at the operating system level, generates a large volume of low-level events. Current provenance systems infer causal flow from these event traces, but do not infer application structure, such as loops and branches. The absence of these inferred structures decreases accuracy when comparing two event traces, leading to low-quality answers from a provenance system. In this paper, we infer nested natural and unnatural loop structures over a collection of provenance event traces. We describe an 'unrolling method' that uses the inferred nested loop structure to systematically mark loop iterations. Our loop-based unrolling improves the accuracy of trace comparison by 20-70% over trace comparisons that do not rely on inferred structures.