OpenRaiser/Intern-Atlas huggingface dataset:
Intelligible and explainable machine learning: best practices and practical challenges
Authors: Rich Caruana, Scott M. Lundberg, Marco Túlio Ribeiro, Harsha Nori, Samuel Jenkins
Venue: Conference KDD 2020
Year: 2020

Learning methods such as boosting and deep learning have made ML models harder to understand and interpret. This puts data scientists and ML developers in the position of often having to make a tradeoff between accuracy and intelligibility. Research in IML (Interpretable Machine Learning) and XAI (Explainable AI) focus on minimizing this trade-off by developing more accurate interpretable models and by developing new techniques to explain black-box models. Such models and techniques make it easier for data scientists, engineers and model users to debug models and achieve important objectives such as ensuring the fairness of ML decisions and the reliability and safety of AI systems. In this tutorial, we present an overview of various interpretability methods and provide a framework for thinking about how to choose the right explanation method for different real-world scenarios. We will focus on the application of XAI in practice through a variety of case studies from domains such as healthcare, finance, and bias and fairness. Finally, we will present open problems and research directions for the data mining and machine learning community. What audience will learn: When and how to use a variety of machine learning interpretability methods through case studies of real-world situations. The difference between glass-box and black-box explanation methods and when to use them. How to use open source interpretability toolkits that are now available
Follow the prophet: Accurate online conversion rate prediction in the face of delayed feedback
Authors: Haoming Li 0011, Feiyang Pan, Xiang Ao 0001, Zhao Yang, Min Lu, Junwei Pan, Dapeng Liu, Lei Xiao 0001, Qing He 0003
Venue: Conference SIGIR 2021
Year: 2021

The delayed feedback problem is one of the imperative challenges in online advertising, which is caused by the highly diversified feedback delay of a conversion varying from a few minutes to several days. It is hard to design an appropriate online learning system under these non-identical delay for different types of ads and users. In this paper, we propose to tackle the delayed feedback problem in online advertising by "Following the Prophet" (FTP for short). The key insight is that, if the feedback came instantly for all the logged samples, we could get a model without delayed feedback, namely the "prophet". Although the prophet cannot be obtained during online learning, we show that we could predict the prophet's predictions by an aggregation policy on top of a set of multi-task predictions, where each task captures the feedback patterns of different periods. We propose the objective and optimization approach for the policy, and use the logged data to imitate the prophet. Extensive experiments on three real-world advertising datasets show that our method outperforms the previous state-of-the-art baselines.
Finding meta winning ticket to train your maml
Authors: Dawei Gao, Yuexiang Xie, Zimu Zhou, Zhen Wang 0036, Yaliang Li, Bolin Ding
Venue: Conference KDD 2022
Year: 2022

The lottery ticket hypothesis (LTH) states that a randomly initialized dense network contains sub-networks that can be trained in isolation to the performance of the dense network. In this paper, to achieve rapid learning with less computational cost, we explore LTH in the context of meta learning. First, we experimentally show that there are sparse sub-networks, known as meta winning tickets, which can be meta-trained to few-shot classification accuracy to the original backbone. The application of LTH in meta learning enables the adaptation of meta-trained networks on various IoT devices with fewer computation. However, the status quo to identify winning tickets requires iterative training and pruning, which is particularly expensive for finding meta winning tickets. To this end, then we investigate the inter- and intra-layer patterns among different meta winning tickets, and propose a scheme for early detection of a meta winning ticket. The proposed scheme enables efficient training in resource-limited devices. Besides, it also designs a lightweight solution to search the meta winning ticket. Evaluations on standard few-shot classification benchmarks show that we can find competitive meta winning tickets with 20% weights of the original backbone, while incurring only 8%-14% (Conv-4) and 19%-29% (ResNet-12) computation overhead (measured by FLOPs) of the standard winning ticket finding scheme.
Neural relational topic models for scientific article analysis
Authors: Haoli Bai, Zhuangbin Chen, Michael R. Lyu, Irwin King, Zenglin Xu
Venue: Conference CIKM 2018
Year: 2018

Topic modelling and citation recommendation of scientific articles are important yet challenging research problems in scientific article analysis. In particular, the inference on coherent topics can be easily affected by irrelevant contents in articles. Meanwhile, the extreme sparsity of citation networks brings difficulty to a valid citation recommendation. Intuitively, articles with similar topics are more likely to cite each other, and cited articles tend to share similar themes. Motivated from this intuition, we aim to boost the performance of both topic modelling and citation recommendation by effectively leverage this underlying correlation between latent topics and citation networks. To this end, we propose a novel Bayesian deep generative model termed as Neural Relational Topic Model (NRTM), which is composed with a Stacked Variational Auto-Encoder (SVAE) and a multilayer perception (MLP). Specifically, the SVAE utilizes an inference network to learn more representative topics of document contents, which can help to enrich the latent factors in collaborative filtering of citations. Furthermore, the MLP network conducts nonlinear collaborative filtering of citations, which can further benefit the inference of topics by leveraging the knowledge of citation networks. Extensive experiments on two real-world datasets demonstrate that our model can effectively take advantages of the coherence between topic learning and citation recommendation, and significantly outperform the state-of-the-art methods on both tasks.
Patric: A parallel algorithm for counting triangles in massive networks
Authors: Shaikh Arifuzzaman, Maleq Khan, Madhav V. Marathe
Venue: Conference CIKM 2013
Year: 2013

Massive networks arising in numerous application areas poses significant challenges for network analysts as these networks grow to billions of nodes and are prohibitively large to fit in the main memory. Finding the number of triangles in a network is an important problem in the analysis of complex networks. Several interesting graph mining applications depend on the number of triangles in the graph. In this paper, we present an efficient MPI-based distributed memory parallel algorithm, called PATRIC, for counting triangles in massive networks. PATRIC scales well to networks with billions of nodes and can compute the exact number of triangles in a network with one billion nodes and 10 billion edges in 16 minutes. Balancing computational loads among processors for a graph problem like counting triangles is a challenging issue. We present and analyze several schemes for balancing load among processors for the triangle counting problem. These schemes achieve very good load balancing. We also show how our parallel algorithm can adapt an existing edge sparsification technique to approximate the number of triangles with very high accuracy. This modification allows us to count triangles in even larger networks.
Expscore: Learning metrics for recommendation explanation
Authors: Bingbing Wen, Yunhe Feng, Yongfeng Zhang, Chirag Shah
Venue: Conference WWW 2022
Year: 2022

Many information access and machine learning systems, including recommender systems, lack transparency and accountability. High-quality recommendation explanations are of great significance to enhance the transparency and interpretability of such systems. However, evaluating the quality of recommendation explanations is still challenging due to the lack of human-annotated data and benchmarks. In this paper, we present a large explanation dataset named RecoExp, which contains thousands of crowdsourced ratings of perceived quality in explaining recommendations. To measure explainability in a comprehensive and interpretable manner, we propose ExpScore, a novel machine learning-based metric that incorporates the definition of explainability from various perspectives (e.g., relevance, readability, subjectivity, and sentiment polarity). Experiments demonstrate that ExpScore not only vastly outperforms existing metrics and but also keeps itself explainable. Both the RecoExp dataset and open-source implementation of ExpScore will be released for the whole community. These resources and our findings can serve as forces of public good for scholars as well as recommender systems users.
Algorithms for hiring and outsourcing in the online labor market
Authors: Aris Anagnostopoulos, Carlos Castillo, Adriano Fazzone, Stefano Leonardi, Evimaria Terzi
Venue: Conference KDD 2018
Year: 2018

Although freelancing work has grown substantially in recent years, in part facilitated by a number of online labor marketplaces, %(e.g., Guru, Freelancer, Amazon Mechanical Turk), traditional forms of "in-sourcing" work continue being the dominant form of employment. % in most companies. This means that, at least for the time being, freelancing and salaried employment will continue to co-exist. In this paper, we provide algorithms for outsourcing and hiring workers in a general setting, where workers form a team and contribute different skills to perform a task. We call this model team formation with outsourcing. In our model, tasks arrive in an online fashion: neither the number nor the composition of the tasks are known a-priori. At any point in time, there is a team of hired workers who receive a fixed salary independently of the work they perform. This team is dynamic: new members can be hired and existing members can be fired, at some cost. Additionally, some parts of the arriving tasks can be outsourced and thus completed by non-team members, at a premium. Our contribution is an efficient online cost-minimizing algorithm for hiring and firing team members and outsourcing tasks. We present theoretical bounds obtained using a primal--dual scheme proving that our algorithms have logarithmic competitive approximation ratio. We complement these results with experiments using semi-synthetic datasets based on actual task requirements and worker skills from three large online labor marketplaces.
Scalable first-order Bayesian optimization via structured automatic differentiation
Authors: Sebastian E. Ament, Carla P. Gomes
Venue: Conference ICML 2022
Year: 2022

Bayesian Optimization (BO) has shown great promise for the global optimization of functions that are expensive to evaluate, but despite many successes, standard approaches can struggle in high dimensions. To improve the performance of BO, prior work suggested incorporating gradient information into a Gaussian process surrogate of the objective, giving rise to kernel matrices of size $nd \times nd$ for $n$ observations in $d$ dimensions. Naïvely multiplying with (resp. inverting) these matrices requires $\mathcal{O}(n^2d^2)$ (resp. $\mathcal{O}(n^3d^3$)) operations, which becomes infeasible for moderate dimensions and sample sizes. Here, we observe that a wide range of kernels gives rise to structured matrices, enabling an exact $\mathcal{O}(n^2d)$ matrix-vector multiply for gradient observations and $\mathcal{O}(n^2d^2)$ for Hessian observations. Beyond canonical kernel classes, we derive a programmatic approach to leveraging this type of structure for transformations and combinations of the discussed kernel classes, which constitutes a structure-aware automatic differentiation algorithm. Our methods apply to virtually all canonical kernels and automatically extend to complex kernels, like the neural network, radial basis function network, and spectral mixture kernels without any additional derivations, enabling flexible, problem-dependent modeling while scaling first-order BO to high $d$.
How to win friends and influence people, truthfully: influence maximization mechanisms for social networks
Authors: Yaron Singer
Venue: Conference WSDM 2012
Year: 2012

Throughout the past decade there has been extensive research on algorithmic and data mining techniques for solving the problem of influence maximization in social networks: if one can incentivize a subset of individuals to become early adopters of a new technology, which subset should be selected so that the word-of-mouth effect in the social network is maximized? Despite the progress in modeling and techniques, the incomplete information aspect of the problem has been largely overlooked. While data can often provide the network structure and influence patterns may be observable, the inherent cost individuals have to become early adopters is difficult to extract.
Visualization of navigation patterns on a web site using model-based clustering
Authors: Igor V. Cadez, David Heckerman, Christopher Meek, Padhraic Smyth, Steven White
Venue: Conference KDD 2000
Year: 2000

We present a new methodology for visualizing navigation patterns on a Web site. In our approach, we rst partition site users into clusters such that only users with similar navigation paths through the site are placed into the same cluster. Then, for each cluster, we display these paths for users within that cluster. The clustering approach we employ is model based (as opposed to distance based) and partitions users according to the order in which they request Web pages. In particular, we cluster users by learning a mixture of rst-order Markov models using the ExpectationMaximization algorithm. Our algorithm scales linearly with both number of users and number of clusters, and our implementation easily handles millions of users and thousands of clusters in memory. In the paper, we describe the details of our technology and a tool based on it called WebCANVAS. We illustrate the use of our technology on user-traAEc data from msnbc.com.
On smoothed explanations: Quality and robustness
Authors: Ahmad Ajalloeian, Seyed-Mohsen Moosavi-Dezfooli, Michalis Vlachos, Pascal Frossard
Venue: Conference CIKM 2022
Year: 2022

Explanation methods highlight the importance of the input features in taking a predictive decision, and represent a solution to increase the transparency and trustworthiness in machine learning and deep neural networks (DNNs). However, explanation methods can be easily manipulated generating misleading explanations particularly under visually imperceptible adversarial perturbations. Recent work has identified the decision surface geometry of DNNs as the main cause of this phenomenon. To make explanation methods more robust against adversarially crafted perturbations, recent research has promoted several smoothing approaches. These approaches smooth either the explanation map or the decision surface. In this work, we initiate a very thorough evaluation of the quality and robustness of the explanations offered by smoothing approaches. Different properties are evaluated. We present settings in which the smoothed explanations are both better, and worse, than the explanations derived by the commonly-used (non-smoothed) Gradient explanation method. By making the connection with the literature on adversarial attacks, we demonstrate that such smoothed explanations are robust primarily against additive attacks. However, a combination of additive and non-additive attacks can still manipulate these explanations, revealing important shortcomings in their robustness properties.
Adversarial vulnerability of randomized ensembles
Authors: Hassan Dbouk, Naresh R. Shanbhag
Venue: Conference ICML 2022
Year: 2022

Despite the tremendous success of deep neural networks across various tasks, their vulnerability to imperceptible adversarial perturbations has hindered their deployment in the real world. Recently, works on randomized ensembles have empirically demonstrated significant improvements in adversarial robustness over standard adversarially trained (AT) models with minimal computational overhead, making them a promising solution for safety-critical resource-constrained applications. However, this impressive performance raises the question: Are these robustness gains provided by randomized ensembles real? In this work we address this question both theoretically and empirically. We first establish theoretically that commonly employed robustness evaluation methods such as adaptive PGD provide a false sense of security in this setting. Subsequently, we propose a theoretically-sound and efficient adversarial attack algorithm (ARC) capable of compromising random ensembles even in cases where adaptive PGD fails to do so. We conduct comprehensive experiments across a variety of network architectures, training schemes, datasets, and norms to support our claims, and empirically establish that randomized ensembles are in fact more vulnerable to $\ell_p$-bounded adversarial perturbations than even standard AT models. Our code can be found at https://github.com/hsndbk4/ARC.
Metarule: A meta-path guided ensemble rule set learning for explainable fraud detection
Authors: Lu Yu, Meng Li, Xiaoguang Huang, Wei Zhu, Yanming Fang, Jun Zhou, Longfei Li
Venue: Conference CIKM 2022
Year: 2022

Machine learning methods for fraud detection have achieved impressive prediction performance, but often sacrifice critical interpretability in many applications. In this work, we propose to learn interpretable models for fraud detection as a simple rule set. More specifically, we design a novel neural rule learning method by building a condition graph with an expectation to capture the high-order feature interactions. Each path in this condition graph can be regarded as a single rule. Inspired by the key idea of meta learning, we combine the neural rules with rules extracted from the tree-based models in order to provide generalizable rule candidates. Finally, we propose a flexible rule set learning framework by designing a greedy optimization method towards maximizing the recall number of fraud samples with a predefined criterion as the cost. We conduct comprehensive experiments on large-scale industrial datasets. Interestingly, we find that the neural rules and rules extracted from tree-based models can be complementary to each other to improve the prediction performance.
Finrule: Feature interactive neural rule learning
Authors: Lu Yu, Meng Li, Ya-Lin Zhang, Longfei Li, Jun Zhou
Venue: Conference CIKM 2023
Year: 2023

Though neural networks have achieved impressive prediction performance, it's still hard for people to understand what neural networks have learned from the data. The black-box property of neural networks already becomes one of the main obstacles preventing from being applied to many high-stakes applications, such as finance and medicine that have critical requirement on the model transparency and interpretability. In order to enhance the explainability of neural networks, we propose a neural rule learning method-Feature Interactive Neural Rule Learning (FINRule) to incorporate the expressivity of neural networks and the interpretability of rule-based systems. Specifically, we conduct rule learning as differential discrete combination encoded by a feedforward neural network, in which each layer acts as a logical operator of explainable decision conditions. The first hidden layer can act as sharable atomic conditions which are connected to next hidden layer for formulating decision rules. Moreover, we propose to represent both atomic condition and rules with contextual embeddings, with aim to enrich the expressivity power by capturing high-order feature interactions. We conduct comprehensive experiments on real-world datasets to validate both effectiveness and explainability of the proposed method.
Copycat: Near-duplicates within and between the clueweb and the common crawl
Authors: Maik Fröbe, Janek Bevendorff, Lukas Gienapp, Michael Völske, Benno Stein 0001, Martin Potthast, Matthias Hagen
Venue: Conference SIGIR 2021
Year: 2021

The amount of near-duplicates in web crawls like the ClueWeb or Common Crawl demands from their users either to develop a preprocessing pipeline for deduplication, which is costly both computationally and in person hours, or accepting the undesired effects that near-duplicates have on reliability and validity of experiments. We introduce ChatNoir-CopyCat-21, which simplifies deduplication significantly. It comes in two parts: (1) A compilation of near-duplicate documents within the ClueWeb09, the ClueWeb12, and two Common Crawl snapshots, as well as between selections of these crawls, and (2) a software library that implements the deduplication of arbitrary document sets. Our analysis shows that 14--52, of the documents within a crawl and around~0.7--2.5, between the crawls are near-duplicates. Two showcases demonstrate the application and usefulness of our resource.
Prospective modeling of users for online display advertising via deep time-aware model
Authors: Djordje Gligorijevic, Jelena Gligorijevic, Aaron Flores
Venue: Conference CIKM 2020
Year: 2020

Prospective display advertising poses a particular challenge for large advertising platforms. The existing machine learning algorithms are easily biased towards the highly predictable retargeting events that are often non-eligible for the prospective campaigns, thus exhibiting a decline in advertising performance. To that end, efforts are made to design powerful models that can learn from signals of various strength and temporal impact collected about each user from different data sources and provide a good quality and early estimation of users' conversion rates. In this study, we propose a novel deep time-aware approach designed to model sequences of users' activities and capture implicit temporal signals of users' conversion intents. On several real-world datasets, we show that the proposed approach consistently outperforms other, previously proposed approaches by a significant margin while providing interpretability of signal impact to conversion probability.