Reviewerly/exHarmony huggingface dataset:
Learning to Discover Various Simpson's Paradoxes

Simpson's paradox is a well-known statistical phenomenon that has captured the attention of statisticians, mathematicians, and philosophers for more than a century. The paradox often confuses people when it appears in data, and ignoring it may lead to incorrect decisions. Recent studies have found many examples of Simpson's paradox in social data and proposed a few methods to detect the paradox automatically. However, these methods suffer from many limitations, such as being only suitable for categorical variables or one specific paradox. To address these problems, we develop a learning-based approach to discover various Simpson's paradoxes. Firstly, we propose a framework from a statistical perspective that unifies multiple variants of Simpson's paradox currently known. Secondly, we present a novel loss function, Multi-group Pearson Correlation Coefficient (MPCC), to calculate the association strength of two variables of multiple subgroups. Then, we design a neural network model, coined SimNet, to automatically disaggregate data into multiple subgroups by optimizing the MPCC loss. Experiments on various datasets demonstrate that SimNet can discover various Simpson's paradoxes caused by discrete and continuous variables, even hidden variables. The code is available at https://github.com/ant-research/Learning-to-Discover-Various-Simpson-Paradoxes.
GraphStorm an Easy-to-use and Scalable Graph Neural Network Framework: From Beginners to Heroes

Applying Graph Neural Networks (GNNs) to real-world problems is challenging for machine learning (ML) practitioners due to two major obstacles. The first hurdle is the high barrier to learn programming GNNs from scratch. The second challenge lies in overcoming engineering difficulties when scaling GNN models for large graphs at an industry-level. To address these challenges, GraphStorm, an open-source framework, offers a solution by providing an easy-to-use user interface and an end-to-end GNN training/inference pipeline that seamlessly handles extremely large graphs in a distributed manner This tutorial aims to provide participants with a comprehensive understanding of GraphStorm, including its design principles, target users, and use cases, through presentations. The hands-on sections will enable attendees to walk through four practical GraphStorm use cases that can assist them in leveraging GNNs to address real-world business problems.
Distributed Optimization for Big Data Analytics: Beyond Minimization

The traditional machine learning model can be formulated as an empirical risk minimization problem, which is typically optimized via stochastic gradient descent (SGD). With the emergence of big data, distributed optimization, e.g., distributed SGD, has been attracting increasing attention to facilitate machine learning models for big data analytics. However, existing distributed optimization mainly focuses on the standard empirical risk minimization problem, failing to deal with the emerging machine learning models that are beyond that category. Thus, of particular interest of this tutorial includes the stochastic minimax optimization, stochastic bilevel optimization, and stochastic compositional optimization, which covers a wide range of emerging machine learning models, e.g., model-agnostic meta-learning models, adversarially robust machine learning models, imbalanced data classification models, etc. Since these models have been widely used in big data analytics, it is necessary to provide a comprehensive introduction about the new distributed optimization algorithms designed for these models. Therefore, the goal of this tutorial is to present the state-of-the-art and recent advances in distributed minimax optimization, distributed bilevel optimization, and distributed compositional optimization. In particular, we will introduce the typical applications in each category and discuss the corresponding distributed optimization algorithms in both centralized and decentralized settings. Through this tutorial, the researchers will be exposed to the fundamental algorithmic design and basic convergence theories, and the practitioners will be able to benefit from this tutorial to apply these algorithms to real-world data mining applications.
OPORP: One Permutation + One Random Projection

OPORP is a variant of the count-sketch data structure by using a fixed-length binning scheme and a normalization step for the estimation. In our experience, we find engineers like the name "one permutation + one random projection" as it tells the exact steps. Consider two vectors (e.g., embeddings): u, v ? R D with p = cos(u, v). In embedding-based applications (e.g., EBR), D = 256 - 4096 are common. With OPORP, we first apply a permutation on the data vectors. A vector r ∈ R D is generated i.i.d. with E(i) = 0, E(r 2 i ) = 1, E(r 3 i ) = 0, E(r4 i ) = s, where s ≥ 1. We multiply (as Hadamard product) r with all permuted data vectors. Then we break the D columns into k equal-length bins and aggregate (i.e., sum) the values in each bin to obtain k samples from each data vector. One crucial step is to normalize the k samples to the unit l2 norm. We show that the estimation variance equals:(s - 1)A + D-k/D-1 1/k [(1-p2)2 -2A ] , A ≥ 0, s ≥ 1, which reveals several key properties of the proposed scheme:
Analysis of COVID-19 Offensive Tweets and Their Targets

During the global COVID-19 pandemic, people utilized social media platforms, especially Twitter, to spread and express opinions about the pandemic. Such discussions also drove the rise in COVID-related offensive speech. In this work, focusing on Twitter, we present a comprehensive analysis of COVID-related offensive tweets and their targets. We collected a COVID-19 dataset with over 747 million tweets for 30 months and fine-tuned a BERT classifier to detect offensive tweets. Our offensive tweets analysis shows that the ebb and flow of COVID-related offensive tweets potentially reflect events in the physical world. We then studied the targets of these offensive tweets. There was a large number of offensive tweets with abusive words, which could negatively affect the targeted groups or individuals. We also conducted a user network analysis, and found that offensive users interact more with other offensive users and that the pandemic had a lasting impact on some offensive users. Our study offers novel insights into the persistence and evolution of COVID-related offensive tweets during the pandemic
Context-Aware Hashing

With the development of social media and mobile computing technology, the past decade has witnessed tremendous growth in social images. Consequently, there has been an increasing interest in information retrieval and multimedia computing communities to study intelligent image retrieval techniques. In particular, image retrieval techniques [1, 2], where only an image is used as the query, are gaining importance due to a wide range of promising applications.
Hands-on Tutorial: "Explanations in AI: Methods, Stakeholders and Pitfalls"

While using vast amounts of training data and sophisticated models has enhanced the predictive performance of Machine Learning (ML) and Artificial Intelligence (AI) solutions, it has also led to an increased difficulty in comprehending their predictions. The ability to explain predictions is often one of the primary desiderata for adopting AI and ML solutions [6, 13]. The desire for explainability has led to a rapidly growing body of literature on explainable AI (XAI) and has also resulted in the development of hundreds of XAI methods targeting different domains (e.g., finance, healthcare), applications (e.g., model debugging, actionable recourse), data modalities (e.g., tabular data, images), models (e.g., transformers, convolutional neural networks) and stakeholders (e.g., end-users, regulatory authorities, data scientists). The goal of this tutorial is to present a comprehensive overview of the XAI field to the participants. As a hands-on tutorial, we will showcase state-of-the-art methods that can be used for different data modalities and contexts to extract the right abstractions for interpretation. We will also cover common pitfalls when using explanations, e.g., misrepresentation, and lack of robustness of explanations.
Narrow the Input Mismatch in Deep Graph Neural Network Distillation

Graph neural networks (GNNs) have been widely studied for modeling graph-structured data. Thanks to the over-parameterization and large receptive field of deep GNNs, "deep" is a promising direction to develop GNNs further and has shown some superior performances. However, the over-stacked structures of deep architectures incur high inference cost in deployment. To compress deep GNNs, we can use knowledge distillation (KD) to make shallow student GNNs mimic teacher GNNs. Existing KD methods in graph domain focus on constructing diverse supervision on embedding or prediction produced by student GNNs, but overlook the gap of the receptive field (i.e., input information) between student and teacher, which brings difficulties to KD. We call this gap "input mismatch". To alleviate this problem, we propose a lightweight stochastic extended module to provide an estimation for missing input information for student GNNs. The estimator models the distribution of missing information. Specifically, we model the missing information as an independent distribution from graph level and a conditional distribution from node level (given the condition of observable input). These two estimates are optimized using a Bayesian methodology and combined into a balanced estimate as additional input to student GNNs. To the best of our knowledge, we are the first to address the "input mismatch" problem in deep GNNs distillation. Experiments on extensive benchmarks demonstrate that our method outperforms existing KD methods for GNNs in distillation performance, which confirms that the estimations are reasonable and effective.
Partial-label Learning with Mixed Closed-set and Open-set Out-of-candidate Examples

Partial-label learning (PLL) relies on a key assumption that the true label of each training example must be in the candidate label set. This restrictive assumption may be violated in complex real-world scenarios, and thus the true label of some collected examples could be unexpectedly outside the assigned candidate label set. In this paper, we term the examples whose true label is outside the candidate label set OOC (Out-Of-Candidate) examples, and pioneer a new PLL study to learn with OOC examples. We consider two types of OOC examples in reality, i.e., the closed-set/open-set OOC examples whose true label is inside/outside the known label space. To solve this new PLL problem, we first calculate the wooden cross-entropy loss from candidate and non-candidate labels respectively, and dynamically differentiate the two types of OOC examples based on specially designed criteria. Then, for closed-set OOC examples, we conduct reversed label disambiguation in the non-candidate label set; for open-set OOC examples, we leverage them for training by utilizing an effective regularization strategy that dynamically assigns random candidate labels from the candidate label set. In this way, the two types of OOC examples can be differentiated and further leveraged for model training. Extensive experiments demonstrate that our proposed method outperforms state-of-the-art PLL methods.
Improving Expressivity of GNNs with Subgraph-specific Factor Embedded Normalization

Graph Neural Networks~(GNNs) have emerged as a powerful category of learning architecture for handling graph-structured data. However, existing GNNs typically ignore crucial structural characteristics in node-induced subgraphs, which thus limits their expressiveness for various downstream tasks. In this paper, we strive to strengthen the representative capabilities of GNNs by devising a dedicated plug-and-play normalization scheme, termed as SUbgraph-sPEcific FactoR Embedded Normalization (SuperNorm), that explicitly considers the intra-connection information within each node-induced subgraph. To this end, we embed the subgraph-specific factor at the beginning and the end of the standard BatchNorm, as well as incorporate graph instance-specific statistics for improved distinguishable capabilities. In the meantime, we provide theoretical analysis to support that, with the elaborated SuperNorm, an arbitrary GNN is at least as powerful as the 1-WL test in distinguishing non-isomorphism graphs. Furthermore, the proposed SuperNorm scheme is also demonstrated to alleviate the over-smoothing phenomenon. Experimental results related to predictions of graph, node, and link properties on the eight popular datasets demonstrate the effectiveness of the proposed method. The code is available at https://github.com/chenchkx/SuperNorm.
Cost reference particle filter with multi-probability distribution for nonlinear dynamic systems with unknown noise statistics

The lack of particle diversity is a vital problem in particle filtering for nonlinear dynamic systems with unknown noise statistics. In this paper, cost reference particle filter with multi-probability distribution (CRPF_MPD) is introduced to minimize the effects of the lack of particle diversity and to obtain more realistic state estimations. The core of multi-probability distribution is that two subsets are parallelly initialized and resampled by the processes of cost reference particle filter (CRPF) and are respectively updated with different distributions. Information interaction and particle selection form a set of particles with multi-probability distribution to estimate the more realistic states of systems. CRPF_MPD relies on multi-probability distribution, information interaction and particle selection to have excellent performance. As can be seen from the results of simulations, CRPF_MPDs can enhance the accuracy of state estimation and are more robust to different systems. Thus, CRPF_MPDs are available for dynamic systems with unknown noise statistics.
Clenshaw Graph Neural Networks

Graph Convolutional Networks (GCNs), which use a message-passing paradigm with stacked convolution layers, are foundational spatial methods for learning graph representations. Polynomial filters, which have an advantage on heterophilous graphs, are motivated differently from the spectral perspective of graph convolutions. Recent spatial GCN models use various residual connection techniques to alleviate the model degradation problem such as over-smoothing and gradient vanishing. However, current residual connections do not effectively harness the full potential of polynomial filters, which are commonly employed in the spectral domain of GNNs.
Deep snapshot HDR imaging using multi-exposure color filter array

In this paper, we propose a deep snapshot high dynamic range (HDR) imaging framework that can effectively reconstruct an HDR image from the RAW data captured using a multi-exposure color filter array (ME-CFA), which consists of a mosaic pattern of RGB filters with different exposure levels. To effectively learn the HDR image reconstruction network, we introduce the idea of luminance normalization that simultaneously enables effective loss computation and input data normalization by considering relative local contrasts in the "normalized-by-luminance" HDR domain. This idea enables the network to equally handle the errors in both bright and dark areas regardless of absolute luminance levels, which significantly improves the visual image quality. Experimental results using public HDR image datasets demonstrate that our framework outperforms other snapshot methods and produces high-quality HDR images with fewer visual artifacts, resulting in more than 4dB color peak signal-to-noise ratio improvement in the linear HDR domain.
Explicit Feature Interaction-aware Uplift Network for Online Marketing

As a key component in online marketing, uplift modeling aims to accurately capture the degree to which different treatments motivate different users, such as coupons or discounts, also known as the estimation of individual treatment effect (ITE). In an actual business scenario, the options for treatment may be numerous and complex, and there may be correlations between different treatments. In addition, each marketing instance may also have rich user and contextual features. However, existing methods still fall short in both fully exploiting treatment information and mining features that are sensitive to a particular treatment. In this paper, we propose an explicit feature interaction-aware uplift network (EFIN) to address these two problems. Our EFIN includes four customized modules: 1) a feature encoding module encodes not only the user and contextual features, but also the treatment features; 2) a self-interaction module aims to accurately model the user's natural response with all but the treatment features; 3) a treatment-aware interaction module accurately models the degree to which a particular treatment motivates a user through interactions between the treatment features and other features, i.e., ITE; and 4) an intervention constraint module is used to balance the ITE distribution of users between the control and treatment groups so that the model would still achieve a accurate uplift ranking on data collected from a non-random intervention marketing scenario. We conduct extensive experiments on two public datasets and one product dataset to verify the effectiveness of our EFIN. In addition, our EFIN has been deployed in a credit card bill payment scenario of a large online financial platform with a significant improvement.
Multi-modal Discrete Collaborative Filtering

With the development of E-commerce, recommender systems have been widely adopted by many online services for helping their customers find desirable products to purchase. However, the ever-growing scales of products and users render recommendation more challenging than ever before [1]. For example, there are more than 0.82 billion active Taobao users and over one billion products for sale now. Consequently, it is challenging to make the immediate response to match products for potential customers accurately and efficiently, by analyzing large-scale yet sparse user interaction history.
Large-Scale Graph Neural Networks: The Past and New Frontiers

Graph Neural Networks (GNNs) have gained significant attention in recent years due to their ability to model complex relationships between entities in graph-structured data such as social networks, protein structures, and knowledge graphs. However, due to the size of real-world industrial graphs and the special architecture of GNNs, it is a long-lasting challenge for engineers and researchers to deploy GNNs on large-scale graphs, which significantly limits their applications in real-world applications. In this tutorial, we will cover the fundamental scalability challenges of GNNs, frontiers of large-scale GNNs including classic approaches and some newly emerging techniques, the evaluation and comparison of scalable GNNs, and their large-scale real-world applications. Overall, this tutorial aims to provide a systematic and comprehensive understanding of the challenges and state-of-the-art techniques for scaling GNNs. The summary and discussion on future directions will inspire engineers and researchers to explore new ideas and developments in this rapidly evolving field. The website of this tutorial is available at https://sites.google.com/ncsu.edu/gnnkdd2023tutorial.
CampER: An Effective Framework for Privacy-Aware Deep Entity Resolution

Entity Resolution (ER) is a fundamental problem in data preparation. Standard deep ER methods have achieved state-of-the-art effectiveness, assuming that relations from different organizations are centrally stored. However, due to privacy concerns, it can be difficult to centralize data in practice, rendering standard deep ER solutions inapplicable. Despite efforts to develop rule-based privacy-preserving ER methods, they often neglect subtle matching mechanisms and have poor effectiveness as a result. To bridge effectiveness and privacy, in this paper, we propose CampER, an effective framework for privacy-aware deep entity resolution. Specifically, we first design a training pair self-generation strategy to overcome the absence of manually labeled data in privacy-aware scenarios. Based on the self-constructed training pairs, we present a collaborative fine-tuning approach to learn the match-aware and uni-space individual tuple embeddings for accurate matching decisions. During the matching decision-making process, we first introduce a cryptographically secure approach to determine matches. Furthermore, we propose an order-preserving perturbation strategy to significantly accelerate the matching computation while guaranteeing the consistency of ER results. Extensive experiments on eight widely-used benchmark datasets demonstrate that CampER not only is comparable with the state-of-the-art standard deep ER solutions in effectiveness, but also preserves privacy.
End-to-End Query Term Weighting

Bag-of-words based lexical retrieval systems are still the most commonly used methods for real-world search applications. Recently deep learning methods have shown promising results to improve this retrieval performance but are expensive to run in an online fashion, non-trivial to integrate into existing production systems, and might not generalize well in out-of-domain retrieval scenarios. Instead, we build on top of lexical retrievers by proposing a Term Weighting BERT (TW-BERT) model. TW-BERT learns to predict the weight for individual n-gram (e.g., uni-grams and bi-grams) query input terms. These inferred weights and terms can be used directly by a retrieval system to perform a query search. To optimize these term weights, TW-BERT incorporates the scoring function used by the search engine, such as BM25, to score query-document pairs. Given sample query-document pairs we can compute a ranking loss over these matching scores, optimizing the learned query term weights in an end-to-end fashion. Aligning TW-BERT with search engine scorers minimizes the changes needed to integrate it into existing production applications, whereas existing deep learning based search methods would require further infrastructure optimization and hardware requirements. The learned weights can be easily utilized by standard lexical retrievers and by other retrieval techniques such as query expansion. We show that TW-BERT improves retrieval performance over strong term weighting baselines within MSMARCO and in out-of-domain retrieval on TREC datasets.
A Message Passing Neural Network Space for Better Capturing Data-dependent Receptive Fields

Recently, the message passing neural network (MPNN) has attracted a lot of attention, which learns node representations based on the receptive field of the given node. Despite its success in many graph-related tasks, recent studies find that conventional MPNNs are incapable of handling variant receptive fields required in different graphs, and thereby some upgraded MPNNs have been developed. However, these methods are limited to designing a common solution for different graphs, which fails to capture the impact of different graph properties on the receptive fields. To alleviate such issues, we propose a novel MPNN space for data-dependent receptive fields (MpnnDRF), which enables us to dynamically design suitable MPNNs to capture the receptive field for the given graph. More concretely, we systemically investigate the capability of existing designs and propose several key design dimensions to improve them. Then, to fully explore the proposed designs and useful designs in existing works, we propose a novel search space to incorporate them and formulate a search framework. In the empirical study, the proposed MpnnDRF shows very strong robustness against the increased receptive field, which allows MpnnDRF to learn node representations based on a larger perceptual field. Therefore, MpnnDRF consistently achieves outstanding performance on benchmark node and graph classification tasks.