Reviewerly/exHarmony huggingface dataset:
A Survey of Collective Anomaly Detection on Sequence Dataset

Anomaly detection on sequence dataset typically focuses on the detection of collective anomalies, aiming to find anomalous patterns consisting of sequences of data with specific relationships rather than individual observations. In this survey, existing studies are summarized to align with temporal sequence dataset and spatial sequence dataset. For the first category, the detection can be subdivided into symbolic dataset based and time series dataset based, which include similarity, probabilistic, and trend approaches. For the second category, it can be subdivided into homogeneous datasets based heterogeneous datasets based, which include multi-dataset fusion and joint approaches. Compared to the state-of-the-art survey papers, the contribution of this paper lies in providing a deep analysis of various representations of collective anomaly in different application field and their corresponding detection methods, representative techniques. As a result, practitioners can receive some guidance for selecting the most suitable methods for their particular case.
S <sup>2</sup> phere: Semi-Supervised Pre-training for Web Search over Heterogeneous Learning to Rank Data

While Learning to Rank (LTR) models on top of transformers have been widely adopted to achieve decent performance, it is still challenging to train the model with sufficient data as only an extremely small number of query-webpage pairs could be annotated versus trillions of webpages available online and billions of web search queries everyday. In the meanwhile, industry research communities have released a number of open-source LTR datasets with well annotations but incorporating different designs of LTR features/labels (i.e., heterogeneous domains). In this work, inspired by the recent progress in pre-training transformers for performance advantages, we study the problem of pre-training LTR models using both labeled and unlabeled samples, especially we focus on the use of well-annotated samples in heterogeneous open-source LTR datasets to boost the performance of pre-training. Hereby, we propose S2phere-Semi-Supervised Pre-training with Heterogeneous LTR data strategies for LTR models using both unlabeled and labeled query-webpage pairs across heterogeneous LTR datasets. S2phere consists of a three-step approach: (1) Semi-supervised Feature Extraction Pre-training via Perturbed Contrastive Loss, (2) Cross-domain Ranker Pre-training over Heterogeneous LTR Datasets and (3) End-to-end LTR Fine-tuning via Modular Network Composition. Specifically, given an LTR model composed of a backbone (the feature extractor), a neck (the module to reason the orders) and a head (the predictor of ranking scores), S2phere uses unlabeled/labeled data from the search engine to pre-train the backbone in Step (1) via semi-supervised learning; then Step (2) incorporates multiple open-source heterogeneous LTR datasets to improve pre-training of the neck module as shared parameters of cross-domain learning; and finally, S2phere in Step (3) composes the backbone and neck with a randomly-initialized head into a whole LTR model and fine-tunes the model using search engine data with various learning strategies. Extensive experiments have been done with both offline experiments and online A/B Test on top of Baidu search engine. The comparisons against numbers of baseline algorithms confirmed the advantages of S2phere in producing high-performance LTR models for web-scale search.
2nd Workshop on Uncertainty Reasoning and Quantification in Decision Making

Uncertainty reasoning and quantification play a critical role in decision making across various domains, prompting increased attention from both academia and industry. As real-world applications become more complex and data-driven, effectively handling uncertainty becomes paramount for accurate and reliable decision making. This workshop focuses on the critical topics of uncertainty reasoning and quantification in decision making. It provides a platform for experts and researchers from diverse backgrounds to exchange ideas on cutting-edge techniques and challenges in this field. The interdisciplinary nature of uncertainty reasoning and quantification, spanning artificial intelligence, machine learning, statistics, risk analysis, and decision science, will be explored. The workshop aims to address the need for robust and interpretable methods for modeling and quantifying uncertainty, fostering reasoning decision-making in various domains. Participants will have the opportunity to share research findings and practical experiences, promoting collaboration and advancing decision-making practices under uncertainty.
Deep Pipeline Embeddings for AutoML

Automated Machine Learning (AutoML) is a promising direction for democratizing AI by automatically deploying Machine Learning systems with minimal human expertise. The core technical challenge behind AutoML is optimizing the pipelines of Machine Learning systems (e.g. the choice of preprocessing, augmentations, models, optimizers, etc.). Existing Pipeline Optimization techniques fail to explore deep interactions between pipeline stages/components. As a remedy, this paper proposes a novel neural architecture that captures the deep interaction between the components of a Machine Learning pipeline. We propose embedding pipelines into a latent representation through a novel per-component encoder mechanism. To search for optimal pipelines, such pipeline embeddings are used within deep-kernel Gaussian Process surrogates inside a Bayesian Optimization setup. Furthermore, we meta-learn the parameters of the pipeline embedding network using existing evaluations of pipelines on diverse collections of related datasets (a.k.a. meta-datasets). Through extensive experiments on three large-scale meta-datasets, we demonstrate that pipeline embeddings yield state-of-the-art results in Pipeline Optimization.
Underwater image restoration based on second-order statistical characteristics

Underwater image restoration is of great significance to underwater vision operation. We propose an underwater image restoration method based on the independent color channel of a single image. In this method, the relationship between the transmittance parameters and the second-order statistics of the scattered light underwater is established through a multi-layer continuous model of the water. First, the region with approximate depth is obtained by super-pixel segmentation, then the sub-color channel statistical calculation is carried out for each super-pixel independently, and finally the transmittance of each color channel is estimated combined with the relational expression. Experimental results show that this method has better effect on color fidelity of underwater image restoration.
Financial Default Prediction via Motif-preserving Graph Neural Network with Curriculum Learning

User financial default prediction plays a critical role in credit risk forecasting and management. It aims at predicting the probability that the user will fail to make the repayments in the future. Previous methods mainly extract a set of user individual features regarding his own profiles and behaviors and build a binary-classification model to make default predictions. However, these methods cannot get satisfied results, especially for users with limited information. Although recent efforts suggest that default prediction can be improved by social relations, they fail to capture the higher-order topology structure at the level of small subgraph patterns. In this paper, we fill in this gap by proposing a motif-preserving Graph Neural Network with curriculum learning (MotifGNN) to jointly learn the lower-order structures from the original graph and higher-order structures from multi-view motif-based graphs for financial default prediction. Specifically, to solve the problem of weak connectivity in motif-based graphs, we design the motif-based gating mechanism. It utilizes the information learned from the original graph with good connectivity to strengthen the learning of the higher-order structure. And considering that the motif patterns of different samples are highly unbalanced, we propose a curriculum learning mechanism on the whole learning process to more focus on the samples with uncommon motif distributions. Extensive experiments on one public dataset and two industrial datasets all demonstrate the effectiveness of our proposed method.
DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization

Personalized federated learning collaboratively trains client-specific models, which holds potential for various mobile and IoT applications with heterogeneous data. However, existing solutions are vulnerable to distribution shifts between training and test data, and involve high training workloads on local devices. These two shortcomings hinder the practical usage of personalized federated learning on real-world mobile applications. To overcome these drawbacks, we explore efficient shift-robust personalization for federated learning. The principle is to hitchhike the global model to improve the shift-robustness of personalized models with minimal extra training overhead. To this end, we present DM-PFL, a novel framework that utilizes a dual masking mechanism to train both global and personalized models with weight-level parameter sharing and end-to-end sparse training. Evaluations on various datasets show that our methods not only improve the test accuracy in presence of test-time distribution shifts but also save the communication and computation costs compared to state-of-the-art personalized federated learning schemes.
Learning Discrete Document Representations in Web Search

Product quantization (PQ) has been usually applied to dense retrieval (DR) of documents thanks to its competitive time, memory efficiency and compatibility with other approximate nearest search (ANN) methods. Originally, PQ was learned to minimize the reconstruction loss, i.e., the distortions between the original dense embeddings and the reconstructed embeddings after quantization. Unfortunately, such an objective is inconsistent with the goal of selecting ground-truth documents for the input query, which may cause a severe loss of retrieval quality. Recent research has primarily concentrated on jointly training the biencoders and PQ to ensure consistency for improved performance. However, it is still difficult to design an approach that can cope with challenges like discrete representation collapse, mining informative negatives, and deploying effective embedding-based retrieval (EBR) systems in a real search engine.
VQNE: Variational Quantum Network Embedding with Application to Network Alignment

Learning of network embedding with vector-based node representation has attracted wide attention over the decade. It differs from the general setting of graph node embedding whereby the node attributes are also considered and yet may incur privacy issues. In this paper, we depart from the classic CPU/GPU architecture to consider the well-established network alignment problem based on network embedding, and develop a quantum machine learning approach with a low qubit cost for its near-future applicability on Noisy Intermediate-Scale Quantum (NISQ) devices. Specifically, our model adopts the discrete-time quantum walk (QW) and conducts the QW on the tailored merged network to extract structure information from the two aligning networks without the need for quantum state preparation which otherwise requires high quantum gate cost. Then the quantum states from QW are fed to a quantum embedding ansatz (i.e., parameterized circuit) to learn the latent representation of each node. The key part of our approach is to connect these two quantum modules to achieve a pure quantum paradigm without involving classical modules. To our best knowledge, there has not been any classic-quantum hybrid approach to network embedding, let alone a pure quantum paradigm being free from the bottleneck of communication between classic devices and quantum devices, which is still an open problem. Experimental results on two real-world datasets show the effectiveness of our quantum embedding approach in comparison with classical embedding approaches. Our model is readily and efficiently implemented in Python with a full-amplitude simulation of the QW and the quantum circuit. Therefore, our model can be readily deployed on an existing NISQ device with all the circuits provided, and only 13 qubits are needed in the experiments, which is rarely attained in existing quantum graph learning works.
Unbiased Locally Private Estimator for Polynomials of Laplacian Variables

This work presents a mechanism to debias polynomial functions computed from locally differentially private data. Local differential privacy is a widely used privacy notion where users add Laplacian noise to their information before submitting it to a central server. That, however, causes bias when we calculate non-linear functions based on those noisy information. Our proposed recursive algorithm debiases these functions, with a calculation time of O(r n log n), where r is the polynomial degree and n is the number of users. We evaluate our method on the problems of k-star counting and variance estimation, comparing results with state-of-the-art algorithms. The results show that our method not only eliminates bias, but also provides at least 100 times more accuracy than previous works.
Fresh Content Needs More Attention: Multi-funnel Fresh Content Recommendation

Recommendation system serves as a conduit connecting users to an incredibly large, diverse and ever growing collection of contents. In practice, missing information on fresh (and tail) contents needs to be filled in order for them to be exposed and discovered by their audience. We here share our success stories in building a dedicated fresh content recommendation stack on a large commercial platform. To nominate fresh contents, we built a multi-funnel nomination system that combines (i) a two-tower model with strong generalization power for coverage, and (ii) a sequence model with near real-time update on user feedback for relevance. The multi-funnel setup effectively balances between coverage and relevance. An in-depth study uncovers the relationship between user activity level and their proximity toward fresh contents, which further motivates a contextual multi-funnel setup. Nominated fresh candidates are then scored and ranked by systems considering prediction uncertainty to further bootstrap content with less exposure. We evaluate the benefits of the dedicated fresh content recommendation stack, and the multi-funnel nomination system in particular, through user corpus co-diverted live experiments. We conduct multiple rounds of live experiments on a commercial platform serving billion of users demonstrating efficacy of our proposed methods.
B <sup>2</sup> -Sampling: Fusing Balanced and Biased Sampling for Graph Contrastive Learning

Graph contrastive learning (GCL), aiming for an embedding space where semantically similar nodes are closer, has been widely applied in graph-structured data. Researchers have proposed many approaches to define positive and negative pairs (i.e., semantically similar and dissimilar pairs) on the graph, serving as labels to learn their embedding distances. Despite the effectiveness, those approaches usually suffer from two typical learning challenges. First, the number of candidate negative pairs is enormous. Thus, it is non-trivial to select representative ones to train the model in a more effective way. Second, the heuristics (e.g., graph views or meta-path patterns) to define positive and negative pairs are sometimes less reliable, causing considerable noise for both "labelled'' positive and negative pairs. In this work, we propose a novel sampling approach B2-Sampling to address the above challenges in a unified way. On the one hand, we use balanced sampling to select the most representative negative pairs regarding both the topological and embedding diversities. On the other hand, we use biased sampling to learn and correct the labels of the most error-prone negative pairs during the training. The balanced and biased samplings can be applied iteratively for discriminating and correcting training pairs, boosting the performance of GCL models. B2-Sampling is designed as a framework to support many known GCL models. Our extensive experiments on node classification, node clustering, and graph classification tasks show that B2-Sampling significantly improves the performance of GCL models with acceptable runtime overhead. Our website[11] https://sites.google.com/view/b2-sampling/home provides access to our codes and additional experiment results.
MGNN: Graph Neural Networks Inspired by Distance Geometry Problem

Graph Neural Networks (GNNs) have emerged as a prominent research topic in the field of machine learning. Existing GNN models are commonly categorized into two types: spectral GNNs, which are designed based on polynomial graph filters, and spatial GNNs, which utilize a message-passing scheme as the foundation of the model. For the expressive power and universality of spectral GNNs, a natural approach is to improve the design of basis functions for better approximation ability. As for spatial GNNs, models like Graph Isomorphism Networks (GIN) analyze their expressive power based on Graph Isomorphism Tests. Recently, there have been attempts to establish connections between spatial GNNs and geometric concepts like curvature and cellular sheaves, as well as physical phenomena like oscillators. However, despite the recent progress, there is still a lack of comprehensive analysis regarding the universality of spatial GNNs from the perspectives of geometry and physics.
Graph Neural Networks in TensorFlow

Graphs are general data structures that can represent information from a variety of domains (social, biomedical, online transactions, and many more). Graph Neural Networks (GNNs) are quickly becoming the de-facto Machine Learning models for learning from Graph data and hereby infer missing information, such as, predicting labels of nodes or imputing missing edges.
Sequence As Genes: An User Behavior Modeling Framework for Fraud Transaction Detection in E-commerce

With the explosive growth of e-commerce, detecting fraudulent transactions in real-world scenarios is becoming increasingly important for e-commerce platforms. Recently, several supervised approaches have been proposed to use user behavior sequences, which record the user's track on platforms and contain rich information for fraud transaction detection. Nevertheless, these methods always suffer from the scarcity of labeled data in real-world scenarios. The recent remarkable pre-training methods in Natural Language Processing (NLP) and Computer Vision (CV) domains offered glimmers of light. However, user behavior sequences differ intrinsically from text, images, and videos. In this paper, we propose a novel and general user behavior pre-training framework, named Sequence As GEnes (SAGE), which provides a new perspective for user behavior modeling. Following the inspiration of treating sequences as genes, we carefully designed the user behavior data organization paradigm and pre-training scheme. Specifically, we propose an efficient data organization paradigm inspired by the nature of DNA expression, which decouples the length of behavior sequences and the corresponding time spans. Also inspired by the natural mechanisms in genetics, we propose two pre-training tasks, namely sequential mutation and sequential recombination, to improve the robustness and consistency of user behavior representations in complicated real-world scenes. Extensive experiments on four differentiated fraud transaction detection real scenarios demonstrate the effectiveness of our proposed framework.
Semantic Dissimilarity Guided Locality Preserving Projections for Partial Label Dimensionality Reduction

Partial label learning (PLL) is a significant weakly supervised learning framework, where each training example corresponds to a set of candidate labels among which only one is the ground-truth label. Existing works on partial label dimensionality reduction only exploit the disambiguated labels, but overlook the available semantic dissimilarity relationship hidden in the disambiguated labeling confidence, i.e., the smaller the inner product of the labeling confidences of two instances, the less likely they have the same ground-truth label. By combining such global dissimilarity relationship with local neighborhood information, we propose a novel partial label dimensionality reduction method named SDLPP, which employs an alternating procedure including candidate label disambiguation, semantic dissimilarity generation and dimensionality reduction. The labeling confidences of candidate labels and semantic dissimilarity relationship are constantly updated through the alternating procedure, where the processes in each iteration are based on the low-dimensional data obtained in the previous iteration. After the alternating procedure, SDLPP maps the original data to a pre-specified low-dimensional feature space. Comprehensive experiments on both synthetic and real-world data sets validate that SDLPP can improve the generalization performance of different PLL algorithms, and outperform state-of-the-art partial label dimensionality reduction methods. The codes can be publicly accessible on the link https://github.com/jhjiangSEU/SDLPP.
Contrastive Learning for User Sequence Representation in Personalized Product Search

Providing personalization in product search has attracted increasing attention in both industry and research communities. Most existing personalized product search methods model users' individual search interests based on their historical search logs to generate personalized search results. However, the search logs may be sparse or noisy in the real scenario, which is difficult for existing methods to learn accurate and robust user representations. To address this issue, we propose a contrastive learning framework CoPPS that aims to learn high-quality user representations for personalized product search. Specifically, we design three data augmentation and contrastive learning strategies to construct self-supervision signals from the original search behaviours. The contrastive learning tasks utilize an external knowledge graph and exploit the correlations within and between user sequences, thereby facilitating the discovery of more meaningful search patterns and ultimately enhancing the quality of personalized search. Experimental results on the public Amazon datasets verify the effectiveness of our approach.
Improving Search Clarification with Structured Information Extracted from Search Results

Search clarification in conversational search systems exhibits a clarification pane composed of several candidate aspect items and a clarifying question. To generate a pane, existing studies usually rely on unstructured document texts. However, important structured information in search results is not effectively considered, making the generated panes inaccurate in some cases. In this paper, we emphasize the importance of structured information in search results for improving search clarification. We propose enhancing unstructured documents with two kinds of structured information: one is "In-List'' relation obtained from HTML list structures, which helps extract groups of high-quality items with abundant parallel information. Another is "Is-A'' relation extracted from knowledge bases, which is helpful to generate good questions with explicit prompts. To avoid introducing excessive noises, we design a relation selection process to filter out ineffective relations. We further design a BART-based model for generating clarification panes. The experimental results show that the structured information is good supplement for generating high-quality clarification panes.