Miracle-dz/UniFAR-FTUs huggingface dataset:
Efficient Learning to Learn a Robust CTR Model for Web-scale Online Sponsored Search Advertising

Click-through rate (CTR) prediction is crucial for online sponsored search advertising. Several successful CTR models have been adopted in the industry, including the regularized logistic regression (LR). Nonetheless, the learning process suffers from two limitations: 1) Feature crosses for high-order information may generate trillions of features, which are sparse for online learning examples; 2) Rapid changing of data distribution brings challenges to the accurate learning since the model has to perform a fast adaptation on the new data. Moreover, existing adaptive optimizers are ineffective in handling the sparsity issue for high-dimensional features. In this paper, we propose to learn an optimizer in a meta-learning scenario, where the optimizer is learned on prior data and can be easily adapted to the new data. We firstly build a low-dimensional feature embedding on prior data to encode the association among features. Then, the gradients on new data can be decomposed into the low-dimensional space, enabling the parameter update smoothed and relieving the sparsity. Note that this technology could be deployed into a distributed system to ensure efficient online learning on the trillions-level parameters. We conduct extensive experiments to evaluate the algorithm in terms of prediction accuracy and actual revenue. Experimental results demonstrate that the proposed framework achieves a promising prediction on the new data. The final online revenue is noticeably improved compared to the baseline. This framework was initially deployed in Baidu Search Ads (a.k.a. Phoenix Nest) in 2014 and is currently still being used in certain modules of Baidu's ads systems.
Student Can Also be a Good Teacher: Extracting Knowledge from Vision-and-Language Model for Cross-Modal Retrieval

Astounding results from transformer models with Vision-and Language Pretraining (VLP) on joint vision-and-language downstream tasks have intrigued the multi-modal community. On the one hand, these models are usually so huge that make us more difficult to fine-tune and serve real-time online applications. On the other hand, the compression of the original transformer block will ignore the difference in information between modalities, which leads to the sharp decline of retrieval accuracy. In this work, we present a very light and effective cross-modal retrieval model compression method. With this method, by adopting a novel random replacement strategy and knowledge distillation, our module can learn the knowledge of the teacher with nearly the half number of parameters reduction. Furthermore, our compression method achieves nearly 130x acceleration with acceptable accuracy. To overcome the sharp decline in retrieval tasks because of compression, we introduce the co-attention interaction module to reflect the different information and interaction information. Experiments show that a multi-modal co-attention block is more suitable for cross-modal retrieval tasks rather than the source transformer encoder block.
Automatic Error Correction Using the Wikipedia Page Revision History

Error correction is one of the most crucial and time-consuming steps of data preprocessing. State-of-the-art error correction systems leverage various signals, such as predefined data constraints or user-provided correction examples, to fix erroneous values in a semi-supervised manner. While these approaches reduce human involvement to a few labeled tuples, they still need supervision to fix data errors. In this paper, we propose a novel error correction approach to automatically fix data errors of dirty datasets. Our approach pretrains a set of error corrector models on correction examples extracted from the Wikipedia page revision history. It then fine-tunes these models on the dirty dataset at hand without any required user labels. Finally, our approach aggregates the fine-tuned error corrector models to find the actual correction of each data error. As our experiments show, our approach automatically fixes a large portion of data errors of various dirty datasets with high precision.
k-sums Clustering: A Stochastic Optimization Approach

In this paper, we revisit the decades-old clustering method k -means. The egg-chicken loop in traditional k -means has been replaced by a pure stochastic optimization procedure. The optimization is undertaken from the perspective of each individual sample. Different from existing incremental k -means, an individual sample is tentatively joined into a new cluster to evaluate its distance to the corresponding new centroid, in which the contribution from this sample is accounted. The sample is moved to this new cluster concretely only after we find the reallocation makes the sample closer to the new centroid than it is to the current one. Compared with traditional k -means and other variants, this new procedure allows the clustering to converge faster to a better local minimum. This fundamental modification over the k -means loop leads to the redefinition of a family of k -means variants, such as hierarchical k -means, and Sequential k -means. As an extension, a new target function that minimizes the summation of pairwise distances within clusters is presented. Under l2-norm, it could be solved under the same stochastic optimization procedure. The re-defined traditional k -means, hierarchical k -means, as well as Sequential k-means all show considerable performance improvement over their traditional counterparts under different settings and on various types of datasets.
Robust Dynamic Clustering for Temporal Networks

Dynamic community detection (or graph clustering) in temporal networks has attracted much attention because it is promising for revealing the underlying mechanism of complex real-world systems. Current methods are criticized for the independence of graph representation learning and graph clustering, considerable noise during temporal information smoothing, and high time complexity. We propose a R obust T emporal S moothing C lustering method (RTSC), which involves joint graph representation learning and graph clustering, to solve these problems. RTSC can be formulated as a constrained multi-objective optimization problem. Specifically, three-order successive snapshots are first projected into the same subspace via graph embedding. We then use the embedding matrices to learn a common low-rank block-diagonal matrix that contains current clustering information and specific noise matrices with a sparse constraint to remove noise at each time step. To efficiently solve the challenging optimization problem, we also propose an optimization procedure based on the augmented Lagrangian multiplier (ALM) scheme. Experimental results on six artificial datasets and four real-world dynamic network datasets indicate that RTSC performs better than six state-of-the-art algorithms for dynamic clustering in temporal networks.
Toward an Effective Black-Box Adversarial Attack on Functional JavaScript Malware against Commercial Anti-Virus

Machine learning has been a rising technique in signatureless malware detection and is popular in the anti-virus industry. Despite the powerful ability of machine learning, it is known to be vulnerable to attack by injecting specially crafted input noise (adversarial example). In this paper, we develop a systematic attack method that is effective, general and also efficient which automatically generates functional malware. Experiment results showed that such adversarial malware could deceive commercial anti-virus and completely defeat learning-based malware detector provided by a well-known anti-virus vendor. We further examine the effectiveness of our approach on multiple anti-virus engines on VirusTotal and investigate the transferability of our proposed method between different features and classification algorithms. Finally, we show how our attack could resist JavaScript de-obfuscation techniques.
SMAD: Scalable Multi-view Ad Retrieval System for E-Commerce Sponsored Search

Ad retrieval in sponsored search aims to understand user search intentions (user queries) and retrieves a set of ads inferred as being relevant to the queries. Due to the huge amount of search traffic and multiple views of relevance (such as co-clicking, co-bidding or textual similar), it is highly desirable but remain challenging to achieve a large-scale, multi-view matching between queries and ads, particularly in industrial settings. In this paper, we propose a scalable multi-view ad retrieval engine SMAD that we developed and deployed at Taobao, the largest e-commerce platform in China. We construct a multi-relation query-item-ad graph capturing different views of query-ad relevance, which is of large scale and with complex structure. Since in e-commerce platform, the queries and products are organized into a category tree, to deal with the large scale of the graph, we propose a category constrained graph sampling and partition method to enable distributed parallel offline training. To tackle the complex multi-view structure, we propose a multi-view parallel deep neural network (DNN) model to combine the information from different views in a principled way. According to offline experiments and online A/B tests, our framework significantly outperforms baselines in terms of relevance, coverage, and revenue.
Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.
HAO Unity: A Graph-based System for Unifying Heterogeneous Data

Many real-world applications have to face the problem of diversity in data formats and semantics. Currently, how to deal with heterogeneous data effectively is still a big challenge. With the rise of knowledge graphs, more and more applications are built upon graph-like data models, which benefit from flexible schemas and convenient support for relationship queries. We propose a graph-based unifying system for heterogeneous data unification, which helps to (1) transform data in many other formats into graphs, or conversely, from graph to other formats, (2) integrate graph data based on HAO intelligence, which achieves schema integration and entity consolidation, and (3) explore data at different levels via querying the integrated graphs. In this paper, we introduce the overall system architecture, explain in detail the implementation, and display the usage in two practical scenarios.
ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning

Anomaly detection on graphs plays a significant role in various domains, including cybersecurity, e-commerce, and financial fraud detection. However, existing methods on graph anomaly detection usually consider the view in a single scale of graphs, which results in their limited capability to capture the anomalous patterns from different perspectives. Towards this end, we introduce a novel graph anomaly detection framework, namely ANEMONE, to simultaneously identify the anomalies in multiple graph scales. Concretely, ANEMONE first leverages a graph neural network backbone encoder with multi-scale contrastive learning objectives to capture the pattern distribution of graph data by learning the agreements between instances at the patch and context levels concurrently. Then, our method employs a statistical anomaly estimator to evaluate the abnormality of each node according to the degree of agreement from multiple perspectives. Experiments on three benchmark datasets demonstrate the superiority of our method.
SPARQL-vision: A Platform for Querying, Visualising and Exploring SPARQL endpoints

The wide adaptation of the Semantic Web and the Resource Description Framework (RDF) has made available many important datasets. The SPARQL query language facilitates the exploration of this information, which is available in a semi-structured way that does not comply with relational data models, deviating from exploration techniques that most researchers are familiar with. Usually, only people with training and extensive knowledge of the RDF model can explore and understand it in depth. We present here a platform that supports the users with querying, exploring and visualizing information available in SPARQL endpoints. A dedicated visualization module, built upon a knowledge database, allows us to provide case-specific visualization solutions for SPARQL query results. The selection is based exclusively on features extracted from the result, without any knowledge about the structure, content and characteristics of the underlying dataset.
Fast and Accurate Anchor Graph-based Label Prediction

Anchor graphs are a popular tool used in label prediction of sparsely labeled data. In anchor graphs, labels of labeled data are propagated to unlabeled data via anchor points; anchor points are the centers of k-means clusters. Anchor graph-based label prediction determines local weights between data points and anchor points by exploiting Nesterov's method to obtain the graph's adjacency matrix, and it inverts a matrix obtained from the adjacency matrix to predict labels. , however, incurs high computation cost since (1) Nesterov's method is applied to all closest anchor points to compute local weights, and (2) the computation cost of the inversion matrix is cubic in the number of anchor points. We propose an approach that can efficiently perform anchor graph-based label prediction because of its two key advances: (1) it prunes unnecessary anchor points so they are not passed to Nesterov's method, and (2) it applies the conjugate gradient method in computing labels of data points to avoid matrix inversion. In addition, we propose to exploit basis vectors computed by SVD as anchor points to improve label prediction accuracy. Experiments show that our approach outperforms the previous approaches in terms of efficiency and accuracy.
Relation Prediction via Graph Neural Network in Heterogeneous Information Networks with Missing Type Information

Relation prediction is a fundamental task in network analysis which aims to predict the relationship between two nodes. Thus, this differes from the traditional link prediction problem predicting whether a link exists between a pair of nodes, which can be viewed as a binary classification task. However, in the heterogeneous information network (HIN) which contains multiple types of nodes and multiple relations between nodes, the relation prediction task is more challenging. In addition, the HIN might have missing relation types on some edges and missing node types on some nodes, which makes the problem even harder. In this work, we propose RPGNN, a novel relation prediction model based on the graph neural network (GNN) and multi-task learning to solve this problem. Existing GNN models for HIN representation learning usually focus on the node classification/clustering task. They require the type information of all edges and nodes and always learn a weight matrix for each type, thus requiring a large number of learning parameters on HINs with rich schema. In contrast, our model directly encodes and learns relations in HIN and avoids the requirement of type information during message passing in GNN. Hence, our model is more robust to the missing types for the relation prediction task on HINs. The experiments on real HINs show that our model can consistently achieve better performance than several state-of-the-art HIN representation learning methods.
SGCL: Contrastive Representation Learning for Signed Graphs

Graph contrastive representation learning aims to learn discriminative node representations by contrasting positive and negative samples. It helps models learn more generalized representations to achieve better performances on downstream tasks, which has aroused increasing research interest in recent years. Simultaneously, signed graphs consisting of both positive and negative links have become ubiquitous with the growing popularity of social media. However, existing works on graph contrastive representation learning are only proposed for unsigned graphs (containing only positive links) and it remains unexplored how they could be applied to signed graphs due to the distinct semantics and complex relations between positive and negative links. Therefore we propose a novel Signed Graph Contrastive Learning model (SGCL) to bridge this gap, which to the best of our knowledge is the first research to employ graph contrastive representation learning on signed graphs. Concretely, we design two types of graph augmentations specific to signed graphs based on a significant signed social theory, i.e., balance theory. Besides, inter-view and intra-view contrastive learning are proposed to learn discriminative node representations from perspectives of graph augmentations and signed structures respectively. Experimental results demonstrate the superiority of the proposed model over state-of-the-art methods on both real-world social datasets and online game datasets.
Centerpoint Query Authentication

The rise of online map services drives data owners to outsource spatial data to potentially untrusted database providers. Query results are provided along with verification objects that allow confirming their authenticity. Such authentication schemes have been proposed for several spatial and geometric queries, as well as for median queries in one dimension. However, to date, no authentication mechanism exists for centerpoint queries, which return a point lying in the middle of other points in multidimensional space. In this paper, we propose an authentication scheme for centerpoint queries, grounded on the algorithm for centerpoint queries on a finite planar set of points and authenticated aggregation R-trees and accompanying authenticated aggregation queries. We also provide methods for finding the centerpoint of a subset of the complete data set, and implement a range-based method. Our solution has a worst-case time-complexity of O(n log n) and space-complexity of O(n). Our experimental study confirms these claims.
ScarceGAN: Discriminative Classification Framework for Rare Class Identification for Longitudinal Data with Weak Prior

This paper introduces ScarceGAN which focuses on identification of extremely rare or scarce samples from multi-dimensional longitudinal telemetry data with small and weak label prior. We specifically address: (i) severe scarcity in positive class, stemming from both underlying organic skew in the data, as well as extremely limited labels; (ii) multi-class nature of the negative samples, with uneven density distributions and partially overlapping feature distributions; and (iii) massively unlabelled data leading to tiny and weak prior on both positive and negative classes, and possibility of unseen or unknown behavior in the unlabelled set, especially in the negative class. Although related to PU learning problems, we contend that knowledge (or lack of it) on the negative class can be leveraged to learn the compliment of it (i.e., the positive class) better in a semi-supervised manner. To this effect, ScarceGAN re-formulates semi-supervised GAN by accommodating weakly labelled multi- class negative samples and the available positive samples. It relaxes the supervised discriminator's constraint on exact differentiation be- tween negative samples by introducing a 'leeway' term for samples with noisy prior. We propose modifications to the cost objectives of discriminator, in supervised and unsupervised path as well as that of the generator. For identifying risky players in skill gaming, this formulation in whole gives us a recall of over 85% (~60% jump over vanilla semi-supervised GAN) on our scarce class with very minimal verbosity in the unknown space. Further ScarceGAN out- performs the recall benchmarks established by recent GAN based specialized models for the positive imbalanced class identification and establishes a new benchmark in identifying one of rare attack classes (0.09%) in the intrusion dataset from the KDDCUP99 challenge. We establish ScarceGAN to be one of new competitive benchmark frameworks in the rare class identification for longitudinal telemetry data.
epl draft SU(3) Landau-Zener Interferometry

We propose a universal approach to Landau-Zener problem in a three-level system. The problem is formulated in terms of Gell-Mann operators which generate SU(3) algebra and map the Hamiltonian on the effective anisotropic pseudospin 1 model. The vector Bloch equation for the density matrix describing the temporal evolution of three-level crossing problem is also derived and solved analytically for the case where the diabatic states of the SU(3) Hamiltonian form a triangle. This analytic solution is in excellent quantitative agreement with numerical solution of Schrödinger equation for a 3-level crossing problem. The model demonstrates oscillation patterns which radically differ from the standard patterns for two-level Landau-Zener problem. The triangle works as an interferometer and the interplay between two paths results in formation of ”beats” and ”steps” pattern in the time-dependent transition probability. The characteristic time scales describing the ”beats” and ”steps” depend on a dwell times in the triangle. These scales are related to the geometric size of interferometer. The possibilities of experimental realization of this effect in triple quantum dots and in two-well traps for cold gases are discussed.
Support Vector Machines and Kernels

This is a rather striking result, since the vectors x1, . ,xN do not generally span all of R, especially if you consider that it is entirely possible that there are fewer data points than dimensions, N d. For instance, if the data points xn are color images, they can have millions of pixels, with each pixel color represented by three numbers, so d is in the many millions in that case. The number N of training images, on the other hand, is often much smaller. While the representer theorem may seem to be just a theoretical curiosity, we will see soon that it has profound practical consequences, especially when combined with the fact that a large fraction of the βn turn out to be zero. Let us first formulate and prove the result itself. The representer theorem holds for a broader class of SVMs than we have seen so far, namely, it holds for all so-called kernel SVMs, which we will look at later on. In order to be able to use this theorem in this broader setting, we first abstract the structure of the SVM optimization problem in equations 2, 3 into more general notation, and prove the theorem for this more general formulation.
OXIDATIVE DNA BASE DAMAGE IN MOUSE EXTRACTS

The DNA is a relatively unstable molecule that can be easily modified (Lindahl, 1993). Modifications of DNA bases, particularly those caused by oxidative stress, can lead to mutations and have been directly linked with some diseases such as cancer. DNA lesions introduced by oxidative stress also accumulate during the normal aging process. To counteract their deleterious effects, living organisms have developed DNA repair systems that remove the damaged base and replace it with a new, unmodified nucleotide. DNA repair is a set of enzymatic processes that work jointly to correct DNA damage. These modifications are recognized by specific proteins and then cascades of enzymatic reactions take place. The particular process and set of enzymes utilized is then dependent upon the kind of lesion that is being dealt with. Small, non-helix distorting lesions, such as oxidative modifications, are repaired by the base excision repair pathway (BER). BER is active both in the nucleus as well as in mitochondria, and thus functions to prevent accumulation of oxidative DNA damage in both genomes (Bohr et al., 2002).