Reviewerly/exHarmony huggingface dataset:
Evaluating Captioning Models using Markov Logic Networks

Multimodal problems such as caption generation advances AI as a whole since they require integration of several key domains such as computer vision, NLP and knowledge representation. In this paper, we develop a new approach to evaluate captioning models by verifying them using Markov Logic Networks (MLNs). Specifically, we compile an MLN from training data and perform probabilistic inference to estimate uncertainty in a generated caption. To reify the caption, we leverage advances in Natural Language Inference (NLI) models and convert a caption into a query for the MLN. Further, we add visual context into the MLN distribution using an attention-based Multiple Instance Learning model and evaluate a caption based on this augmented distribution. We perform experiments using MSCOCO on several state-of-the-art benchmarks and show that our approach can evaluate captioning models just as effectively as methods that require human-generated captions.
Trade less Accuracy for Fairness and Trade-off Explanation for GNN

Graphs are widely found in social network analysis and e-commerce, where Graph Neural Networks (GNNs) are the state-of the-art model. GNNs can be biased due to sensitive attributes and network topology. With existing work that learns a fair node representation or adjacency matrix, achieving a strong guarantee of group fairness while preserving prediction accuracy is still challenging, with the fairness-accuracy trade-off remaining obscure to human decision-makers. We first define and analyze a novel upper bound of group fairness to optimize the adjacency matrix for fairness without significantly h arming prediction accuracy. To understand the nuance of fairness-accuracy tradeoff, we further propose macroscopic and microscopic explanation methods to reveal the trade-offs and the space that one can exploit. The macroscopic explanation method is based on stratified sampling and linear programming to deterministically explain the dynamics of the group fairness and prediction accuracy. Driving down to the microscopic level, we propose a path-based explanation that reveals how network topology leads to the tradeoff. On seven graph datasets, we demonstrate the novel upper bound can achieve more efficient fairness-accuracy trade-offs and the intuitiveness of the explanation methods can clearly pinpoint where the trade-off is improved.
Causal Discovery for Feature Selection in Physical Process-Based Hydrological Systems

Physical process-based hydrological models are widely adopted to simulate the water quantity or quality. One of the most commonly used hydrological models is Soil and Water Assessment Tool (SWAT). SWAT models for a large watershed can have over tens of thousands of Hydrological Resource Units (HRUs) which necessitates considerable computational resources. One way to speed up applications of the SWAT model could be to leverage machine learning techniques to identify the crucial features for the prediction task – feature selection. However, majority of the feature selection techniques rely on correlations or some form of a score metric (e.g. mutual information). Furthermore, since correlation does not imply causation, it is important to identify the causal features to improve the prediction accuracy while enhancing the interpretability of machine learning models. However, the SWAT model uses multiple data inputs and features that typically vary by space/HRUs, but may or may not vary over time. This makes it difficult to directly utilize causal discovery models to infer the causal relations. Furthermore, due to the lack of the ground truth causal graph for the SWAT model it is difficult to comment on the validity of the learned causal relations. To overcome these problems, we propose a novel framework that first infers the causal relations for the daily scale of the SWAT data using causal discovery algorithms. Then, it utilizes a community detection module to group similar features together for better interpretability. Finally, it identifies the stable causal relations that appear most often across all the timesteps and leverage them for the prediction of the water quantity. By utilizing only the causal features for the prediction of the target variable can lead to high accuracy as it removes the reliance on spurious correlations. Furthermore, we conduct extensive experiments to validate the effectiveness of the proposed framework along with a real-world case study to evaluate whether the selected features are interpretable or not.
Wikidata-lite for Knowledge Extraction and Exploration

Wikidata is the largest collaborative general knowledge graph supported by a worldwide community. It includes many helpful topics for knowledge exploration and data science applications. However, due to the enormous size of Wikidata, it is challenging to retrieve a large amount of data with millions of results, make complex queries requiring large aggregation operations, or access too many statement references. This paper introduces our preliminary works on Wikidata-lite, a toolkit to build a database offline for knowledge extraction and exploration, e.g., retrieving item information, statements, provenances, or searching entities by their keywords, attributes. Wikidata-lite has high performance and memory efficiency, much faster than the official Wikidata SPARQL endpoint for big queries. The Wikidata-lite repository is available at https://github.com/phucty/wikidb.
Exploring the Target Distribution for Surrogate-Based Black-Box Attacks

Deep Neural Networks are shown to be prone to adversarial attacks. In the black-box setting, where no information about the target is available, surrogate-based black-box attacks train a surrogate on samples queried from the target to imitate the black-box’s behavior. The trained surrogate is then attacked to generate adversarial examples. Existing surrogate-based attacks suffer from low success rates because they fail to accurately capture the target’s behavior, i.e., their surrogates only mimic the target’s outputs for a given set of inputs. Moreover, their attack strategy relies on noisy estimations of high dimensional gradients w.r.t. the inputs (i.e., surrogate’s gradients) to generate adversarial examples. Ideally, a successful surrogate-based attack should possess two properties: (1) Train and employ a surrogate that accurately imitates the target behavior for every pair of input and output, i.e., the joint distribution of the target over its input and outputs; and (2) Generate adversarial examples by directly manipulating the class-dependent factors of the input, i.e., factors that affect the target’s output, rather than relying on noisy estimations of gradients. We propose a novel surrogate-based attack framework with a surrogate architecture that learns the target distribution over its inputs and outputs while disentangling the class-dependent factors from class-irrelevant ones. The framework is equipped with a novel attack strategy that fully utilizes the target distribution captured by the surrogate while generating adversarial examples by directly manipulating the class-dependent factors. Extensive experiments demonstrate the efficacy of our attack in generating highly successful adversarial examples compared to state-of-the-art methods.
Continual Learning with Network Intrusion Dataset

Deep learning-based cybersecurity applications should be able to continually accumulate threat knowledge for new types of threats over time while maintaining the knowledge of threats already exposed to the application. This paper proposes episodic memory management for continual learning with network intrusion datasets. For new attacks, the number of samples may not be sufficiently large for training, and thus the memory management algorithm should retain as many samples as possible instead of random sampling in the episodic memory for continual learning. The experiment results indicated that the proposed algorithm outperforms offline learning in terms of average per-class accuracy in a continual scenario with a network intrusion dataset.
Flexibly Mining Better Patterns

Correlated high-utility pattern mining (CoUPM) considers the correlation between items in a pattern and offers a more reliable analysis for users. In real-world applications, the discovered patterns from CoUPM can present more interpretable information, but not all of them are useful. Generally, users pay attention to the number of items a pattern contains, which allows them to make reasonable decisions. In this paper, we solve the problem of mining those correlated high-utility patterns whose length is specified. A utility-list-based algorithm called Flexible Correlated Utility-based Pattern (FCoUP) is proposed. Furthermore, we propose some pruning strategies with the designed upper bounds for the two evaluation metrics: correlation and utility, reducing unwanted patterns generated and nodes visited during the mining process. Experiments show that FCoUP variants can produce more intelligent and flexible correlated high-utility patterns on a variety of datasets.
Optimizing the Descendant-Aware Clustering Parameters

Topic evolution is a recently introduced field of research as a substitute for a more traditional text-based topic evolution, allowing the tracking of more complex evolutionary events with the use of network structures. Network-based topic evolution showed that the neighborhood characteristics of newly introduced topics can be utilized to determine when a topic would emerge in a given domain. Predicting emerging topics requires a method for generating pseudo-neighbors of previously unseen topics as the neighborhood for an emerging topic is not known before its appearance. The authors proposed the Descendant-Aware Clustering algorithm to generate a set of neighborhood candidates for future emerging topics, surpassing existing algorithms in both performances and computation times. Optimizing the algorithm parameters enhances the performance even further. Significant performance improvements were observed when NSGA-III multi-objective algorithm was applied to over 100 research domains. A set of enhanced default values are introduced to the proposed algorithm removing the necessity for dataset-specific optimization, cementing the position of the Descendant-Aware Clustering as the best clustering algorithm for detecting ancestors of future emerging topics.
OSP-Class: Open Set Pseudo-labeling with Noise Robust Training for Text Classification

Although text classification performance has been greatly improved with the capability of deep learning based language models, the cost of building labeled training data is still a significant burden. To address this issue, pseudo-labeling based weakly supervised text classification models have attracted significant attention. However, prior works on weakly-supervised classification usually assume that each unlabeled sample belongs to one and only one of the target classes. Such an assumption may not be appropriate or realistic for open set scenarios where the unlabeled data are obtained via a coarse collection process like crowdsourcing. This paper studies weakly-supervised text classification under the open set scenarios, relaxing such a closed set assumption and allowing for out-of-label-space data. We propose a simple combination framework of multiple existing techniques called OSP-Class, an Open Set Pseudo-labeling with noise robust training for text Classification, which is designed to generate a category-specific denoising model that can remove text corpus containing unknown data by jointly combining (1) category vocabulary expansion with pre-trained BERT masked language model, (2) binary pseudo-labeling with weighted random oversampling, and (3) noise robust training with NCE+MAE loss function. Our model achieves the highest F1 score with an average of 92.49% in the open set environment. We demonstrate that the proposed method is comparable and slightly improved performance with some minor settings even if it does not use fundamentally innovative or different techniques.
Decomposing User-APP Graph into Subgraphs for Effective APP and User Embedding Learning

APP-installation information is helpful to describe users’ characteristics. Users with similar APPs installed might share common interests and behave similarly in some scenarios. In this work, we learn a user embedding vector based on each user’s APP-installation information. Since the user APP-installation embedding is learnable without dependency on the historical intra-APP behavioral data of the user, it complements the intra-APP embedding learned within each specific APP. Thus, they considerably help improve the effectiveness of the personalized advertising in each APP, and they are particularly beneficial for the cold start of the new users in the APP. In this paper, we formulate the APP-installation user embedding learning into a bipartite graph embedding problem. The main challenge in learning an effective APP-installation user embedding is the imbalanced data distribution, as graph learning tends to be dominated by the popular APPs installed by billions of users. In comparison, niche/specialized APPs might have a marginal influence o n g raph l earning. To e ffectively e xploit t he valuable information from niche APPs, we decompose the APP-installation graph into a set of subgraphs, each containing only one APP node and the users who install the APP. For each mini-batch, we only sample the users from the same subgraph in the training process. Thus, each APP can be involved in the training process in a more balanced manner. A considerable increase in CTR, CVR, and revenue has been observed after integrating the learned APP-installation user embedding into our online personal advertising platform. Additionally, the embeddings learned from our design can be efficiently searched and ranked by various embedding-based retrieval techniques, in particular, the fast neural ranking approach using bipartite graphs [24] (also developed at Baidu Cognitive Computing Lab) would be naturally applicable.
Knowledge Tracing Based on Gated Heterogeneous Graph Convolutional Networks

The advancement of science and technology provides the possibility of personalized intelligent education. Representation learning of students’ behavior data is challenging because whether time sequences and interactive behaviors or the correlation between knowledge points and students carrying important information. Some researchers propose knowledge tracing to provide ideas for solving this dilemma. However, existing knowledge tracing methods are divided into machine learning and deep learning. Machine learning-based methods require manual feature extraction and a large amount of prior knowledge. Although deep learning-based methods can automatically extract features, most methods either only use the time series information of the data, or use the association between knowledge points. All the methods ignore the association between knowledge points and students. To fill this gap, we propose a Gated Heterogeneous Graph Convolutional Network (GHGCN) model. We utilize the encoder-decoder framework to predict student performance using the representations of nodes, which is learned from heterogeneous convolutional networks and gate recurrent unit. To validate the effectiveness of the proposed GHGCN model, we conduct the experiments on three public datasets: Simulated Data, Assistments 2009, and Assistments 2015. The results indicate that our method can achieve better performance compared with state-of-the-art algorithms.
Dynamic Ensemble Size Adjustment for Memory Constrained Mondrian Forest

Supervised learning algorithms generally assume the availability of enough memory to store data models during the training and test phases. However, this assumption is unrealistic when data comes in the form of infinite data streams, or when learning algorithms are deployed on devices with reduced amounts of memory. Such memory constraints impact the model behavior and assumptions. In this paper, we show that under memory constraints, increasing the size of a tree-based ensemble classifier can worsen its performance. In particular, we experimentally show the existence of an optimal ensemble size for a memory-bounded Mondrian forest on data streams and we design an algorithm to guide the forest toward that optimal number by using an estimation of overfitting. We tested different variations for this algorithm on a variety of real and simulated datasets, and we conclude that our method can achieve up to 95% of the performance of an optimally-sized Mondrian forest for stable datasets, and can even outperform it for datasets with concept drifts. All our methods are implemented in the OrpailleCC open-source library and are ready to be used on embedded systems and connected objects.
TgraphSpot: Fast and Effective Anomaly Detection for Time-Evolving Graphs

Given a large, time-evolving graph of who-calls-whom-when, how can we help analysts find anomalies and fraudsters? How can we explain our decisions? We provide TgraphSpot, which carefully extracts features that are often related to fraud; and which provides informative, interactive plots that help analysts zoom down to the few strange nodes. We present the architecture and design decisions of TgraphSpot. Thanks to our careful feature-extraction algorithms, it scales linearly, taking 2.5 hours on a stock laptop, to process 29 million phone calls. More importantly, when applied on a real dataset of millions of phone calls, it discovered suspicious nodes; experts confirmed that those nodes are fraudsters that had been undetected so far.
RAF: Recursive Adversarial Attacks on Face Recognition Using Extremely Limited Queries

Recent successful adversarial attacks on face recognition show that, despite the remarkable progress of face recognition models, they are still far behind human intelligence for perception and recognition. It reveals the vulnerability of deep convolutional neural networks (CNNs) as a state-of-the-art building block for face recognition models against adversarial examples, which can cause certain consequences for secure systems. Gradient-based adversarial attacks have been widely studied and proved successful against face recognition models. However, finding the optimized perturbation per each face needs to submit a significant number of queries to the target model. In this paper, we propose a recursive adversarial attack on face recognition using automatic face warping, which needs an extremely limited number of queries to fool the target model. Instead of a random face warping procedure, the warping functions are applied on specific detected regions of face like eyebrows, nose, lips, etc. We evaluate the robustness of the proposed method in the decision-based black-box attack setting, where the attackers have no access to the model parameters and gradients, but the target model provides hard-label predictions and confidence scores.
ADIU:An Antiquarian Document Image Unwarping Dataset

As a class of historical relics, the importance of antiquarian documents is self-evident. In the process of digitizing these documents, due to uncontrollable factors such as physical deformation of the paper and position offset during camera capture, the photos often show different degrees of geometric distortion. Most document image unwarping models (Fig. 2.) can achieve the restoration of geometric distortion of the antiquarian document images. However, there has been lack of ancient document image unwarping datasets in academia, which makes it more difficult to propose and optimize models. To solve this dilemma, we construct the Antiquarian Document Image Unwarping dataset (ADIU), which covers most of the geometric distortion types. To the best of our knowledge, this is the first comprehensive and sizable document restoration dataset that can remove geometric and appearance distortion from camera images of antiquarian documents significantly, which contains 11,000 antiquarian document images.
Personalized Federated Contrastive Learning

This paper studies the problem of developing contrastive learning into the privacy-protected federated learning (FL), which is to achieve more data samples for model training. The existing methods usually encourage the global model and local models in FL to be the same one, often ignoring the data heterogeneity of the clients. In this paper, we proposed a method of personalized federated contrastive learning to improve the FL model performance for each client’s task, by learning a global representation and a local representation simultaneously. Our method is a novel FL framework that borrows the scheme of contrastive learning (CL), where one CL branch is the global model while the other branch is the local model divided into a share part and a personalized part. The proposed model is then trained by maximizing the agreement between the global model and the sharing part of the local model and meanwhile minimizing the agreement between the global model and the personalized part. We conducted evaluations on three public datasets for federated image classification. The results show that the proposed method can benefit from the personalization of local models and thus achieve better accuracy in comparison with the state-of-the-art FL models.
Answer Comments As Reviews: Predicting Acceptance By Measuring Valence On Stack Exchange

Online communication has increased the need to interpret complex emotions rapidly; due to the volatility of the data involved, machine learning tasks that process text aim to address the related challenges. Exploring text in comments that supports ideas through computational methods is a logical next step considering similar research for these question-and-answer sites. A lack of current algorithms that can accurately predict accepted answers with equal votes suggests a gap in this knowledge. Measuring collaborative signals in comments finds common keywords to move a problem toward a solution.Using a dataset from questions posted to Stack Exchange on the subject area of machine learning, the researchers constructed a model using the comments of posts made on accepted answers. Intending to discover whether predictions of marked solutions are accurate by treating the comments as reviews, the researchers find a reduction in error by incorporating reviews of answers as a feature in the predictive algorithm.
Detecting languages in streetscapes using deep convolutional neural networks

Cities are laden with visual clues—public road signs, advertising billboards, street names, place names, street art, and commercial shop signs. In urban studies literature, it is common to use such clues, littered along streetscapes, to understand cities better. Sociologists have tried to derive meaning from the linguistic diversity in streetscapes, but their efforts are limited by the laborious effort of field s tudies. Taking a dvantage of the availability of street view imagery (SVI) today and the advent of deep learning driven by the availability of big data, we undertake a machine learning approach to quantify the linguistic diversity in cities. In this paper, we introduce a language detection tool to identify the presence of English, Swedish, Arabic and Chinese in streetscapes. The tool is built on a pretrained DenseNet-121 model and trained with both synthetic images, and real streetscapes scraped from Google Street View (GSV). We achieve a test accuracy of 80.8% across all four languages, surpassing the performance of best performing optical character recognition (OCR) tools by 9 percentage points. The F1 score of 79.8% also surpasses that of present OCR tools by 11 percentage points, which suggests that our model is both accurate and reliable. We use the gradient-weighted class activation map (Grad-CAM) to show that our model is visually interpretable. We then apply the tool to Stockholm, Sweden to create a portrait of the linguistic diversity of the city and find a chasm between linguistic diversity and population diversity, which points to the importance of measuring the two measures separately.