Reviewerly/exHarmony huggingface dataset:
Twitter Bot Detection Through Unsupervised Machine Learning

Identification a nd r emoval o f f ake internet accounts have become crucial for internet safety. Much of existing research for bot detection uses supervised machine learning, in addition to detecting URL usage and sentiment analysis. Due to the limited availability of labeled bot data and the difficulties s upervised a lgorithms f ace i n detecting adaptive behavior, an unsupervised approach would be more appropriate. Past studies have attempted this by clustering accounts based on post activity and detecting spam through embedded URLs. Our research attempts to detect a broader range of bots through K-Means and Agglomerative clustering using account activity, popularity, and Twitter verification, among others. After using the Scikit-learn Python library and collecting data from the Twitter API, we used a dataset of around 2,000 known bot and human accounts from the Bot Repository and 11,000 unlabelled accounts to create four distinct clusters. We then measured our results based on our confidence that an account in a particular cluster was a bot or a human.
CSK-Detector: Commonsense in object detection

We propose an approach CSK-Detector for object detection and image categorization, well-suited for big data, by transferring commonsense knowledge from a knowledge base, augmented with premises and quantifiers. It is implemented for domestic robotics, especially with the motivation that next-generation and multipurpose domestic robots should be able to seamlessly discern environments for specific tasks without prior annotation of excessive images. CSK-Detector is evaluated on real data, yielding better results than deep learning without commonsense, while also providing an explainable approach. It broadly impacts human-robot collaboration and smart living.
HateNet: A Graph Convolutional Network Approach to Hate Speech Detection

The COVID-19 pandemic has caused hate speech on online social networks to become a growing issue in recent years, affecting millions. Our work aims to improve automatic hate speech detection to prevent escalation to hate crimes. The first c hallenge i n h ate s peech r esearch i s t hat e xisting datasets suffer from quite severe class imbalances. The second challenge is the sparsity of information in textual data. The third challenge is the difficulty i n b alancing t he t radeoff b etween utilizing semantic similarity and noisy network language. To combat these challenges, we establish a framework for automatic short text data augmentation by using a semi-supervised hybrid of Substitution Based Augmentation and Dynamic Query Expansion (DQE), which we refer to as SubDQE, to extract more data points from a specific c lass f rom T witter. W e a lso p ropose the HateNet model, which has two main components, a Graph Convolutional Network and a Weighted Drop-Edge. First, we propose a Graph Convolutional Network (GCN) classifier, using a graph constructed from the thresholded cosine similarities between tweet embeddings to provide new insights into how ideas are connected. Second, we propose a weighted Drop-Edge based stochastic regularization technique, which removes edges randomly based on weighted probabilities assigned by the semantic similarities between Tweets. Using 3 different SubDQE-augmented datasets, we compare our HateNet model using eight different tweet embedding methods, six other baseline classification models, and seven other baseline data augmentation techniques previously used in the realm of hate speech detection. Our results show that our proposed HateNet model matches or exceeds the performance of the baseline models, as indicated by the accuracy and F1 score.
Outlier Detection Based on Stacked Autoencoder and Gaussian Mixture Model

The outlier detection of high-dimensional data is still of challenge. The performance of existing unsupervised approaches will be affected with the increase of outliers in a dataset. The stacked autoencoder and GMM are introduced to the detection of outliers, and an approach termed SAGMM is proposed. The stacked autoencoder can reduce the reconstruction error of observations, and the GMM determines the outliers based on the mixture distributions of observations obtained in model training. Experiments on public datasets show that the proposed approach SAGMM outperforms the similar approaches in precision, and has a good balance between the precision and recall rate, since it improves the F1 scores compared with them.
Machine Learning Platform for Extreme Scale Computing on Compressed IoT Data

With the lowering costs of sensors, high-volume and high-velocity data are increasingly being generated and analyzed, especially in IoT domains like energy and smart homes. Consequently, applications that require accurate short-term forecasts and predictions are also steadily increasing. In this paper, we provide an overview of a novel end-to-end platform that provides efficient ingestion, compression, transfer, query processing, and machine learning-based analytics for high-frequency and high-volume time series from IoT. The performance of the platform is evaluated using real-world dataset from RES installations. The results show the importance of high-frequency analytics and the surprisingly positive impact of error bounded lossy compression on machine learning in the form of AutoML. For example, when detecting yaw misalignments in wind turbines, an improvement of 9% in accuracy was observed for AutoML models on lossy compressed data compared to the current industry standard of 10-minute aggregated data. Thus, these small-scale experiments show the potential of the platform, and larger pilots are planned.
ParaGraph: Mapping Wikidata Tail Entities to Wikipedia Paragraphs

Bridging unstructured data with knowledge bases is an essential task in many problems related to natural language understanding. Traditionally, this task is considered in one direction only: linking entity mentions in a text to their counterpart in a knowledge base (also known as entity linking). In this paper, we propose to tackle this problem from a different angle: linking entities from a knowledge base to paragraphs describing those entities. We argue that such a new perspective can be beneficial to several applications, including information retrieval, knowledge base population, and joint entity and word embedding. We present a transformer-based model, ParaGraph, which, given a Wikidata entity as input, retrieves its corresponding Wikipedia section. To perform this task, ParaGraph first generates an entity summary and compares it to sections to select an initial set of candidates. The candidates are then ranked using additional information from the entity’s textual description and contextual information. Our experimental results show that ParaGraph achieves 87% Hits@10 when ranking Wikipedia sections given a Wikidata entity as input. The obtained results show that ParaGraph can reduce the information gap between Wikipedia-based entities and tail entities and demonstrate the effectiveness of our proposed approach towards linking knowledge graph entities to their text counterparts.
Functional Ball Dropping: A superfast hypergraph generation scheme

We present a straightforward paradigm for designing simple, efficient generators for a wide variety of hypergraph models. This approach yields asymptotically optimal algorithms and concise implementations that outperform existing implementations by multiple orders of magnitude. Our generalization of ball dropping, called functional ball dropping, allows us to generate models that have not been previously treated by ball dropping. We identify natural building blocks of a given hypergraph model (e.g an edge) that can be concatenated to form a sensible representation of hypergraph (e.g. a list of edges). Next, we design a sampler subject to two constraints: the sampler’s resulting distribution of hypergraphs must match the model we generate, and the runtime of the sampler must be proportional to the size of the component it samples. Our algorithm incrementally generates the graph by repeated execution of the sampling function. In this paper, we present a general description of the functional ball dropping approach before applying it to several nontrivial graph models. We present arguments for correctness and complexity of each treatment. Following theoretical analysis, we perform empirical comparative runtime analysis that demonstrates the scalability of our implementations.
A Text Classification Method Based on BERT-Att-TextCNN Model

In text classification methods, the key to correct text classification is to extract semantic information accurately. However, important semantic information cannot be recognized, and existing models focus too much on local information and ignore global information often make the semantic bias, which in turn affects the classification results. To address these problems, this paper designs a text classification method based on BERT-Att-TextCNN model, which significantly improves the classification accuracy. First, this paper uses the feature of BERT using bidirectional architecture to obtain the accurate semantic representation of each word through multiple rounds of training iterative feedback, and then obtains the high-quality sentence vector. Secondly, this paper takes advantage of the attention mechanism in capturing important information to compute weights on the word vectors to enhance the semantic representation, and feeds the results into the TextCNN network to extract local key features. Then, the model in this paper fuses the hidden output of BERT with the pooled output of TextCNN to converge global semantic features and local semantic features to strengthen the judgment ability and robustness of the model. Finally, the classification results are obtained through the fully connected layer. Experiments show that this paper's method is tested on THUCNews, SST-2, and content security datasets, and the accuracy reaches 94.8%, 94.6%, and 99.4%, respectively, which is better than the classical classification model. The model in this paper has good generalization ability while improving the accuracy.
Obtaining Dyadic Fairness by Optimal Transport

Fairness has been taken as a critical metric in machine learning models, which is considered as an important component of trustworthy machine learning. In this paper, we focus on obtaining fairness for popular link prediction tasks, which are measured by dyadic fairness. A novel pre-processing methodology is proposed to establish dyadic fairness through data repairing based on optimal transport theory. With the well-established theoretical connection between the dyadic fairness for graph link prediction and a conditional distribution alignment problem, the dyadic repairing scheme can be equivalently transformed into a conditional distribution alignment problem. Furthermore, an optimal transport-based dyadic fairness algorithm called DyadicOT is obtained by efficiently solving the alignment problem, satisfying flexibility and unambiguity requirements. The proposed DyadicOT algorithm shows superior results in obtaining fairness compared to other fairness methods on two benchmark graph datasets.
Temporal Graph Representation Learning via Maximal Cliques

Graph Neural Networks (GNNs) have been proposed to learn graph representations for various graph mining tasks such as link prediction and node classification. These methods aggregate information from neighbors of a node to generate the node representation vector. Temporal GNN models consider the temporal and neighborhood information of nodes. However, few temporal GNN methods consider network substructures such as triads and cliques. In this paper, we present a temporal GNN-based method that generates node embeddings by aggregating neighbors of a node that exist in the maximal cliques of the graph containing the node. The reason for considering neighbors that form a maximal clique with the target node is that nodes in a maximal clique are highly connected to each other and most likely share similar characteristics. In addition, we consider the time dependency of nodes by generating temporal walks on the cliques such that in these walks the time order of the nodes is respected. The node embedding is based on the aggregation of the node’s temporal walks. Our experiments on seven datasets show the effectiveness of our method in both link prediction and node classification tasks. Furthermore, our method is faster than other baselines we evaluate.
Prequential Model Selection for Time Series Forecasting based on Saliency Maps

Over the last few years, incremental machine learning for streaming data has gained significant attention due to the need to learn from a constantly evolving stream of data without the need to store it. The advent of big data has further fuelled research on developing systems that can cope with continuously changing data streams and tackle the challenges associated with historical data requirements. The problem of time series forecasting has been studied using varied approaches like neural networks, ensemble methods, decision trees and rules, support vector machines to name a few. However, neural network models have gained particular attention in dealing with changing data distribution due to their generalization abilities. In this paper, we propose a prequential framework named PS-PGSM which involves incrementally training the base models and online Regions of Competence (ROC) computation followed by selection of the best forecaster for the task of time series forecasting using saliency maps. We build upon the state-of-the-art approach named OS-PGSM (Online Model Selection using Performance Gradient based Saliency Maps) in which the model training and ROC computation is performed offline. Past research has demonstrated that a set of different models enables specialization for each model compared to a single forecasting model which is particularly useful when predicting for an evolving time series sequence. Our approach uses saliency maps for prequential calculation of ROC for each model to find the best forecaster based on the performance of each model. We evaluate the proposed approach against OS-PGSM, as well as against previous best performing model by first conducting preliminary experiments on 10 real-world time series datasets and then using 2 real-world big datasets to showcase its applicability to big data. Experimental results not only validate the effectiveness of our approach for big data but also demonstrate superior performance in terms of prediction accuracy and computational time efficiency while also handling concept drift.
ClinicalLayoutLM: A Pre-trained Multi-modal Model for Understanding Scanned Document in Electronic Health Records

Scanned documents (e.g., faxes) are still widely used in clinical practice and are prevalent in Electronic Health Records (EHR). Unlocking information in scanned documents in EHRs is critical for clinical operation and research. However, it is challenging as it requires converting images to texts before applying information extraction technologies. Here we propose a multi-modal approach (ClinicalLayoutLM) that jointly models text extracted from Optical Character Recognition (OCR) and layout/image information to classify scanned clinical documents into different categories (e.g., lab reports and CT scans). Using a clinical corpus of 348, 311 scanned documents, we continually pretrained ClinicalLayoutLM based on LayoutLMv3, a multi-modal model from the open domain. For the task to classify the scanned clinical documents into 16 categories, ClinicalLayoutLM achieved an F1 score of 0.9051, which outperformed the baseline model (0.8840) that was based on text from OCR only. ClinicalLayoutLM is the first of its kind of multi-modal models for clinical documents and we believe it could benefit other clinical natural language processing (NLP) tasks such as layout analysis, information extraction and so on. The code is available at https://github.com/UTHealth-CCB/ClinicalLayoutLM and the pre-trained model is available upon request.
DL-BERT: a time-aware double-level BERT-style model with pre-training for disease prediction

Disease prediction based on the Electronic Health Record (EHR) is an important task in healthcare. EHR records patients’ every visit by time, and there are many kinds of medical codes within a visit, therefore, EHR has characteristics of temporal irregularity and hierarchical structure. Some recent works employ BERT-style models to process EHR data for disease prediction. However, few of these models can give consideration to capture both the interaction between medical codes and the impact of temporal irregularity. To solve this problem, we propose the Double-Level BERT-style model (DL-BERT). Considering EHR’s hierarchical structure, the model contains a code-level and a visit-level representation layer which can learn the relationship between medical codes and temporal influence respectively. In the code-level representation layer, the model achieves the representation power by employing external medical ontologies to provide multi-resolution information of medical codes and the Transformer to embed medical codes. Besides, the model adopts two pre-training tasks to enhance the ability to capture the link between different kinds of codes. In the visit-level representation layer, DL-BERT utilizes a special time-aware Transformer to model temporal information. And the model adopts a visit-level pre-training task for better learning context information. Experiments are conducted on two real-world healthcare datasets and show that our model outperforms all baselines demonstrating the effectiveness of our model.
HITS-GNN: A Simplified Propagation Scheme for Graph Neural Networks

In recent years, Graph Neural Networks (GNNs) have gained popularity for solving a wide range of problems, primarily due to the proliferation of graph data across various domains. GNNs offer expressive power but are computationally expensive at the same time. Some studies have suggested that altering their traditional message passing mechanism with Personalized PageRank as a propagation scheme reduces the computational complexity, improves performance, and optimizes scalability in semi-supervised learning problems. This paper presents a propagation mechanism based on the Hyperlink-Induced Topic Search (HITS) algorithm. The HITS-based approach propagates information in a graph by using a recursive update of authority and hub scores. Using this terminology, Personalized PageRank based propagation considers only in-links, thus, embraces the concept of authority (in-links) scores while ignoring the important concept of hub (out-links), which leads to trailing down some valuable information. According to our approach, a Multi-Layer Perceptron (MLP) is applied in combination with a HITS-based propagation algorithm to separate node prediction and propagation. Experimental results demonstrate that the proposed method outperforms baseline methods on graph benchmark datasets with a significant margin for semi-supervised node classification.
Applications of Integrated Gradients in Credit Risk Modeling

Deep Learning models are often called ‘Black Box’ models because of the difficulty in providing logical, actionable explanations of their decisions. Attribution methods try to solve this issue by determining the contribution of each feature to the decision. Recently, the Integrated Gradients method has shown promising results in variable attribution for differentiable models [1]. Integrated Gradients belong to the class of a more general attribution method called Path Integrated Gradients. Path Integrated Gradients is a parametric path-dependent and baseline-dependent attribution method. If the path is a straight line, the method becomes the canonical Integrated Gradients method. Choosing any other path leads to different attributions in general. Similarly, using different baselines may also lead to a change in attributions. Sundararajan et al. [1] recommend choosing a baseline that contains as little signal (discriminatory features used by the model for solving the task at hand) as possible, with the expectation that a comparison between the baseline and any input will reveal the signals of that input. In this paper, we show that choosing baselines with signals can also have very informative, distinct, and useful interpretations. We also propose two useful applications in the credit industry using two such baselines.
PaddleBox: Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising

Click-through rate (CTR) prediction is one of the most crucial components in the online advertising industry. In order to produce a personalized CTR prediction, an industry-level CTR prediction model commonly takes a high-dimensional (∼ 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">12</sup> ) sparse vector (that is encoded from query keywords, user portraits, etc.) as input. As a result, the model requires Terabyte scale parameters to embed the high-dimensional input. Hierarchical distributed GPU parameter server has been developed at Baidu to enable GPU with limited memory to train the massive network by leveraging CPU main memory and SSDs as secondary storage. In this work, we identify two major challenges in the existing GPU training framework for massive-scale ad models and propose a collection of optimizations to tackle these challenges: (a) the GPU, CPU, SSD rapidly communicate with each other during the training. The connections between GPUs and CPUs are non-uniform due to the hardware topology. The data communication route should be optimized according to the hardware topology; (b) GPUs in different computing nodes frequently communicates to synchronize parameters. It is thus required to optimize the communications so that the distributed system can become scalable. In this paper, we propose a hardware-aware training workflow that couples the hardware topology into the algorithm design. To reduce the extensive communication between computing nodes, we introduce a k-step model merging algorithm for Adam and provide its convergence rate in non-convex optimization. To the best of our knowledge, this is the first application of k-step adaptive optimization method in industrial CTR model training. Experiments on commercial search ads data confirm the effectiveness of our proposed training framework.
Fast Support Vector Machine Using Singular Value Decomposition

Nowadays, data is being generated very rapidly worldwide, and we need to analyze this data distributedly and efficiently. In this paper, we provide a method named SVDSVM that can compute linear support vector classification (SVM) distributedly and efficiently. SVDSVM reduces the computation and communication overhead of QRSVM (a recently proposed SVM solver) by replacing householder QR factorization with stochastic singular value decomposition. By removing additional gathering, scattering, and distributed computation, this method reduces the per iteration time of the dual ascent step and improves overall training time compared to QRSVM. Our evaluation with benchmark datasets shows that our method reduces per iteration time of dual ascent step around 5× compared to QRSVM, which results in an overall 50% time reduction of the total algorithm. However, as singular value decomposition is not entirely lossless, it results in a small accuracy drop which we found around 0.5-1.5% for different datasets.