OpenRaiser/Intern-Atlas huggingface dataset:
Uncertainty-aware Cardinality Estimation by Neural Network Gaussian Process
Venue: arXiv cs.DB 2021
Year: 2021

Deep Learning (DL) has achieved great success in many real applications. Despite its success, there are some main problems when deploying advanced DL models in database systems, such as hyper-parameters tuning, the risk of overfitting, and lack of prediction uncertainty. In this paper, we study cardinality estimation for SQL queries with a focus on uncertainty, which we believe is important in database systems when dealing with a large number of user queries on various applications. With uncertainty ensured, instead of trusting an estimator learned as it is, a query optimizer can explore other options when the estimator learned has a large variance, and it also becomes possible to update the estimator to improve its prediction in areas with high uncertainty. The approach we explore is different from the direction of deploying sophisticated DL models in database systems to build cardinality estimators. We employ Bayesian deep learning (BDL), which serves as a bridge between Bayesian inference and deep learning.The prediction distribution by BDL provides principled uncertainty calibration for the prediction. In addition, when the network width of a BDL model goes to infinity, the model performs equivalent to Gaussian Process (GP). This special class of BDL, known as Neural Network Gaussian Process (NNGP), inherits the advantages of Bayesian approach while keeping universal approximation of neural network, and can utilize a much larger model space to model distribution-free data as a nonparametric model. We show that our uncertainty-aware NNGP estimator achieves high accuracy, can be built very fast, and is robust to query workload shift, in our extensive performance studies by comparing with the existing approaches.
Towards Probabilistic Session-Type Monitoring
Venue: arXiv cs.PL 2021
Year: 2021

We present a tool-based approach for the runtime analysis of communicating processes grounded on probabilistic binary session types. We synthesise a monitor out of a probabilistic session type where each choice point is decorated by a probability distribution. The monitor observes the execution of a process, infers its probabilistic behaviour and issues warnings when the observed behaviour deviates from the one specified by the probabilistic session type.
CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software
Venue: arXiv cs.SE 2021
Year: 2021

Data-driven research on the automated discovery and repair of security vulnerabilities in source code requires comprehensive datasets of real-life vulnerable code and their fixes. To assist in such research, we propose a method to automatically collect and curate a comprehensive vulnerability dataset from Common Vulnerabilities and Exposures (CVE) records in the public National Vulnerability Database (NVD). We implement our approach in a fully automated dataset collection tool and share an initial release of the resulting vulnerability dataset named CVEfixes. The CVEfixes collection tool automatically fetches all available CVE records from the NVD, gathers the vulnerable code and corresponding fixes from associated open-source repositories, and organizes the collected information in a relational database. Moreover, the dataset is enriched with meta-data such as programming language, and detailed code and security metrics at five levels of abstraction. The collection can easily be repeated to keep up-to-date with newly discovered or patched vulnerabilities. The initial release of CVEfixes spans all published CVEs up to 9 June 2021, covering 5365 CVE records for 1754 open-source projects that were addressed in a total of 5495 vulnerability fixing commits. CVEfixes supports various types of data-driven software security research, such as vulnerability prediction, vulnerability classification, vulnerability severity prediction, analysis of vulnerability-related code changes, and automated vulnerability repair.
Experimental Investigation and Evaluation of Model-based Hyperparameter Optimization
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning algorithms such as random forests or xgboost are gaining more importance and are increasingly incorporated into production processes in order to enable comprehensive digitization and, if possible, automation of processes. Hyperparameters of these algorithms used have to be set appropriately, which can be referred to as hyperparameter tuning or optimization. Based on the concept of tunability, this article presents an overview of theoretical and practical results for popular machine learning algorithms. This overview is accompanied by an experimental analysis of 30 hyperparameters from six relevant machine learning algorithms. In particular, it provides (i) a survey of important hyperparameters, (ii) two parameter tuning studies, and (iii) one extensive global parameter tuning study, as well as (iv) a new way, based on consensus ranking, to analyze results from multiple algorithms. The R package mlr is used as a uniform interface to the machine learning models. The R package SPOT is used to perform the actual tuning (optimization). All additional code is provided together with this paper.
Finite-state source-channel coding for individual sequences with source side information at the decoder
Venue: arXiv cs.IT 2021
Year: 2021

We study the following semi-deterministic setting of the joint source-channel coding problem: a deterministic source sequence (a.k.a. individual sequence) is transmitted via a memoryless channel, using delay-limited encoder and decoder, which are both implementable by periodically-varying finite-state machines, and the decoder is granted with access to side information, which is a noisy version of the source sequence. We first derive a lower bound on the achievable expected distortion in terms of the empirical statistics of the source sequence, the number of states of the encoder, the number of states of the decoder, their period, and the overall delay. The bound is shown to be asymptotically achievable by universal block codes in the limit of long blocks. We also derive a lower bound to the best achievable excess--distortion probability and discuss situations where it is achievable. Here, of course, source coding and channel coding cannot be completely separated without loss of optimality. Finally, we outline a few extensions of the model considered, such as: (i) incorporating a common reconstruction constraint, (ii) availability of side information at both ends, and (iii) extension to the Shannon channel with causal state information at the encoder. This work both extends and improves on earlier work of the same flavor (Ziv 1980, Merhav 2014), which focused only on the expected distortion, without side information at either end, and without the above mentioned additional ingredients.
Auctions with Interdependence and SOS: Improved Approximation
Venue: arXiv cs.GT 2021
Year: 2021

Interdependent values make basic auction design tasks -- in particular maximizing welfare truthfully in single-item auctions -- quite challenging. Eden et al. recently established that if the bidders valuation functions are submodular over their signals (a.k.a. SOS), a truthful 4-approximation to the optimal welfare exists. We show existence of a mechanism that is truthful and achieves a tight 2-approximation to the optimal welfare when signals are binary. Our mechanism is randomized and assigns bidders only 0 or 0.5 probabilities of winning the item. Our results utilize properties of submodular set functions, and extend to matroid settings.
Simultaneous Speech Translation for Live Subtitling: from Delay to Display
Venue: arXiv cs.CL 2021
Year: 2021

With the increased audiovisualisation of communication, the need for live subtitles in multilingual events is more relevant than ever. In an attempt to automatise the process, we aim at exploring the feasibility of simultaneous speech translation (SimulST) for live subtitling. However, the word-for-word rate of generation of SimulST systems is not optimal for displaying the subtitles in a comprehensible and readable way. In this work, we adapt SimulST systems to predict subtitle breaks along with the translation. We then propose a display mode that exploits the predicted break structure by presenting the subtitles in scrolling lines. We compare our proposed mode with a display 1) word-for-word and 2) in blocks, in terms of reading speed and delay. Experiments on three language pairs (en$\rightarrow$it, de, fr) show that scrolling lines is the only mode achieving an acceptable reading speed while keeping delay close to a 4-second threshold. We argue that simultaneous translation for readable live subtitles still faces challenges, the main one being poor translation quality, and propose directions for steering future research.
One-Class Classification for Wafer Map using Adversarial Autoencoder with DSVDD Prior
Venue: arXiv cs.LG 2021
Year: 2021

Recently, semiconductors' demand has exploded in virtual reality, smartphones, wearable devices, the internet of things, robotics, and automobiles. Semiconductor manufacturers want to make semiconductors with high yields. To do this, manufacturers conduct many quality assurance activities. Wafer map pattern classification is a typical way of quality assurance. The defect pattern on the wafer map can tell us which process has a problem. Most of the existing wafer map classification methods are based on supervised methods. The supervised methods tend to have high performance, but they require extensive labor and expert knowledge to produce labeled datasets with a balanced distribution in mind. In the semiconductor manufacturing process, it is challenging to get defect data with balanced distribution. In this paper, we propose a one-class classification method using an Adversarial Autoencoder (AAE) with Deep Support Vector Data Description (DSVDD) prior, which generates random vectors within the hypersphere of DSVDD. We use the WM-811k dataset, which consists of a real-world wafer map. We compare the F1 score performance of our model with DSVDD and AAE.
Reinforcement Learning for Education: Opportunities and Challenges
Venue: arXiv cs.LG 2021
Year: 2021

This survey article has grown out of the RL4ED workshop organized by the authors at the Educational Data Mining (EDM) 2021 conference. We organized this workshop as part of a community-building effort to bring together researchers and practitioners interested in the broad areas of reinforcement learning (RL) and education (ED). This article aims to provide an overview of the workshop activities and summarize the main research directions in the area of RL for ED.
Time Series Anomaly Detection for Smart Grids: A Survey
Venue: arXiv cs.LG 2021
Year: 2021

With the rapid increase in the integration of renewable energy generation and the wide adoption of various electric appliances, power grids are now faced with more and more challenges. One prominent challenge is to implement efficient anomaly detection for different types of anomalous behaviors within power grids. These anomalous behaviors might be induced by unusual consumption patterns of the users, faulty grid infrastructures, outages, external cyberattacks, or energy fraud. Identifying such anomalies is of critical importance for the reliable and efficient operation of modern power grids. Various methods have been proposed for anomaly detection on power grid time-series data. This paper presents a short survey of the recent advances in anomaly detection for power grid time-series data. Specifically, we first outline current research challenges in the power grid anomaly detection domain and further review the major anomaly detection approaches. Finally, we conclude the survey by identifying the potential directions for future research.
Regular Perturbation and Achievable Rates of Space-Division Multiplexed Optical Channels
Venue: arXiv cs.IT 2021
Year: 2021

Regular perturbation is applied to space-division multiplexing (SDM) on optical fibers and motivates a correlated rotation-and-additive noise (CRAN) model. For S spatial modes, or 2S complex-alphabet channels, the model has 4S(S+1) hidden independent real Gauss-Markov processes, of which 2S model phase noise, 2S(2S-1) model spatial mode rotation, and 4S model additive noise. Achievable information rates of multi-carrier communication are computed by using particle filters. For S=2 spatial modes with strong coupling and a 1000 km link, joint processing of the spatial modes gains 0.5 bits/s/Hz/channel in rate and 1.4 dB in power with respect to separate processing of 2S complex-alphabet channels without considering CRAN.
VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
Venue: arXiv cs.LG 2021
Year: 2021

End-to-end AutoML has attracted intensive interests from both academia and industry, which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VolcanoML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VolcanoML introduces and implements basic building blocks that decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VolcanoML further supports a Volcano-style execution model - akin to the one supported by modern database systems - to execute the plan constructed. Our evaluation demonstrates that, not only does VolcanoML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn.
RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
Venue: arXiv cs.LG 2021
Year: 2021

Federated learning is a widely used distributed deep learning framework that protects the privacy of each client by exchanging model parameters rather than raw data. However, federated learning suffers from high communication costs, as a considerable number of model parameters need to be transmitted many times during the training process, making the approach inefficient, especially when the communication network bandwidth is limited. This article proposes RingFed, a novel framework to reduce communication overhead during the training process of federated learning. Rather than transmitting parameters between the center server and each client, as in original federated learning, in the proposed RingFed, the updated parameters are transmitted between each client in turn, and only the final result is transmitted to the central server, thereby reducing the communication overhead substantially. After several local updates, clients first send their parameters to another proximal client, not to the center server directly, to preaggregate. Experiments on two different public datasets show that RingFed has fast convergence, high model accuracy, and low communication cost.
Nonlinear photonic dynamical systems for unconventional computing
Venue: arXiv cs.ET 2021
Year: 2021

Driven by the remarkable breakthroughs during the past decade, photonics neural networks have experienced a revival. Here, we provide a general overview of progress over the past decade, and sketch a roadmap of important future developments. We focus on photonic implementations of the reservoir computing machine learning paradigm, which offers a conceptually simple approach that is amenable to hardware implementations. In particular, we provide an overview of photonic reservoir computing implemented via either spatio temporal or delay dynamical systems. Going beyond reservoir computing, we discuss recent advances and future challenges of photonic implementations of deep neural networks, of the quest for learning methods that are hardware-friendly as well as realizing autonomous photonic neural networks, i.e. with minimal digital electronic auxiliary hardware.
Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

Maintaining the long-term exploration capability of the agent remains one of the critical challenges in deep reinforcement learning. A representative solution is to leverage reward shaping to provide intrinsic rewards for the agent to encourage exploration. However, most existing methods suffer from vanishing intrinsic rewards, which cannot provide sustainable exploration incentives. Moreover, they rely heavily on complex models and additional memory to record learning procedures, resulting in high computational complexity and low robustness. To tackle this problem, entropy-based methods are proposed to evaluate the global exploration performance, encouraging the agent to visit the state space more equitably. However, the sample complexity of estimating the state visitation entropy is prohibitive when handling environments with high-dimensional observations. In this paper, we introduce a novel metric entitled Jain's fairness index (JFI) to replace the entropy regularizer, which solves the exploration problem from a brand new perspective. In sharp contrast to the entropy regularizer, JFI is more computable and robust and can be easily applied generalized into arbitrary tasks. Furthermore, we leverage a variational auto-encoder (VAE) model to capture the life-long novelty of states, which is combined with the global JFI score to form multimodal intrinsic rewards. Finally, extensive simulation results demonstrate that our multimodal reward shaping (MMRS) method can achieve higher performance than other benchmark schemes.
Analysing Cyberbullying using Natural Language Processing by Understanding Jargon in Social Media
Venue: arXiv cs.CL 2021
Year: 2021

Cyberbullying is of extreme prevalence today. Online-hate comments, toxicity, cyberbullying amongst children and other vulnerable groups are only growing over online classes, and increased access to social platforms, especially post COVID-19. It is paramount to detect and ensure minors' safety across social platforms so that any violence or hate-crime is automatically detected and strict action is taken against it. In our work, we explore binary classification by using a combination of datasets from various social media platforms that cover a wide range of cyberbullying such as sexism, racism, abusive, and hate-speech. We experiment through multiple models such as Bi-LSTM, GloVe, state-of-the-art models like BERT, and apply a unique preprocessing technique by introducing a slang-abusive corpus, achieving a higher precision in comparison to models without slang preprocessing.
GenRadar: Self-supervised Probabilistic Camera Synthesis based on Radar Frequencies
Venue: arXiv cs.CV 2021
Year: 2021

Autonomous systems require a continuous and dependable environment perception for navigation and decision-making, which is best achieved by combining different sensor types. Radar continues to function robustly in compromised circumstances in which cameras become impaired, guaranteeing a steady inflow of information. Yet, camera images provide a more intuitive and readily applicable impression of the world. This work combines the complementary strengths of both sensor types in a unique self-learning fusion approach for a probabilistic scene reconstruction in adverse surrounding conditions. After reducing the memory requirements of both high-dimensional measurements through a decoupled stochastic self-supervised compression technique, the proposed algorithm exploits similarities and establishes correspondences between both domains at different feature levels during training. Then, at inference time, relying exclusively on radio frequencies, the model successively predicts camera constituents in an autoregressive and self-contained process. These discrete tokens are finally transformed back into an instructive view of the respective surrounding, allowing to visually perceive potential dangers for important tasks downstream.
Clinical Relation Extraction Using Transformer-based Models
Venue: arXiv cs.CL 2021
Year: 2021

The newly emerged transformer technology has a tremendous impact on NLP research. In the general English domain, transformer-based models have achieved state-of-the-art performances on various NLP benchmarks. In the clinical domain, researchers also have investigated transformer models for clinical applications. The goal of this study is to systematically explore three widely used transformer-based models (i.e., BERT, RoBERTa, and XLNet) for clinical relation extraction and develop an open-source package with clinical pre-trained transformer-based models to facilitate information extraction in the clinical domain. We developed a series of clinical RE models based on three transformer architectures, namely BERT, RoBERTa, and XLNet. We evaluated these models using 2 publicly available datasets from 2018 MADE1.0 and 2018 n2c2 challenges. We compared two classification strategies (binary vs. multi-class classification) and investigated two approaches to generate candidate relations in different experimental settings. In this study, we compared three transformer-based (BERT, RoBERTa, and XLNet) models for relation extraction. We demonstrated that the RoBERTa-clinical RE model achieved the best performance on the 2018 MADE1.0 dataset with an F1-score of 0.8958. On the 2018 n2c2 dataset, the XLNet-clinical model achieved the best F1-score of 0.9610. Our results indicated that the binary classification strategy consistently outperformed the multi-class classification strategy for clinical relation extraction. Our methods and models are publicly available at https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction. We believe this work will improve current practice on clinical relation extraction and other related NLP tasks in the biomedical domain.
T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems
Venue: arXiv cs.CY 2021
Year: 2021

Simulation has emerged as a popular method to study the long-term societal consequences of recommender systems. This approach allows researchers to specify their theoretical model explicitly and observe the evolution of system-level outcomes over time. However, performing simulation-based studies often requires researchers to build their own simulation environments from the ground up, which creates a high barrier to entry, introduces room for implementation error, and makes it difficult to disentangle whether observed outcomes are due to the model or the implementation. We introduce T-RECS, an open-sourced Python package designed for researchers to simulate recommendation systems and other types of sociotechnical systems in which an algorithm mediates the interactions between multiple stakeholders, such as users and content creators. To demonstrate the flexibility of T-RECS, we perform a replication of two prior simulation-based research on sociotechnical systems. We additionally show how T-RECS can be used to generate novel insights with minimal overhead. Our tool promotes reproducibility in this area of research, provides a unified language for simulating sociotechnical systems, and removes the friction of implementing simulations from scratch.