OpenRaiser/Intern-Atlas huggingface dataset:
The VLSAT-3 Benchmark Suite
Venue: arXiv cs.DS 2021
Year: 2021

This report presents VLSAT-3 (an acronym for "Very Large Boolean SATisfiability problems"),the third part of a benchmark suite to be used in scientific experimentsand software competitions addressing SAT and SMT (Satisfiability Modulo Theories) solving issues.VLSAT-3 contains 1200 (600~satisfiable and 600~unsatisfiable) quantifier-free first-order logic formulasof increasing complexity, proposed in SMT-LIB format under a permissive Creative Commons license.More than 90% of these benchmarks have been used during the 16th International Satisfiability Modulo TheoriesCompetition (SMT-COMP~2021).
GKS: Graph-based Knowledge Selector for Task-oriented Dialog System
Venue: arXiv cs.CL 2021
Year: 2021

In previous research, knowledge-selection tasks mostly rely on language model-based methods or knowledge ranking. However, while approaches that rely on the language models take all knowledge as sequential input, knowledge does not contain sequential information in most circumstances. On the other hand, the knowledge-ranking methods leverage dialog history and each given knowledge snippet separately, but they do not consider information between knowledge snippets. In the Tenth Dialog System Technology Challenges (DSTC10), we participated in the second Knowledge-grounded Task-oriented Dialogue Modeling on Spoken Conversations. To deal with the problems mentioned above, we modified training methods based on state-of-the-art (SOTA) models for the first and third sub-tasks. As for the second sub-task of knowledge selection, we proposed Graph-Knowledge Selector (GKS), utilizing a graph-attention base model incorporated with the language model. GKS makes knowledge-selection decisions in the dialog by simultaneously considering each knowledge embedding generated from the language model without sequential features. Moreover, GKS leverages considerable knowledge in decision-making and takes relations across knowledge as part of the selection process. As a result, GKS outperforms several SOTA models proposed in the data-set on knowledge selection from the Ninth Dialog System Technology Challenges (DSTC9).
A Continuous-time Stochastic Gradient Descent Method for Continuous Data
Venue: arXiv cs.LG 2021
Year: 2021

Optimization problems with continuous data appear in, e.g., robust machine learning, functional data analysis, and variational inference. Here, the target function is given as an integral over a family of (continuously) indexed target functions - integrated with respect to a probability measure. Such problems can often be solved by stochastic optimization methods: performing optimization steps with respect to the indexed target function with randomly switched indices. In this work, we study a continuous-time variant of the stochastic gradient descent algorithm for optimization problems with continuous data. This so-called stochastic gradient process consists in a gradient flow minimizing an indexed target function that is coupled with a continuous-time index process determining the index. Index processes are, e.g., reflected diffusions, pure jump processes, or other Lévy processes on compact spaces. Thus, we study multiple sampling patterns for the continuous data space and allow for data simulated or streamed at runtime of the algorithm. We analyze the approximation properties of the stochastic gradient process and study its longtime behavior and ergodicity under constant and decreasing learning rates. We end with illustrating the applicability of the stochastic gradient process in a polynomial regression problem with noisy functional data, as well as in a physics-informed neural network.
In-flight Novelty Detection with Convolutional Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Gas turbine engines are complex machines that typically generate a vast amount of data, and require careful monitoring to allow for cost-effective preventative maintenance. In aerospace applications, returning all measured data to ground is prohibitively expensive, often causing useful, high value, data to be discarded. The ability to detect, prioritise, and return useful data in real-time is therefore vital. This paper proposes that system output measurements, described by a convolutional neural network model of normality, are prioritised in real-time for the attention of preventative maintenance decision makers. Due to the complexity of gas turbine engine time-varying behaviours, deriving accurate physical models is difficult, and often leads to models with low prediction accuracy and incompatibility with real-time execution. Data-driven modelling is a desirable alternative producing high accuracy, asset specific models without the need for derivation from first principles. We present a data-driven system for online detection and prioritisation of anomalous data. Biased data assessment deriving from novel operating conditions is avoided by uncertainty management integrated into the deep neural predictive model. Testing is performed on real and synthetic data, showing sensitivity to both real and synthetic faults. The system is capable of running in real-time on low-power embedded hardware and is currently in deployment on the Rolls-Royce Pearl 15 engine flight trials.
New Lower Bounds on the Capacity of Optical Fiber Channels via Optimized Shaping and Detection
Venue: arXiv cs.IT 2021
Year: 2021

Constellation shaping is a practical and effective technique to improve the performance and the rate adaptivity of optical communication systems. In principle, it could also be used to mitigate the impact of nonlinear effects, possibly increasing the information rate beyond the current limit dictated by fiber nonlinearity. However, this appealing idea is frustrated by the difficulty of designing an effective shaping strategy that takes into account the nonlinearity and long memory of the fiber channel, as well as the possible interplay with other nonlinearity mitigation strategies. As a result, only little progress has been made so far, while the optimal shaping distribution and the ultimate channel capacity remain unknown. In this work, we describe a novel technique to optimize the shaping distribution in a very general setting and high-dimensional space. For a simplified block-memoryless nonlinear optical channel, the capacity lower bound obtained by the proposed technique can be expressed analytically, establishing the conditions for an unbounded growth of capacity with power. In a more realistic scenario, the technique can be implemented by a rejection sampling algorithm driven by a suitable cost function, and the corresponding achievable information rate estimated numerically. The combination of the proposed technique with an improved (non-Gaussian) decoding metric yields a new capacity lower bound for the dual-polarization WDM channel.
Suppressing Static Visual Cues via Normalizing Flows for Self-Supervised Video Representation Learning
Venue: arXiv cs.CV 2021
Year: 2021

Despite the great progress in video understanding made by deep convolutional neural networks, feature representation learned by existing methods may be biased to static visual cues. To address this issue, we propose a novel method to suppress static visual cues (SSVC) based on probabilistic analysis for self-supervised video representation learning. In our method, video frames are first encoded to obtain latent variables under standard normal distribution via normalizing flows. By modelling static factors in a video as a random variable, the conditional distribution of each latent variable becomes shifted and scaled normal. Then, the less-varying latent variables along time are selected as static cues and suppressed to generate motion-preserved videos. Finally, positive pairs are constructed by motion-preserved videos for contrastive learning to alleviate the problem of representation bias to static cues. The less-biased video representation can be better generalized to various downstream tasks. Extensive experiments on publicly available benchmarks demonstrate that the proposed method outperforms the state of the art when only single RGB modality is used for pre-training.
raceBERT -- A Transformer-based Model for Predicting Race and Ethnicity from Names
Venue: arXiv cs.CL 2021
Year: 2021

This paper presents raceBERT -- a transformer-based model for predicting race and ethnicity from character sequences in names, and an accompanying python package. Using a transformer-based model trained on a U.S. Florida voter registration dataset, the model predicts the likelihood of a name belonging to 5 U.S. census race categories (White, Black, Hispanic, Asian & Pacific Islander, American Indian & Alaskan Native). I build on Sood and Laohaprapanon (2018) by replacing their LSTM model with transformer-based models (pre-trained BERT model, and a roBERTa model trained from scratch), and compare the results. To the best of my knowledge, raceBERT achieves state-of-the-art results in race prediction using names, with an average f1-score of 0.86 -- a 4.1% improvement over the previous state-of-the-art, and improvements between 15-17% for non-white names.
Attention-Based Model and Deep Reinforcement Learning for Distribution of Event Processing Tasks
Venue: arXiv cs.LG 2021
Year: 2021

Event processing is the cornerstone of the dynamic and responsive Internet of Things (IoT). Recent approaches in this area are based on representational state transfer (REST) principles, which allow event processing tasks to be placed at any device that follows the same principles. However, the tasks should be properly distributed among edge devices to ensure fair resources utilization and guarantee seamless execution. This article investigates the use of deep learning to fairly distribute the tasks. An attention-based neural network model is proposed to generate efficient load balancing solutions under different scenarios. The proposed model is based on the Transformer and Pointer Network architectures, and is trained by an advantage actor-critic reinforcement learning algorithm. The model is designed to scale to the number of event processing tasks and the number of edge devices, with no need for hyperparameters re-tuning or even retraining. Extensive experimental results show that the proposed model outperforms conventional heuristics in many key performance indicators. The generic design and the obtained results show that the proposed model can potentially be applied to several other load balancing problem variations, which makes the proposal an attractive option to be used in real-world scenarios due to its scalability and efficiency.
Stupid, Evil, or Both? Understanding the Smittestopp conflict
Venue: arXiv cs.CY 2021
Year: 2021

Like many governments, the Norwegian government provided a contact tracing application to help in combating the COVID-19 pandemic at its outset. However, the application was widely criticized for enabling an unacceptable intrusion into its subjects' lives, leading to its discontinuation only four months into the pandemic. In this essay, we will take a closer look at what went wrong, attempt to gain a deeper understanding of the passionate nature of the conflict, and how both sides came to view the other as being either stupid, or evil, or both.
Designing a Real-Time IoT Data Streaming Testbed for Horizontally Scalable Analytical Platforms: Czech Post Case Study
Venue: arXiv cs.DC 2021
Year: 2021

There is a growing trend for enterprise-level Internet of Things (IoT) applications requiring real-time horizontally scalable data processing platforms. Real-time processing platforms receiving data streams from sensor networks (e.g., autonomous and connected vehicles, smart security for businesses and homes, smartwatches, fitness trackers, and other wearables) require distributed MQTT brokers. This case study presents an IoT data streaming testbed platform prepared for the Czech Post. The presented platform has met the throughput requirement of 2 million messages per 24 hours (comprising SMS and emails). The tested MQTT broker runs on a single virtual node of a horizontally scalable testbed platform. Soon the Czech Post will modernise its eServices to increase package deliveries aligned with eCommerce and eGovernment demands. The presented testbed platform fulfils all requirements, and it is also capable of processing thousands of messages per second. The presented platform and concepts are transferable to healthcare systems, transport operations, the automotive industry, and other domains such as smart cities.
A Robust Completed Local Binary Pattern (RCLBP) for Surface Defect Detection
Venue: arXiv cs.CV 2021
Year: 2021

In this paper, we present a Robust Completed Local Binary Pattern (RCLBP) framework for a surface defect detection task. Our approach uses a combination of Non-Local (NL) means filter with wavelet thresholding and Completed Local Binary Pattern (CLBP) to extract robust features which are fed into classifiers for surface defects detection. This paper combines three components: A denoising technique based on Non-Local (NL) means filter with wavelet thresholding is established to denoise the noisy image while preserving the textures and edges. Second, discriminative features are extracted using the CLBP technique. Finally, the discriminative features are fed into the classifiers to build the detection model and evaluate the performance of the proposed framework. The performance of the defect detection models are evaluated using a real-world steel surface defect database from Northeastern University (NEU). Experimental results demonstrate that the proposed approach RCLBP is noise robust and can be applied for surface defect detection under varying conditions of intra-class and inter-class changes and with illumination changes.
Accelerating Understanding of Scientific Experiments with End to End Symbolic Regression
Venue: arXiv cs.LG 2021
Year: 2021

We consider the problem of learning free-form symbolic expressions from raw data, such as that produced by an experiment in any scientific domain. Accurate and interpretable models of scientific phenomena are the cornerstone of scientific research. Simple yet interpretable models, such as linear or logistic regression and decision trees often lack predictive accuracy. Alternatively, accurate blackbox models such as deep neural networks provide high predictive accuracy, but do not readily admit human understanding in a way that would enrich the scientific theory of the phenomenon. Many great breakthroughs in science revolve around the development of parsimonious equational models with high predictive accuracy, such as Newton's laws, universal gravitation, and Maxwell's equations. Previous work on automating the search of equational models from data combine domain-specific heuristics as well as computationally expensive techniques, such as genetic programming and Monte-Carlo search. We develop a deep neural network (MACSYMA) to address the symbolic regression problem as an end-to-end supervised learning problem. MACSYMA can generate symbolic expressions that describe a dataset. The computational complexity of the task is reduced to the feedforward computation of a neural network. We train our neural network on a synthetic dataset consisting of data tables of varying length and varying levels of noise, for which the neural network must learn to produce the correct symbolic expression token by token. Finally, we validate our technique by running on a public dataset from behavioral science.
Convolutional codes with a maximum distance profile based on skew polynomials
Venue: arXiv cs.IT 2021
Year: 2021

We construct a family of (n,k) convolutional codes with degree δin {k,n-k} that have a maximum distance profile. The field size required for our construction is of the order n^{2δ}, which improves upon the known constructions of convolutional codes with a maximum distance profile. Our construction is based on the theory of skew polynomials.
FreeTalky: Don't Be Afraid! Conversations Made Easier by a Humanoid Robot using Persona-based Dialogue
Venue: arXiv cs.CL 2021
Year: 2021

We propose a deep learning-based foreign language learning platform, named FreeTalky, for people who experience anxiety dealing with foreign languages, by employing a humanoid robot NAO and various deep learning models. A persona-based dialogue system that is embedded in NAO provides an interesting and consistent multi-turn dialogue for users. Also, an grammar error correction system promotes improvement in grammar skills of the users. Thus, our system enables personalized learning based on persona dialogue and facilitates grammar learning of a user using grammar error feedback. Furthermore, we verified whether FreeTalky provides practical help in alleviating xenoglossophobia by replacing the real human in the conversation with a NAO robot, through human evaluation.
Assessing a Single Image in Reference-Guided Image Synthesis
Venue: arXiv cs.CV 2021
Year: 2021

Assessing the performance of Generative Adversarial Networks (GANs) has been an important topic due to its practical significance. Although several evaluation metrics have been proposed, they generally assess the quality of the whole generated image distribution. For Reference-guided Image Synthesis (RIS) tasks, i.e., rendering a source image in the style of another reference image, where assessing the quality of a single generated image is crucial, these metrics are not applicable. In this paper, we propose a general learning-based framework, Reference-guided Image Synthesis Assessment (RISA) to quantitatively evaluate the quality of a single generated image. Notably, the training of RISA does not require human annotations. In specific, the training data for RISA are acquired by the intermediate models from the training procedure in RIS, and weakly annotated by the number of models' iterations, based on the positive correlation between image quality and iterations. As this annotation is too coarse as a supervision signal, we introduce two techniques: 1) a pixel-wise interpolation scheme to refine the coarse labels, and 2) multiple binary classifiers to replace a naïve regressor. In addition, an unsupervised contrastive loss is introduced to effectively capture the style similarity between a generated image and its reference image. Empirical results on various datasets demonstrate that RISA is highly consistent with human preference and transfers well across models.
Progressive Multi-stage Interactive Training in Mobile Network for Fine-grained Recognition
Venue: arXiv cs.CV 2021
Year: 2021

Fine-grained Visual Classification (FGVC) aims to identify objects from subcategories. It is a very challenging task because of the subtle inter-class differences. Existing research applies large-scale convolutional neural networks or visual transformers as the feature extractor, which is extremely computationally expensive. In fact, real-world scenarios of fine-grained recognition often require a more lightweight mobile network that can be utilized offline. However, the fundamental mobile network feature extraction capability is weaker than large-scale models. In this paper, based on the lightweight MobilenetV2, we propose a Progressive Multi-Stage Interactive training method with a Recursive Mosaic Generator (RMG-PMSI). First, we propose a Recursive Mosaic Generator (RMG) that generates images with different granularities in different phases. Then, the features of different stages pass through a Multi-Stage Interaction (MSI) module, which strengthens and complements the corresponding features of different stages. Finally, using the progressive training (P), the features extracted by the model in different stages can be fully utilized and fused with each other. Experiments on three prestigious fine-grained benchmarks show that RMG-PMSI can significantly improve the performance with good robustness and transferability.
Transformer based trajectory prediction
Venue: arXiv cs.RO 2021
Year: 2021

To plan a safe and efficient route, an autonomous vehicle should anticipate future motions of other agents around it. Motion prediction is an extremely challenging task which recently gained significant attention of the research community. In this work, we present a simple and yet strong baseline for uncertainty aware motion prediction based purely on transformer neural networks, which has shown its effectiveness in conditions of domain change. While being easy-to-implement, the proposed approach achieves competitive performance and ranks 1$^{st}$ on the 2021 Shifts Vehicle Motion Prediction Competition.
Autoencoder-based Communications with Reconfigurable Intelligent Surfaces
Venue: arXiv cs.NI 2021
Year: 2021

This paper presents a novel approach for the joint design of a reconfigurable intelligent surface (RIS) and a transmitter-receiver pair that are trained together as a set of deep neural networks (DNNs) to optimize the end-to-end communication performance at the receiver. The RIS is a software-defined array of unit cells that can be controlled in terms of the scattering and reflection profiles to focus the incoming signals from the transmitter to the receiver. The benefit of the RIS is to improve the coverage and spectral efficiency for wireless communications by overcoming physical obstructions of the line-of-sight (LoS) links. The selection process of the RIS beam codeword (out of a pre-defined codebook) is formulated as a DNN, while the operations of the transmitter-receiver pair are modeled as two DNNs, one for the encoder (at the transmitter) and the other one for the decoder (at the receiver) of an autoencoder, by accounting for channel effects including those induced by the RIS in between. The underlying DNNs are jointly trained to minimize the symbol error rate at the receiver. Numerical results show that the proposed design achieves major gains in error performance with respect to various baseline schemes, where no RIS is used or the selection of the RIS beam is separated from the design of the transmitter-receiver pair.
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
Venue: arXiv cs.CV 2021
Year: 2021

This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long clip. We then extract their dense temporal embeddings. The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips. Our study reveals the impact of temporal granularity with three major findings. 1) Different video tasks may require features of different temporal granularities. 2) Intriguingly, some tasks that are widely considered to require temporal awareness can actually be well addressed by temporally persistent features. 3) The flexibility of TeG gives rise to state-of-the-art results on 8 video benchmarks, outperforming supervised pre-training in most cases.
A Simple Algorithm for Graph Reconstruction
Venue: arXiv cs.DS 2021
Year: 2021

How efficiently can we find an unknown graph using distance queries between its vertices? We assume that the unknown graph is connected, unweighted, and has bounded degree. The goal is to find every edge in the graph. This problem admits a reconstruction algorithm based on multi-phase Voronoi-cell decomposition and using $\tilde O(n^{3/2})$ distance queries. In our work, we analyze a simple reconstruction algorithm. We show that, on random $Δ$-regular graphs, our algorithm uses $\tilde O(n)$ distance queries. As by-products, we can reconstruct those graphs using $O(\log^2 n)$ queries to an all-distances oracle or $\tilde O(n)$ queries to a betweenness oracle, and we bound the metric dimension of those graphs by $\log^2 n$. Our reconstruction algorithm has a very simple structure, and is highly parallelizable. On general graphs of bounded degree, our reconstruction algorithm has subquadratic query complexity.