OpenRaiser/Intern-Atlas huggingface dataset:
Learning Partial Differential Equations from Data Using Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

We develop a framework for estimating unknown partial differential equations from noisy data, using a deep learning approach. Given noisy samples of a solution to an unknown PDE, our method interpolates the samples using a neural network, and extracts the PDE by equating derivatives of the neural network approximation. Our method applies to PDEs which are linear combinations of user-defined dictionary functions, and generalizes previous methods that only consider parabolic PDEs. We introduce a regularization scheme that prevents the function approximation from overfitting the data and forces it to be a solution of the underlying PDE. We validate the model on simulated data generated by the known PDEs and added Gaussian noise, and we study our method under different levels of noise. We also compare the error of our method with a Cramer-Rao lower bound for an ordinary differential equation. Our results indicate that our method outperforms other methods in estimating PDEs, especially in the low signal-to-noise regime.
Stabilising priors for robust Bayesian deep learning
Venue: arXiv cs.LG 2019
Year: 2019

Bayesian neural networks (BNNs) have developed into useful tools for probabilistic modelling due to recent advances in variational inference enabling large scale BNNs. However, BNNs remain brittle and hard to train, especially: (1) when using deep architectures consisting of many hidden layers and (2) in situations with large weight variances. We use signal propagation theory to quantify these challenges and propose self-stabilising priors. This is achieved by a reformulation of the ELBO to allow the prior to influence network signal propagation. Then, we develop a stabilising prior, where the distributional parameters of the prior are adjusted before each forward pass to ensure stability of the propagating signal. This stabilised signal propagation leads to improved convergence and robustness making it possible to train deeper networks and in more noisy settings.
Controlling the Output Length of Neural Machine Translation
Venue: arXiv cs.CL 2019
Year: 2019

The recent advances introduced by neural machine translation (NMT) are rapidly expanding the application fields of machine translation, as well as reshaping the quality level to be targeted. In particular, if translations have to fit some given layout, quality should not only be measured in terms of adequacy and fluency, but also length. Exemplary cases are the translation of document files, subtitles, and scripts for dubbing, where the output length should ideally be as close as possible to the length of the input text. This paper addresses for the first time, to the best of our knowledge, the problem of controlling the output length in NMT. We investigate two methods for biasing the output length with a transformer architecture: i) conditioning the output to a given target-source length-ratio class and ii) enriching the transformer positional embedding with length information. Our experiments show that both methods can induce the network to generate shorter translations, as well as acquiring interpretable linguistic skills.
Identification of primary angle-closure on AS-OCT images with Convolutional Neural Networks
Venue: arXiv cs.CV 2019
Year: 2019

Primary angle-closure disease (PACD) is a severe retinal disease, which might cause irreversible vision loss. In clinic, accurate identification of angle-closure and localization of the scleral spur's position on anterior segment optical coherence tomography (AS-OCT) is essential for the diagnosis of PACD. However, manual delineation might confine in low accuracy and low efficiency. In this paper, we propose an efficient and accurate end-to-end architecture for angle-closure classification and scleral spur localization. Specifically, we utilize a revised ResNet152 as our backbone to improve the accuracy of the angle-closure identification. For scleral spur localization, we adopt EfficientNet as encoder because of its powerful feature extraction potential. By combining the skip-connect module and pyramid pooling module, the network is able to collect semantic cues in feature maps from multiple dimensions and scales. Afterward, we propose a novel keypoint registration loss to constrain the model's attention to the intensity and location of the scleral spur area. Several experiments are extensively conducted to evaluate our method on the angle-closure glaucoma evaluation (AGE) Challenge dataset. The results show that our proposed architecture ranks the first place of the classification task on the test dataset and achieves the average Euclidean distance error of 12.00 pixels in the scleral spur localization task.
An Analytical Lidar Sensor Model Based on Ray Path Information
Venue: arXiv cs.RO 2019
Year: 2019

Two core competencies of a mobile robot are to build a map of the environment and to estimate its own pose on the basis of this map and incoming sensor readings. To account for the uncertainties in this process, one typically employs probabilistic state estimation approaches combined with a model of the specific sensor. Over the past years, lidar sensors have become a popular choice for mapping and localization. However, many common lidar models perform poorly in unstructured, unpredictable environments, they lack a consistent physical model for both mapping and localization, and they do not exploit all the information the sensor provides, e.g. out-of-range measurements. In this paper, we introduce a consistent physical model that can be applied to mapping as well as to localization. It naturally deals with unstructured environments and makes use of both out-of-range measurements and information about the ray path. The approach can be seen as a generalization of the well-established reflection model, but in addition to counting ray reflections and traversals in a specific map cell, it considers the distances that all rays travel inside this cell. We prove that the resulting map maximizes the data likelihood and demonstrate that our model outperforms state-of-the-art sensor models in extensive real-world experiments.
Automated Text Summarization for the Enhancement of Public Services
Venue: arXiv cs.CL 2019
Year: 2019

Natural language processing and machine learning algorithms have been shown to be effective in a variety of applications. In this work, we contribute to the area of AI adoption in the public sector. We present an automated system that was used to process textual information, generate important keywords, and automatically summarize key elements of the Meadville community statements. We also describe the process of collaboration with My Meadville administrators during the development of our system. My Meadville, a community initiative, supported by the city of Meadville conducted a large number of interviews with the residents of Meadville during the community events and transcribed these interviews into textual data files. Their goal was to uncover the issues of importance to the Meadville residents in an attempt to enhance public services. Our AI system cleans and pre-processes the interview data, then using machine learning algorithms it finds important keywords and key excerpts from each interview. It also provides searching functionality to find excerpts from relevant interviews based on specific keywords. Our automated system allowed the city to save over 300 hours of human labor that would have taken to read all interviews and highlight important points. Our findings are being used by My Meadville initiative to locate important information from the collected data set for ongoing community enhancement projects, to highlight relevant community assets, and to assist in identifying the steps to be taken based on the concerns and areas of improvement identified by the community members.
Towards Automated Management and Analysis of Heterogeneous Data Within Cannabinoids Domain
Venue: arXiv cs.DL 2019
Year: 2019

Cannabinoid research requires the cooperation of experts from various field biochemistry and chemistry to psychological and social sciences. The data that have to be managed and analysed are highly heterogeneous, especially because they are provided by a very diverse range of sources. A number of approaches focused on data collection and the corresponding analysis, restricting the scope to a sub-domain. Our goal is to elaborate a solution that would allow for automated management and analysis of heterogeneous data within the complete cannabinoids domain. The corresponding integration of diverse data sources would increase the quality and preciseness of the analysis. In this paper, we introduce the core ideas of the proposed framework as well as present the implemented prototype of a cannabinoids data platform.
Blockchain Methods for Trusted Avionics Systems
Venue: arXiv cs.DC 2019
Year: 2019

Blockchain is a popular method to ensure security for trusted systems. The benefits include an auditable method to provide decentralized security without a trusted third party, but the drawback is the large computational resources needed to process and store the ever-expanding chain of security blocks. The promise of blockchain for edge devices (e.g., internet of things) poses a variety of challenges and strategies before adoption. In this paper, we explore blockchain methods and examples, with experimental data to determine the merits of the capabilities. As for an aerospace example, we address a notional example for Automatic dependent surveillance-broadcast (ADS-B) from Flight24 data (https://www.flightradar24.com/) to determine whether blockchain is feasible for avionics systems. The methods are incorporated into the Lightweight Internet of Things (IoT) based Smart Public Safety (LISPS) framework. By decoupling a complex system into independent sub-tasks, the LISPS system possesses high flexibility in the design process and online maintenance. The Blockchain-enabled decentralized avionics services provide a secured data sharing and access control mechanism. The experimental results demonstrate the feasibility of the approach.
An Optimized, Parallel Computation of the Ghost Layer for Adaptive Hybrid Forest Meshes
Venue: arXiv cs.DC 2019
Year: 2019

We discuss parallel algorithms to gather topological information about off-process mesh neighbor elements. This information is commonly called the ghost layer, whose creation is a fundamental, necessary task in executing most parallel, element-based computer simulations. Approaches differ in that the ghost layer may either be inherently part of the mesh data structure that is maintained and modified, or kept separate and constructed/deleted as needed. In this work, we present an updated design following the latter approach, which we favor for its modularity of algorithms and data structures. We target arbitrary adaptive, non-conforming forest-of-(oc)trees meshes of mixed element shapes, such as cubes, prisms, and tetrahedra, and restrict ourselves to face-ghosts. Our algorithm has low complexity and redundancy since we reduce it to generic codimension-1 subalgorithms that can be flexibly combined. We cover several existing solutions as special cases and optimize further using recursive, amortized tree searches and traversals.
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
Venue: arXiv cs.IR 2019
Year: 2019

Term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. This paper proposes a Deep Contextualized Term Weighting framework that learns to map BERT's contextualized text representations to context-aware term weights for sentences and passages. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for passage retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms. This is novel because most deep neural network based ranking models have higher computational costs, and thus are restricted to later-stage rankers. Experiments on four datasets demonstrate that DeepCT's deep contextualized text understanding greatly improves the accuracy of first-stage retrieval algorithms.
Representation Learning in Heterogeneous Professional Social Networks with Ambiguous Social Connections
Venue: arXiv cs.SI 2019
Year: 2019

Network representations have been shown to improve performance within a variety of tasks, including classification, clustering, and link prediction. However, most models either focus on moderate-sized, homogeneous networks or require a significant amount of auxiliary input to be provided by the user. Moreover, few works have studied network representations in real-world heterogeneous social networks with ambiguous social connections and are often incomplete. In the present work, we investigate the problem of learning low-dimensional node representations in heterogeneous professional social networks (HPSNs), which are incomplete and have ambiguous social connections. We present a general heterogeneous network representation learning model called Star2Vec that learns entity and person embeddings jointly using a social connection strength-aware biased random walk combined with a node-structure expansion function. Experiments on LinkedIn's Economic Graph and publicly available snapshots of Facebook's network show that Star2Vec outperforms existing methods on members' industry and social circle classification, skill and title clustering, and member-entity link predictions. We also conducted large-scale case studies to demonstrate practical applications of the Star2Vec embeddings trained on LinkedIn's Economic Graph such as next career move, alternative career suggestions, and general entity similarity searches.
Hierarchical Transformers for Long Document Classification
Venue: arXiv cs.CL 2019
Year: 2019

BERT, which stands for Bidirectional Encoder Representations from Transformers, is a recently introduced language representation model based upon the transfer learning paradigm. We extend its fine-tuning procedure to address one of its major limitations - applicability to inputs longer than a few hundred words, such as transcripts of human call conversations. Our method is conceptually simple. We segment the input into smaller chunks and feed each of them into the base model. Then, we propagate each output through a single recurrent layer, or another transformer, followed by a softmax activation. We obtain the final classification decision after the last segment has been consumed. We show that both BERT extensions are quick to fine-tune and converge after as little as 1 epoch of training on a small, domain-specific data set. We successfully apply them in three different tasks involving customer call satisfaction prediction and topic classification, and obtain a significant improvement over the baseline models in two of them.
Wasserstein Smoothing: Certified Robustness against Wasserstein Adversarial Attacks
Venue: arXiv cs.LG 2019
Year: 2019

In the last couple of years, several adversarial attack methods based on different threat models have been proposed for the image classification problem. Most existing defenses consider additive threat models in which sample perturbations have bounded L_p norms. These defenses, however, can be vulnerable against adversarial attacks under non-additive threat models. An example of an attack method based on a non-additive threat model is the Wasserstein adversarial attack proposed by Wong et al. (2019), where the distance between an image and its adversarial example is determined by the Wasserstein metric ("earth-mover distance") between their normalized pixel intensities. Until now, there has been no certifiable defense against this type of attack. In this work, we propose the first defense with certified robustness against Wasserstein Adversarial attacks using randomized smoothing. We develop this certificate by considering the space of possible flows between images, and representing this space such that Wasserstein distance between images is upper-bounded by L_1 distance in this flow-space. We can then apply existing randomized smoothing certificates for the L_1 metric. In MNIST and CIFAR-10 datasets, we find that our proposed defense is also practically effective, demonstrating significantly improved accuracy under Wasserstein adversarial attack compared to unprotected models.
Emergent Properties of Finetuned Language Representation Models
Venue: arXiv cs.CL 2019
Year: 2019

Large, self-supervised transformer-based language representation models have recently received significant amounts of attention, and have produced state-of-the-art results across a variety of tasks simply by scaling up pre-training on larger and larger corpora. Such models usually produce high dimensional vectors, on top of which additional task-specific layers and architectural modifications are added to adapt them to specific downstream tasks. Though there exists ample evidence that such models work well, we aim to understand what happens when they work well. We analyze the redundancy and location of information contained in output vectors for one such language representation model -- BERT. We show empirical evidence that the [CLS] embedding in BERT contains highly redundant information, and can be compressed with minimal loss of accuracy, especially for finetuned models, dovetailing into open threads in the field about the role of over-parameterization in learning. We also shed light on the existence of specific output dimensions which alone give very competitive results when compared to using all dimensions of output vectors.
Unknown Identity Rejection Loss: Utilizing Unlabeled Data for Face Recognition
Venue: arXiv cs.CV 2019
Year: 2019

Face recognition has advanced considerably with the availability of large-scale labeled datasets. However, how to further improve the performance with the easily accessible unlabeled dataset remains a challenge. In this paper, we propose the novel Unknown Identity Rejection (UIR) loss to utilize the unlabeled data. We categorize identities in unconstrained environment into the known set and the unknown set. The former corresponds to the identities that appear in the labeled training dataset while the latter is its complementary set. Besides training the model to accurately classify the known identities, we also force the model to reject unknown identities provided by the unlabeled dataset via our proposed UIR loss. In order to 'reject' faces of unknown identities, centers of the known identities are forced to keep enough margin from centers of unknown identities which are assumed to be approximated by the features of their samples. By this means, the discriminativeness of the face representations can be enhanced. Experimental results demonstrate that our approach can provide obvious performance improvement by utilizing the unlabeled data.
Multi-channel Speech Separation Using Deep Embedding Model with Multilayer Bootstrap Networks
Venue: arXiv cs.SD 2019
Year: 2019

Recently, deep clustering (DPCL) based speaker-independent speech separation has drawn much attention, since it needs little speaker prior information. However, it still has much room of improvement, particularly in reverberant environments. If the training and test environments mismatch which is a common case, the embedding vectors produced by DPCL may contain much noise and many small variations. To deal with the problem, we propose a variant of DPCL, named DPCL++, by applying a recent unsupervised deep learning method---multilayer bootstrap networks(MBN)---to further reduce the noise and small variations of the embedding vectors in an unsupervised way in the test stage, which fascinates k-means to produce a good result. MBN builds a gradually narrowed network from bottom-up via a stack of k-centroids clustering ensembles, where the k-centroids clusterings are trained independently by random sampling and one-nearest-neighbor optimization. To further improve the robustness of DPCL++ in reverberant environments, we take spatial features as part of its input. Experimental results demonstrate the effectiveness of the proposed method.
Joint User Selection and Precoding in Multiuser MIMO Systems via Group LASSO
Venue: arXiv cs.IT 2019
Year: 2019

Joint user selection and precoding in multiuser MIMO settings can be interpreted as group sparse recovery in linear models. In this problem, a signal with group sparsity is to be reconstructed from an underdetermined system of equations. This paper utilizes this equivalent interpretation and develops a computationally tractable algorithm based on the method of group LASSO. Compared to the state of the art, the proposed scheme shows performance enhancements in two different respects: higher achievable sum-rate and lower interference at the non-selected user terminals.
Reversible Causal Nets and Reversible Event Structures
Venue: arXiv cs.LO 2019
Year: 2019

One of the well-known results in concurrency theory concerns the relationship between event structures and occurrence nets: an occurrence net can be associated with a prime event structure, and vice versa. More generally, the relationships between various forms of event structures and suitable forms of nets have been long established. Good examples are the close relationship between inhibitor event structures and inhibitor occurrence nets, or between asymmetric event structures and asymmetric occurrence nets. Several forms of event structures suited for the modelling of reversible computation have recently been developed; also a method for reversing occurrence nets has been proposed. This paper bridges the gap between reversible event structures and reversible nets. We introduce the notion of reversible causal net, which is a generalisation of the notion of reversible unfolding. We show that reversible causal nets correspond precisely to a subclass of reversible prime event structures, the causal reversible prime event structures.
Supervised Encoding for Discrete Representation Learning
Venue: arXiv cs.CV 2019
Year: 2019

Classical supervised classification tasks search for a nonlinear mapping that maps each encoded feature directly to a probability mass over the labels. Such a learning framework typically lacks the intuition that encoded features from the same class tend to be similar and thus has little interpretability for the learned features. In this paper, we propose a novel supervised learning model named Supervised-Encoding Quantizer (SEQ). The SEQ applies a quantizer to cluster and classify the encoded features. We found that the quantizer provides an interpretable graph where each cluster in the graph represents a class of data samples that have a particular style. We also trained a decoder that can decode convex combinations of the encoded features from similar and different clusters and provide guidance on style transfer between sub-classes.
Adaptive-time Synchronization Algorithm for Superlattice Key Distribution
Venue: arXiv cs.DC 2019
Year: 2019

This paper presents a synchronization algorithm for superlattice key distribution, which is a symmetric encryption solution, by optimizing the Euclidian distance between the two chaotic waveforms generated in the receiver and the sender, respectively. This algorithm based on time synchronization is capable of reconstructing the generated waveforms in the receiver and the sender within the error of 5% to 6% (given the fact that the waveforms were not perfectly congruent when they were originally created).
Development of a hand pose recognition system on an embedded computer using CNNs
Venue: arXiv cs.CV 2019
Year: 2019

Demand of hand pose recognition systems are growing in the last years in technologies like human-machine interfaces. This work suggests an approach for hand pose recognition in embedded computers using hand tracking and CNNs. Results show a fast time response with an accuracy of 94.50% and low power consumption.