OpenRaiser/Intern-Atlas huggingface dataset:
A Probabilistic Framework for Learning Domain Specific Hierarchical Word Embeddings
Venue: arXiv cs.CL 2019
Year: 2019

The meaning of a word often varies depending on its usage in different domains. The standard word embedding models struggle to represent this variation, as they learn a single global representation for a word. We propose a method to learn domain-specific word embeddings, from text organized into hierarchical domains, such as reviews in an e-commerce website, where products follow a taxonomy. Our structured probabilistic model allows vector representations for the same word to drift away from each other for distant domains in the taxonomy, to accommodate its domain-specific meanings. By learning sets of domain-specific word representations jointly, our model can leverage domain relationships, and it scales well with the number of domains. Using large real-world review datasets, we demonstrate the effectiveness of our model compared to state-of-the-art approaches, in learning domain-specific word embeddings that are both intuitive to humans and benefit downstream NLP tasks.
Design method of Temporary Horizontal Visibility Graph for information sources impact network
Venue: arXiv cs.SI 2019
Year: 2019

The absence of the efficient methods for the design of the information sources impact network does not allow defining the influence of the information sources on one another accurately and detecting the primary sources of the information spreading. The work represents a new approach - Temporary Horizontal Visibility Graph (THVG) method, which is based on the HVG algorithm modification. This method allows building the information sources impact networks and making the assumptions about the primary sources of information. Obtained method uses sources rating in data flow analysis. It is more effective than the usual Horizontal Visibility Graph design method in the following criteria: F-measure - by 7-9%, completeness - by 7-10% and accuracy - by 6-8% more effective.
Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play
Venue: arXiv cs.CL 2019
Year: 2019

End-to-end neural approaches are becoming increasingly common in conversational scenarios due to their promising performances when provided with sufficient amount of data. In this paper, we present a novel methodology to address the interpretability of neural approaches in such scenarios by creating challenge datasets using dialogue self-play over multiple tasks/intents. Dialogue self-play allows generating large amount of synthetic data; by taking advantage of the complete control over the generation process, we show how neural approaches can be evaluated in terms of unseen dialogue patterns. We propose several out-of-pattern test cases each of which introduces a natural and unexpected user utterance phenomenon. As a proof of concept, we built a single and a multiple memory network, and show that these two architectures have diverse performances depending on the peculiar dialogue patterns.
Machine Learning for Error Correction with Natural Redundancy
Venue: arXiv cs.IT 2019
Year: 2019

The persistent storage of big data requires advanced error correction schemes. The classical approach is to use error correcting codes (ECCs). This work studies an alternative approach, which uses the redundancy inherent in data itself for error correction. This type of redundancy, called Natural Redundancy (NR), is abundant in many types of uncompressed or even compressed files. The complex structures of Natural Redundancy, however, require machine learning techniques. In this paper, we study two fundamental approaches to use Natural Redundancy for error correction. The first approach, called Representation-Oblivious, requires no prior knowledge on how data are represented or compressed in files. It uses deep learning to detect file types accurately, and then mine Natural Redundancy for soft decoding. The second approach, called Representation-Aware, assumes that such knowledge is known and uses it for error correction. Furthermore, both approaches combine the decoding based on NR and ECCs. Both experimental results and analysis show that such an integrated scheme can substantially improve the error correction performance.
Bridging the Knowledge Gap: Enhancing Question Answering with World and Domain Knowledge
Venue: arXiv cs.CL 2019
Year: 2019

In this paper we present OSCAR (Ontology-based Semantic Composition Augmented Regularization), a method for injecting task-agnostic knowledge from an Ontology or knowledge graph into a neural network during pretraining. We evaluated the impact of including OSCAR when pretraining BERT with Wikipedia articles by measuring the performance when fine-tuning on two question answering tasks involving world knowledge and causal reasoning and one requiring domain (healthcare) knowledge and obtained 33:3%, 18:6%, and 4% improved accuracy compared to pretraining BERT without OSCAR and obtaining new state-of-the-art results on two of the tasks.
An Exponential Learning Rate Schedule for Deep Learning
Venue: arXiv cs.LG 2019
Year: 2019

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1 +α)$ factor in every epoch for some $α>0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.
Creativity in Robot Manipulation with Deep Reinforcement Learning
Venue: arXiv cs.LG 2019
Year: 2019

Deep Reinforcement Learning (DRL) has emerged as a powerful control technique in robotic science. In contrast to control theory, DRL is more robust in the thorough exploration of the environment. This capability of DRL generates more human-like behaviour and intelligence when applied to the robots. To explore this capability, we designed challenging manipulation tasks to observe robots strategy to handle complex scenarios. We observed that robots not only perform tasks successfully, but also transpire a creative and non intuitive solution. We also observed robot's persistence in tasks that are close to success and its striking ability in discerning to continue or give up.
Using Whole Document Context in Neural Machine Translation
Venue: arXiv cs.CL 2019
Year: 2019

In Machine Translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a simple yet promising approach to add contextual information in Neural Machine Translation. We present a method to add source context that capture the whole document with accurate boundaries, taking every word into account. We provide this additional information to a Transformer model and study the impact of our method on three language pairs. The proposed approach obtains promising results in the English-German, English-French and French-English document-level translation tasks. We observe interesting cross-sentential behaviors where the model learns to use document-level information to improve translation coherence.
On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach
Venue: arXiv cs.LG 2019
Year: 2019

Many tasks in modern machine learning can be formulated as finding equilibria in \emph{sequential} games. In particular, two-player zero-sum sequential games, also known as minimax optimization, have received growing interest. It is tempting to apply gradient descent to solve minimax optimization given its popularity and success in supervised learning. However, it has been noted that naive application of gradient descent fails to find some local minimax and can converge to non-local-minimax points. In this paper, we propose \emph{Follow-the-Ridge} (FR), a novel algorithm that provably converges to and only converges to local minimax. We show theoretically that the algorithm addresses the notorious rotational behaviour of gradient dynamics, and is compatible with preconditioning and \emph{positive} momentum. Empirically, FR solves toy minimax problems and improves the convergence of GAN training compared to the recent minimax optimization algorithms.
A Double Residual Compression Algorithm for Efficient Distributed Learning
Venue: arXiv cs.LG 2019
Year: 2019

Large-scale machine learning models are often trained by parallel stochastic gradient descent algorithms. However, the communication cost of gradient aggregation and model synchronization between the master and worker nodes becomes the major obstacle for efficient learning as the number of workers and the dimension of the model increase. In this paper, we propose DORE, a DOuble REsidual compression stochastic gradient descent algorithm, to reduce over $95\%$ of the overall communication such that the obstacle can be immensely mitigated. Our theoretical analyses demonstrate that the proposed strategy has superior convergence properties for both strongly convex and nonconvex objective functions. The experimental results validate that DORE achieves the best communication efficiency while maintaining similar model accuracy and convergence speed in comparison with start-of-the-art baselines.
Network Scanning and Mapping for IIoT Edge Node Device Security
Venue: arXiv cs.CR 2019
Year: 2019

The amount of connected devices in the industrial environment is growing continuously, due to the ongoing demands of new features like predictive maintenance. New business models require more data, collected by IIoT edge node sensors based on inexpensive and low performance Microcontroller Units (MCUs). A negative side effect of this rise of interconnections is the increased attack surface, enabled by a larger network with more network services. Attaching badly documented and cheap devices to industrial networks often without permission of the administrator even further increases the security risk. A decent method to monitor the network and detect "unwanted" devices is network scanning. Typically, this scanning procedure is executed by a computer or server in each sub-network. In this paper, we introduce network scanning and mapping as a building block to scan directly from the Industrial Internet of Things (IIoT) edge node devices. This module scans the network in a pseudo-random periodic manner to discover devices and detect changes in the network structure. Furthermore, we validate our approach in an industrial testbed to show the feasibility of this approach.
Towards a Precipitation Bias Corrector against Noise and Maldistribution
Venue: arXiv cs.LG 2019
Year: 2019

With broad applications in various public services like aviation management and urban disaster warning, numerical precipitation prediction plays a crucial role in weather forecast. However, constrained by the limitation of observation and conventional meteorological models, the numerical precipitation predictions are often highly biased. To correct this bias, classical correction methods heavily depend on profound experts who have knowledge in aerodynamics, thermodynamics and meteorology. As precipitation can be influenced by countless factors, however, the performances of these expert-driven methods can drop drastically when some un-modeled factors change. To address this issue, this paper presents a data-driven deep learning model which mainly includes two blocks, i.e. a Denoising Autoencoder Block and an Ordinal Regression Block. To the best of our knowledge, it is the first expert-free models for bias correction. The proposed model can effectively correct the numerical precipitation prediction based on 37 basic meteorological data from European Centre for Medium-Range Weather Forecasts (ECMWF). Experiments indicate that compared with several classical machine learning algorithms and deep learning models, our method achieves the best correcting performance and meteorological index, namely the threat scores (TS), obtaining satisfactory visualization effect.
Collaborative Filtering with Label Consistent Restricted Boltzmann Machine
Venue: arXiv cs.LG 2019
Year: 2019

The possibility of employing restricted Boltzmann machine (RBM) for collaborative filtering has been known for about a decade. However, there has been hardly any work on this topic since 2007. This work revisits the application of RBM in recommender systems. RBM based collaborative filtering only used the rating information; this is an unsupervised architecture. This work adds supervision by exploiting user demographic information and item metadata. A network is learned from the representation layer to the labels (metadata). The proposed label consistent RBM formulation improves significantly on the existing RBM based approach and yield results at par with the state-of-the-art latent factor based models.
Information-theoretic metrics for Local Differential Privacy protocols
Venue: arXiv cs.CR 2019
Year: 2019

Local Differential Privacy (LDP) protocols allow an aggregator to obtain population statistics about sensitive data of a userbase, while protecting the privacy of the individual users. To understand the tradeoff between aggregator utility and user privacy, we introduce new information-theoretic metrics for utility and privacy. Contrary to other LDP metrics, these metrics highlight the fact that the users and the aggregator are interested in fundamentally different domains of information. We show how our metrics relate to $\varepsilon$-LDP, the \emph{de facto} standard privacy metric, giving an information-theoretic interpretation to the latter. Furthermore, we use our metrics to quantitatively study the privacy-utility tradeoff for a number of popular protocols.
Effect of Superpixel Aggregation on Explanations in LIME -- A Case Study with Biological Data
Venue: arXiv cs.LG 2019
Year: 2019

End-to-end learning with deep neural networks, such as convolutional neural networks (CNNs), has been demonstrated to be very successful for different tasks of image classification. To make decisions of black-box approaches transparent, different solutions have been proposed. LIME is an approach to explainable AI relying on segmenting images into superpixels based on the Quick-Shift algorithm. In this paper, we present an explorative study of how different superpixel methods, namely Felzenszwalb, SLIC and Compact-Watershed, impact the generated visual explanations. We compare the resulting relevance areas with the image parts marked by a human reference. Results show that image parts selected as relevant strongly vary depending on the applied method. Quick-Shift resulted in the least and Compact-Watershed in the highest correspondence with the reference relevance areas.
PropFuzz -- An IT-Security Fuzzing Framework for Proprietary ICS Protocols
Venue: arXiv cs.CR 2019
Year: 2019

Programmable Logic Controllers are used for smart homes, in production processes or to control critical infrastructures. Modern industrial devices in the control level are often communicating over proprietary protocols on top of TCP/IP with each other and SCADA systems. The networks in which the controllers operate are usually considered as trustworthy and thereby they are not properly secured. Due to the growing connectivity caused by the Internet of Things (IoT) and Industry 4.0 the security risks are rising. Therefore, the demand of security assessment tools for industrial networks is high. In this paper, we introduce a new fuzzing framework called PropFuzz, which is capable to fuzz proprietary industrial control system protocols and monitor the behavior of the controller. Furthermore, we present first results of a security assessment with our framework.
Quasiperiodic bobbin lace patterns
Venue: arXiv cs.DM 2019
Year: 2019

Bobbin lace is a fibre art form in which threads are braided together to form a fabric, often with a very detailed and complex design. In traditional practice, each region of the fabric is filled with a periodic texture. We establish the groundwork for non-periodic lace patterns and present three new quasiperiodic families based on Sturmian words, the Penrose tiling by thick and thin rhombs and the Ammann-bar decoration of the Penrose tiling.
Cross-lingual Parsing with Polyglot Training and Multi-treebank Learning: A Faroese Case Study
Venue: arXiv cs.CL 2019
Year: 2019

Cross-lingual dependency parsing involves transferring syntactic knowledge from one language to another. It is a crucial component for inducing dependency parsers in low-resource scenarios where no training data for a language exists. Using Faroese as the target language, we compare two approaches using annotation projection: first, projecting from multiple monolingual source models; second, projecting from a single polyglot model which is trained on the combination of all source languages. Furthermore, we reproduce multi-source projection (Tyers et al., 2018), in which dependency trees of multiple sources are combined. Finally, we apply multi-treebank modelling to the projected treebanks, in addition to or alternatively to polyglot modelling on the source side. We find that polyglot training on the source languages produces an overall trend of better results on the target language but the single best result for the target language is obtained by projecting from monolingual source parsing models and then training multi-treebank POS tagging and parsing models on the target side.
A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method
Venue: arXiv cs.LG 2019
Year: 2019

Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods.
Uncritical polarized groups: The impact of spreading fake news as fact in social networks
Venue: arXiv cs.SI 2019
Year: 2019

The spread of ideas in online social networks is a crucial phenomenon to understand nowadays the proliferation of fake news and their impact in democracies. This makes necessary to use models that mimic the circulation of rumors. The law of large numbers as well as the probability distribution of contact groups allow us to construct a model with a minimum number of hypotheses. Moreover, we can analyze with this model the presence of very polarized groups of individuals (humans or bots) who spread a rumor as soon as they know about it. Given only the initial number of individuals who know any news, in a population connected by an instant messaging application, we first deduce from our model a simple function of time to study the rumor propagation. We then prove that the polarized groups can be detected and quantified from empirical data. Finally, we also predict the time required by any rumor to reach a fixed percentage of the population.
Heterogeneous Graph Matching Networks
Venue: arXiv cs.CR 2019
Year: 2019

Information systems have widely been the target of malware attacks. Traditional signature-based malicious program detection algorithms can only detect known malware and are prone to evasion techniques such as binary obfuscation, while behavior-based approaches highly rely on the malware training samples and incur prohibitively high training cost. To address the limitations of existing techniques, we propose MatchGNet, a heterogeneous Graph Matching Network model to learn the graph representation and similarity metric simultaneously based on the invariant graph modeling of the program's execution behaviors. We conduct a systematic evaluation of our model and show that it is accurate in detecting malicious program behavior and can help detect malware attacks with less false positives. MatchGNet outperforms the state-of-the-art algorithms in malware detection by generating 50% less false positives while keeping zero false negatives.
Obfuscation via Information Density Estimation
Venue: arXiv cs.IT 2019
Year: 2019

Identifying features that leak information about sensitive attributes is a key challenge in the design of information obfuscation mechanisms. In this paper, we propose a framework to identify information-leaking features via information density estimation. Here, features whose information densities exceed a pre-defined threshold are deemed information-leaking features. Once these features are identified, we sequentially pass them through a targeted obfuscation mechanism with a provable leakage guarantee in terms of $\mathsf{E}_γ$-divergence. The core of this mechanism relies on a data-driven estimate of the trimmed information density for which we propose a novel estimator, named the trimmed information density estimator (TIDE). We then use TIDE to implement our mechanism on three real-world datasets. Our approach can be used as a data-driven pipeline for designing obfuscation mechanisms targeting specific features.
Anatomically-Informed Data Augmentation for functional MRI with Applications to Deep Learning
Venue: arXiv cs.LG 2019
Year: 2019

The application of deep learning to build accurate predictive models from functional neuroimaging data is often hindered by limited dataset sizes. Though data augmentation can help mitigate such training obstacles, most data augmentation methods have been developed for natural images as in computer vision tasks such as CIFAR, not for medical images. This work helps to fills in this gap by proposing a method for generating new functional Magnetic Resonance Images (fMRI) with realistic brain morphology. This method is tested on a challenging task of predicting antidepressant treatment response from pre-treatment task-based fMRI and demonstrates a 26% improvement in performance in predicting response using augmented images. This improvement compares favorably to state-of-the-art augmentation methods for natural images. Through an ablative test, augmentation is also shown to substantively improve performance when applied before hyperparameter optimization. These results suggest the optimal order of operations and support the role of data augmentation method for improving predictive performance in tasks using fMRI.