OpenRaiser/Intern-Atlas huggingface dataset:
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
Venue: arXiv cs.LG 2018
Year: 2018

Several recently proposed architectures of neural networks such as ResNeXt, Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea of having multiple branches and have demonstrated improved performance in many applications. We show that one cause for such success is due to the fact that the multi-branch architecture is less non-convex in terms of duality gap. The duality gap measures the degree of intrinsic non-convexity of an optimization problem: smaller gap in relative value implies lower degree of intrinsic non-convexity. The challenge is to quantitatively measure the duality gap of highly non-convex problems such as deep neural networks. In this work, we provide strong guarantees of this quantity for two classes of network architectures. For the neural networks with arbitrary activation functions, multi-branch architecture and a variant of hinge loss, we show that the duality gap of both population and empirical risks shrinks to zero as the number of branches increases. This result sheds light on better understanding the power of over-parametrization where increasing the network width tends to make the loss surface less non-convex. For the neural networks with linear activation function and $\ell_2$ loss, we show that the duality gap of empirical risk is zero. Our two results work for arbitrary depths and adversarial data, while the analytical techniques might be of independent interest to non-convex optimization more broadly. Experiments on both synthetic and real-world datasets validate our results.
Enabling Cooperative IoT Security via Software Defined Networks (SDN)
Venue: arXiv cs.NI 2018
Year: 2018

Internet of Things (IoT) is becoming an increasingly attractive target for cybercriminals. We observe that many attacks to IoTs are launched in a collusive way, such as brute-force hacking usernames and passwords, to target at a particular victim. However, most of the time our defending mechanisms to such kind of attacks are carried out individually and independently, which leads to ineffective and weak defense. To this end, we propose to leverage Software Defined Networks (SDN) to enable cooperative security for legacy IP-based IoT devices. SDN decouples control plane and data plane, and can help bridge the knowledge divided between the application and network layers. In this paper, we discuss the IoT security problems and challenges, and present an SDN-based architecture to enable IoT security in a cooperative manner. Furthermore, we implemented a platform that can quickly share the attacking information with peer controllers and block the attacks. We carried out our experiments in both virtual and physical SDN environments with OpenFlow switches. Our evaluation results show that both environments can scale well to handle attacks, but hardware implementation is much more efficient than a virtual one.
Probabilistic Deep Learning using Random Sum-Product Networks
Venue: arXiv cs.LG 2018
Year: 2018

The need for consistent treatment of uncertainty has recently triggered increased interest in probabilistic deep learning methods. However, most current approaches have severe limitations when it comes to inference, since many of these models do not even permit to evaluate exact data likelihoods. Sum-product networks (SPNs), on the other hand, are an excellent architecture in that regard, as they allow to efficiently evaluate likelihoods, as well as arbitrary marginalization and conditioning tasks. Nevertheless, SPNs have not been fully explored as serious deep learning models, likely due to their special structural requirements, which complicate learning. In this paper, we make a drastic simplification and use random SPN structures which are trained in a "classical deep learning manner", i.e. employing automatic differentiation, SGD, and GPU support. The resulting models, called RAT-SPNs, yield prediction results comparable to deep neural networks, while still being interpretable as generative model and maintaining well-calibrated uncertainties. This property makes them highly robust under missing input features and enables them to naturally detect outliers and peculiar samples.
Deep Algorithms: designs for networks
Venue: arXiv cs.LG 2018
Year: 2018

A new design methodology for neural networks that is guided by traditional algorithm design is presented. To prove our point, we present two heuristics and demonstrate an algorithmic technique for incorporating additional weights in their signal-flow graphs. We show that with training the performance of these networks can not only exceed the performance of the initial network, but can match the performance of more-traditional neural network architectures. A key feature of our approach is that these networks are initialized with parameters that provide a known performance threshold for the architecture on a given task.
Robust Physical Layer Security for Power Domain Non-orthogonal Multiple Access-Based HetNets and HUDNs, SIC Avoidance at Eavesdroppers
Venue: arXiv cs.CR 2018
Year: 2018

In this paper, we investigate the physical layer security in downlink of Power Domain Non Orthogonal Multiple Access based heterogeneous cellular network in presence of multiple eavesdroppers. Our aim is to maximize the sum secrecy rate of the network. To this end, we formulate joint subcarrier and power allocation optimization problems to increase sum secrecy rate. Moreover, we propose a novel scheme at which the eavesdroppers are prevented from doing Successive Interference Cancellation, while legitimate users are able to do it. In practical systems, availability of eavesdroppers Channel State Information is impractical, hence we consider two scenarios: 1 Perfect CSI of the eavesdroppers, 2 imperfect CSI of the eavesdroppers. Since the proposed optimization problems are nonconvex, we adopt the well known iterative algorithm called Alternative Search Method. In this algorithm, the optimization problems are converted to two subproblems, power allocation and subcarrier allocation. We solve the power allocation problem by the Successive Convex Approximation approach and solve the subcarrier allocation subproblem, by exploiting the Mesh Adaptive Direct Search algorithm. Moreover, in order to study the optimality gap of the proposed solution method, we apply the monotonic optimization method. Moreover, we evaluate the proposed scheme for secure massive connectivity in 5G networks. Numerical results highlight that the proposed scheme significantly improves the sum secrecy rate compared with the conventional case at which the eavesdroppers are able to apply SIC.
Determining the Generalized Hamming Weight Hierarchy of the Binary Projective Reed-Muller Code
Venue: arXiv cs.IT 2018
Year: 2018

Projective Reed-Muller codes correspond to subcodes of the Reed-Muller code in which the polynomials being evaluated to yield codewords, are restricted to be homogeneous. The Generalized Hamming Weights (GHW) of a code ${\cal C}$, identify for each dimension $ν$, the smallest size of the support of a subcode of ${\cal C}$ of dimension $ν$. The GHW of a code are of interest in assessing the vulnerability of a code in a wiretap channel setting. It is also of use in bounding the state complexity of the trellis representation of the code. In prior work by the same authors, a code-shortening algorithm was employed to derive upper bounds on the GHW of binary projective, Reed-Muller (PRM) codes. In the present paper, we derive a matching lower bound by adapting the proof techniques used originally for Reed-Muller (RM) codes by Wei. This results in a characterization of the GHW hierarchy of binary PRM codes.
TextRay: Mining Clinical Reports to Gain a Broad Understanding of Chest X-rays
Venue: arXiv cs.CV 2018
Year: 2018

The chest X-ray (CXR) is by far the most commonly performed radiological examination for screening and diagnosis of many cardiac and pulmonary diseases. There is an immense world-wide shortage of physicians capable of providing rapid and accurate interpretation of this study. A radiologist-driven analysis of over two million CXR reports generated an ontology including the 40 most prevalent pathologies on CXR. By manually tagging a relatively small set of sentences, we were able to construct a training set of 959k studies. A deep learning model was trained to predict the findings given the patient frontal and lateral scans. For 12 of the findings we compare the model performance against a team of radiologists and show that in most cases the radiologists agree on average more with the algorithm than with each other.
Constrained Counting and Sampling: Bridging the Gap between Theory and Practice
Venue: arXiv cs.LO 2018
Year: 2018

Constrained counting and sampling are two fundamental problems in Computer Science with numerous applications, including network reliability, privacy, probabilistic reasoning, and constrained-random verification. In constrained counting, the task is to compute the total weight, subject to a given weighting function, of the set of solutions of the given constraints. In constrained sampling, the task is to sample randomly, subject to a given weighting function, from the set of solutions to a set of given constraints. Consequently, constrained counting and sampling have been subject to intense theoretical and empirical investigations over the years. Prior work, however, offered either heuristic techniques with poor guarantees of accuracy or approaches with proven guarantees but poor performance in practice. In this thesis, we introduce a novel hashing-based algorithmic framework for constrained sampling and counting that combines the classical algorithmic technique of universal hashing with the dramatic progress made in combinatorial reasoning tools, in particular, SAT and SMT, over the past two decades. The resulting frameworks for counting (ApproxMC2) and sampling (UniGen) can handle formulas with up to million variables representing a significant boost up from the prior state of the art tools' capability to handle few hundreds of variables. If the initial set of constraints is expressed as Disjunctive Normal Form (DNF), ApproxMC is the only known Fully Polynomial Randomized Approximation Scheme (FPRAS) that does not involve Monte Carlo steps. By exploiting the connection between definability of formulas and variance of the distribution of solutions in a cell defined by 3-universal hash functions, we introduced an algorithmic technique, MIS, that reduced the size of XOR constraints employed in the underlying universal hash functions by as much as two orders of magnitude.
Deploying Deep Ranking Models for Search Verticals
Venue: arXiv cs.IR 2018
Year: 2018

In this paper, we present an architecture executing a complex machine learning model such as a neural network capturing semantic similarity between a query and a document; and deploy to a real-world production system serving 500M+users. We present the challenges that arise in a real-world system and how we solve them. We demonstrate that our architecture provides competitive modeling capability without any significant performance impact to the system in terms of latency. Our modular solution and insights can be used by other real-world search systems to realize and productionize recent gains in neural networks.
Corpus Conversion Service: A Machine Learning Platform to Ingest Documents at Scale
Venue: arXiv cs.DL 2018
Year: 2018

Over the past few decades, the amount of scientific articles and technical literature has increased exponentially in size. Consequently, there is a great need for systems that can ingest these documents at scale and make the contained knowledge discoverable. Unfortunately, both the format of these documents (e.g. the PDF format or bitmap images) as well as the presentation of the data (e.g. complex tables) make the extraction of qualitative and quantitive data extremely challenging. In this paper, we present a modular, cloud-based platform to ingest documents at scale. This platform, called the Corpus Conversion Service (CCS), implements a pipeline which allows users to parse and annotate documents (i.e. collect ground-truth), train machine-learning classification algorithms and ultimately convert any type of PDF or bitmap-documents to a structured content representation format. We will show that each of the modules is scalable due to an asynchronous microservice architecture and can therefore handle massive amounts of documents. Furthermore, we will show that our capability to gather ground-truth is accelerated by machine-learning algorithms by at least one order of magnitude. This allows us to both gather large amounts of ground-truth in very little time and obtain very good precision/recall metrics in the range of 99\% with regard to content conversion to structured output. The CCS platform is currently deployed on IBM internal infrastructure and serving more than 250 active users for knowledge-engineering project engagements.
Data-driven Probabilistic Atlases Capture Whole-brain Individual Variation
Venue: arXiv cs.LG 2018
Year: 2018

Probabilistic atlases provide essential spatial contextual information for image interpretation, Bayesian modeling, and algorithmic processing. Such atlases are typically constructed by grouping subjects with similar demographic information. Importantly, use of the same scanner minimizes inter-group variability. However, generalizability and spatial specificity of such approaches is more limited than one might like. Inspired by Commowick "Frankenstein's creature paradigm" which builds a personal specific anatomical atlas, we propose a data-driven framework to build a personal specific probabilistic atlas under the large-scale data scheme. The data-driven framework clusters regions with similar features using a point distribution model to learn different anatomical phenotypes. Regional structural atlases and corresponding regional probabilistic atlases are used as indices and targets in the dictionary. By indexing the dictionary, the whole brain probabilistic atlases adapt to each new subject quickly and can be used as spatial priors for visualization and processing. The novelties of this approach are (1) it provides a new perspective of generating personal specific whole brain probabilistic atlases (132 regions) under data-driven scheme across sites. (2) The framework employs the large amount of heterogeneous data (2349 images). (3) The proposed framework achieves low computational cost since only one affine registration and Pearson correlation operation are required for a new subject. Our method matches individual regions better with higher Dice similarity value when testing the probabilistic atlases. Importantly, the advantage the large-scale scheme is demonstrated by the better performance of using large-scale training data (1888 images) than smaller training set (720 images).
Optimization vs. Reinforcement Learning for Wirelessly Powered Sensor Networks
Venue: arXiv cs.IT 2018
Year: 2018

We consider a sensing application where the sensor nodes are wirelessly powered by an energy beacon. We focus on the problem of jointly optimizing the energy allocation of the energy beacon to different sensors and the data transmission powers of the sensors in order to minimize the field reconstruction error at the sink. In contrast to the standard ideal linear energy harvesting (EH) model, we consider practical non-linear EH models. We investigate this problem under two different frameworks: i) an optimization approach where the energy beacon knows the utility function of the nodes, channel state information and the energy harvesting characteristics of the devices; hence optimal power allocation strategies can be designed using an optimization problem and ii) a learning approach where the energy beacon decides on its strategies adaptively with battery level information and feedback on the utility function. Our results illustrate that deep reinforcement learning approach can obtain the same error levels with the optimization approach and provides a promising alternative to the optimization framework.
Polar Code Moderate Deviation: Recovering the Scaling Exponent
Venue: arXiv cs.IT 2018
Year: 2018

In 2008 Arikan proposed polar coding [arXiv:0807.3917] which we summarize as follows: (a) From the root channel $W$ synthesize recursively a series of channels $W_N^{(1)},\dotsc,W_N^{(N)}$. (b) Select sophisticatedly a subset $A$ of synthetic channels. (c) Transmit information using synthetic channels indexed by $A$ and freeze the remaining synthetic channels. Arikan gives each synthetic channel a score (called the Bhattacharyya parameter) that determines whether it should be selected or frozen. As $N$ grows, a majority of the scores are either very high or very low, i.e., they polarize. By characterizing how fast they polarize, Arikan showed that polar coding is able to produce a series of codes that achieve capacity on symmetric binary-input memoryless channels. In measuring how the scores polarize the relation among block length, gap to capacity, and block error probability are studied. In particular, the error exponent regime fixes the gap to capacity and varies the other two. The scaling exponent regime fixes the block error probability and varies the other two. The moderate deviation regime varies all three factors at once. The latest result [arxiv:1501.02444, Theorem 7] in the moderate deviation regime does not imply the scaling exponent regime as a special case. We give a result that does. (See Corollary 8.)
On the Impact of Guest Traffic in Open-Access Domestic Broadband Sharing Schemes
Venue: arXiv cs.NI 2018
Year: 2018

Open-access domestic broadband connection sharing constitutes a voluntary practice that is associated with societal, economic and public-safety benefits. Despite this fact, broadband subscribers are usually hesitant to freely share their broadband connection with guests for a multitude of reasons; one of them being sharing their network might hinder their own broadband quality of experience. In this paper, we investigate experimentally the impact of uplink guest traffic on the sharer's broadband quality of experience under both generic and broadband-sharing-specific packet scheduling policies. Both guest-user traffic and access point profiles employed in our study are developed by analyzing real-world traffic traces and measurements, captured from actual broadband sharing networking environments. Our results validate the suitability of hybrid packet scheduling policies for broadband sharing schemes and show that only a few dozen kilobytes per second of uplink guest traffic can be tolerated by sharers without hampering their broadband quality of experience. In this context, we show that the selection of the most appropriate packet scheduling policy for broadband sharing, as well as its respective configuration, depend largely on the capacity of the broadband connection and the policy's packet-dropping behavior on guest traffic.
MEBN-RM: A Mapping between Multi-Entity Bayesian Network and Relational Model
Venue: arXiv cs.LG 2018
Year: 2018

Multi-Entity Bayesian Network (MEBN) is a knowledge representation formalism combining Bayesian Networks (BN) with First-Order Logic (FOL). MEBN has sufficient expressive power for general-purpose knowledge representation and reasoning. Developing a MEBN model to support a given application is a challenge, requiring definition of entities, relationships, random variables, conditional dependence relationships, and probability distributions. When available, data can be invaluable both to improve performance and to streamline development. By far the most common format for available data is the relational database (RDB). Relational databases describe and organize data according to the Relational Model (RM). Developing a MEBN model from data stored in an RDB therefore requires mapping between the two formalisms. This paper presents MEBN-RM, a set of mapping rules between key elements of MEBN and RM. We identify links between the two languages (RM and MEBN) and define four levels of mapping from elements of RM to elements of MEBN. These definitions are implemented in the MEBN-RM algorithm, which converts a relational schema in RM to a partial MEBN model. Through this research, the software has been released as a MEBN-RM open-source software tool. The method is illustrated through two example use cases using MEBN-RM to develop MEBN models: a Critical Infrastructure Defense System and a Smart Manufacturing System.
Reference Model of Multi-Entity Bayesian Networks for Predictive Situation Awareness
Venue: arXiv cs.AI 2018
Year: 2018

During the past quarter-century, situation awareness (SAW) has become a critical research theme, because of its importance. Since the concept of SAW was first introduced during World War I, various versions of SAW have been researched and introduced. Predictive Situation Awareness (PSAW) focuses on the ability to predict aspects of a temporally evolving situation over time. PSAW requires a formal representation and a reasoning method using such a representation. A Multi-Entity Bayesian Network (MEBN) is a knowledge representation formalism combining Bayesian Networks (BN) with First-Order Logic (FOL). MEBN can be used to represent uncertain situations (supported by BN) as well as complex situations (supported by FOL). Also, efficient reasoning algorithms for MEBN have been developed. MEBN can be a formal representation to support PSAW and has been used for several PSAW systems. Although several MEBN applications for PSAW exist, very little work can be found in the literature that attempts to generalize a MEBN model to support PSAW. In this research, we define a reference model for MEBN in PSAW, called a PSAW-MEBN reference model. The PSAW-MEBN reference model enables us to easily develop a MEBN model for PSAW by supporting the design of a MEBN model for PSAW. In this research, we introduce two example use cases using the PSAW-MEBN reference model to develop MEBN models to support PSAW: a Smart Manufacturing System and a Maritime Domain Awareness System.
Segment-Based Credit Scoring Using Latent Clusters in the Variational Autoencoder
Venue: arXiv cs.CE 2018
Year: 2018

Identifying customer segments in retail banking portfolios with different risk profiles can improve the accuracy of credit scoring. The Variational Autoencoder (VAE) has shown promising results in different research domains, and it has been documented the powerful information embedded in the latent space of the VAE. We use the VAE and show that transforming the input data into a meaningful representation, it is possible to steer configurations in the latent space of the VAE. Specifically, the Weight of Evidence (WoE) transformation encapsulates the propensity to fall into financial distress and the latent space in the VAE preserves this characteristic in a well-defined clustering structure. These clusters have considerably different risk profiles and therefore are suitable not only for credit scoring but also for marketing and customer purposes. This new clustering methodology offers solutions to some of the challenges in the existing clustering algorithms, e.g., suggests the number of clusters, assigns cluster labels to new customers, enables cluster visualization, scales to large datasets, captures non-linear relationships among others. Finally, for portfolios with a large number of customers in each cluster, developing one classifier model per cluster can improve the credit scoring assessment.