OpenRaiser/Intern-Atlas huggingface dataset:
Hebbian-Descent
Venue: arXiv cs.LG 2019
Year: 2019

In this work we propose Hebbian-descent as a biologically plausible learning rule for hetero-associative as well as auto-associative learning in single layer artificial neural networks. It can be used as a replacement for gradient descent as well as Hebbian learning, in particular in online learning, as it inherits their advantages while not suffering from their disadvantages. We discuss the drawbacks of Hebbian learning as having problems with correlated input data and not profiting from seeing training patterns several times. For gradient descent we identify the derivative of the activation function as problematic especially in online learning. Hebbian-descent addresses these problems by getting rid of the activation function's derivative and by centering, i.e. keeping the neural activities mean free, leading to a biologically plausible update rule that is provably convergent, does not suffer from the vanishing error term problem, can deal with correlated data, profits from seeing patterns several times, and enables successful online learning when centering is used. We discuss its relationship to Hebbian learning, contrastive learning, and gradient decent and show that in case of a strictly positive derivative of the activation function Hebbian-descent leads to the same update rule as gradient descent but for a different loss function. In this case Hebbian-descent inherits the convergence properties of gradient descent, but we also show empirically that it converges when the derivative of the activation function is only non-negative, such as for the step function for example. Furthermore, in case of the mean squared error loss Hebbian-descent can be understood as the difference between two Hebb-learning steps, which in case of an invertible and integrable activation function actually optimizes a generalized linear model. ...
ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks
Venue: arXiv cs.CV 2019
Year: 2019

Large-scale face recognition in-the-wild has been recently achieved matured performance in many real work applications. However, such systems are built on GPU platforms and mostly deploy heavy deep network architectures. Given a high-performance heavy network as a teacher, this work presents a simple and elegant teacher-student learning paradigm, namely ShrinkTeaNet, to train a portable student network that has significantly fewer parameters and competitive accuracy against the teacher network. Far apart from prior teacher-student frameworks mainly focusing on accuracy and compression ratios in closed-set problems, our proposed teacher-student network is proved to be more robust against open-set problem, i.e. large-scale face recognition. In addition, this work introduces a novel Angular Distillation Loss for distilling the feature direction and the sample distributions of the teacher's hypersphere to its student. Then ShrinkTeaNet framework can efficiently guide the student's learning process with the teacher's knowledge presented in both intermediate and last stages of the feature embedding. Evaluations on LFW, CFP-FP, AgeDB, IJB-B and IJB-C Janus, and MegaFace with one million distractors have demonstrated the efficiency of the proposed approach to learn robust student networks which have satisfying accuracy and compact sizes. Our ShrinkTeaNet is able to support the light-weight architecture achieving high performance with 99.77% on LFW and 95.64% on large-scale Megaface protocols.
DIANet: Dense-and-Implicit Attention Network
Venue: arXiv cs.CV 2019
Year: 2019

Attention networks have successfully boosted the performance in various vision problems. Previous works lay emphasis on designing a new attention module and individually plug them into the networks. Our paper proposes a novel-and-simple framework that shares an attention module throughout different network layers to encourage the integration of layer-wise information and this parameter-sharing module is referred as Dense-and-Implicit-Attention (DIA) unit. Many choices of modules can be used in the DIA unit. Since Long Short Term Memory (LSTM) has a capacity of capturing long-distance dependency, we focus on the case when the DIA unit is the modified LSTM (refer as DIA-LSTM). Experiments on benchmark datasets show that the DIA-LSTM unit is capable of emphasizing layer-wise feature interrelation and leads to significant improvement of image classification accuracy. We further empirically show that the DIA-LSTM has a strong regularization ability on stabilizing the training of deep networks by the experiments with the removal of skip connections or Batch Normalization in the whole residual network. The code is released at https://github.com/gbup-group/DIANet.
Counting Homomorphisms Modulo a Prime Number
Venue: arXiv cs.CC 2019
Year: 2019

Counting problems in general and counting graph homomorphisms in particular have numerous applications in combinatorics, computer science, statistical physics, and elsewhere. One of the most well studied problems in this area is #GraphHom(H) --- the problem of finding the number of homomorphisms from a given graph G to the graph H. Not only the complexity of this basic problem is known, but also of its many variants for digraphs, more general relational structures, graphs with weights, and others. In this paper we consider a modification of #GraphHom(H), the #_p GraphHom(H) problem, p a prime number: Given a graph G, find the number of homomorphisms from G to H modulo p. In a series of papers Faben and Jerrum, and Goebel et al. determined the complexity of #_2 GraphHom(H) in the case H (or, in fact, a certain graph derived from H) is square-free, that is, does not contain a 4-cycle. Also, Goebel et al. found the complexity of #_p GraphHom(H) for an arbitrary prime p when H is a tree. Here we extend the above result to show that the #_p GraphHom(H) problem is #_p P-hard whenever the derived graph associated with H is square-free and is not a star, which completely classifies the complexity of #_p GraphHom(H) for square-free graphs H.
TEE-aided Write Protection Against Privileged Data Tampering
Venue: arXiv cs.CR 2019
Year: 2019

Unauthorized data alteration has been a longstanding threat since the emergence of malware. System and application software can be reinstalled and hardware can be replaced, but user data is priceless in many cases. Especially in recent years, ransomware has become high-impact due to its direct monetization model. State-of-the-art defenses are mostly based on known signature or behavior analysis, and more importantly, require an uncompromised OS kernel. However, malware with the highest software privileges has shown its obvious existence. We propose to move from current detection/recovery based mechanisms to data loss prevention, where the focus is on armoring data instead of counteracting malware. Our solution, Inuksuk, relies on today's Trusted Execution Environments (TEEs), as available both on the CPU and storage device, to achieve programmable write protection. We back up a copy of user-selected files as write-protected at all times, and subsequent updates are written as new versions securely through TEE. We implement Inuksuk on Windows 7 and 10, and Linux (Ubuntu); our core design is OS and application agnostic, and incurs no run-time performance penalty for applications. File transfer disruption can be eliminated or alleviated through access modes and customizable update policies (e.g., interval, granularity). For Inuksuk's adoptability in modern OSes, we have also ported Flicker (EuroSys 2008), a defacto standard tool for in-OS privileged TEE management, to the latest 64-bit Windows.
Auditory Separation of a Conversation from Background via Attentional Gating
Venue: arXiv cs.SD 2019
Year: 2019

We present a model for separating a set of voices out of a sound mixture containing an unknown number of sources. Our Attentional Gating Network (AGN) uses a variable attentional context to specify which speakers in the mixture are of interest. The attentional context is specified by an embedding vector which modifies the processing of a neural network through an additive bias. Individual speaker embeddings are learned to separate a single speaker while superpositions of the individual speaker embeddings are used to separate sets of speakers. We first evaluate AGN on a traditional single speaker separation task and show an improvement of 9% with respect to comparable models. Then, we introduce a new task to separate an arbitrary subset of voices from a mixture of an unknown-sized set of voices, inspired by the human ability to separate a conversation of interest from background chatter at a cafeteria. We show that AGN is the only model capable of solving this task, performing only 7% worse than on the single speaker separation task.
Ensemble Decision Systems for General Video Game Playing
Venue: arXiv cs.AI 2019
Year: 2019

Ensemble Decision Systems offer a unique form of decision making that allows a collection of algorithms to reason together about a problem. Each individual algorithm has its own inherent strengths and weaknesses, and often it is difficult to overcome the weaknesses while retaining the strengths. Instead of altering the properties of the algorithm, the Ensemble Decision System augments the performance with other algorithms that have complementing strengths. This work outlines different options for building an Ensemble Decision System as well as providing analysis on its performance compared to the individual components of the system with interesting results, showing an increase in the generality of the algorithms without significantly impeding performance.
An Unconditionally Stable First-Order Constraint Solver for Multibody Systems
Venue: arXiv cs.NA 2019
Year: 2019

This article describes an absolutely stable, first-order constraint solverfor multi-rigid body systems that calculates (predicts) constraint forces for typical bilateral and unilateral constraints, contact constraints with friction, and many other constraint types. Redundant constraints do not pose numerical problems or require regularization. Coulomb friction for contact is modeled using a true friction cone, rather than a linearized approximation. The computational expense of the solver is dependent upon the types of constraints present in the input. The hardest (in a computational complexity sense) inputs are reducible to solving convex optimization problems, i.e., polynomial time solvable. The simplest inputs require only solving a linear system. The solver is L-stable, which will imply that the forces due to constraints induce no computational stiffness into the multi-body dynamics differential equations. This approach is targeted to multibodies simulated with coarse accuracy, subject to computational stiffness arising from constraints, and where the number of constraint equations is not large compared to the number of multibody position and velocity state variables. For such applications, the approach should prove far faster than using other implicit integration approaches. I assess the approach on some fundamental multibody dynamics problems.
Feature Map Transform Coding for Energy-Efficient CNN Inference
Venue: arXiv cs.CV 2019
Year: 2019

Convolutional neural networks (CNNs) achieve state-of-the-art accuracy in a variety of tasks in computer vision and beyond. One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements. The latter often dominates the energy footprint on modern hardware. In this paper, we introduce a lossy transform coding approach, inspired by image and video compression, designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results. Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps, which are highly correlated, with variable length coding. Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2. We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% in the memory energy footprint, compared to quantized network, with negligible impact on accuracy. When allowing accuracy degradation of up to 2%, the reduction of 60% is achieved. A reference implementation is available at https://github.com/CompressTeam/TransformCodingInference
On Coresets for Regularized Loss Minimization
Venue: arXiv cs.LG 2019
Year: 2019

We design and mathematically analyze sampling-based algorithms for regularized loss minimization problems that are implementable in popular computational models for large data, in which the access to the data is restricted in some way. Our main result is that if the regularizer's effect does not become negligible as the norm of the hypothesis scales, and as the data scales, then a uniform sample of modest size is with high probability a coreset. In the case that the loss function is either logistic regression or soft-margin support vector machines, and the regularizer is one of the common recommended choices, this result implies that a uniform sample of size $O(d \sqrt{n})$ is with high probability a coreset of $n$ points in $\Re^d$. We contrast this upper bound with two lower bounds. The first lower bound shows that our analysis of uniform sampling is tight; that is, a smaller uniform sample will likely not be a core set. The second lower bound shows that in some sense uniform sampling is close to optimal, as significantly smaller core sets do not generally exist.
Data Race Prediction for Inaccurate Traces
Venue: arXiv cs.PL 2019
Year: 2019

Happens-before based data race prediction methods infer from a trace of events a partial order to check if one event happens before another event. If two two write events are unordered, they are in a race. We observe that common tracing methods provide no guarantee that the trace order corresponds to an actual program run. The consequence of inaccurate tracing is that results (races) reported are inaccurate. We introduce diagnostic methods to examine if (1) a race is guaranteed to be correct regardless of any potential inaccuracies, (2) maybe is incorrect due to inaccurate tracing. We have fully implemented the approach and provide for an empirical comparison with state of the art happens-before based race predictors such as FastTrack and SHB.
Legal Concerns and Challenges in Cloud Computing
Venue: arXiv cs.CY 2019
Year: 2019

Legal issues have risen with the changing landscape of computing, especially when the service, data and infrastructure is not owned by the user. With the Cloud, the question arises as to who is in the possession of the data. The Cloud provider can be considered as a legal custodian, owner or possessor of the data thereby causing complexities in legal matters around trademark infringement, privacy of users and their data, abuse and security. By introducing Cloud design focusing on privacy, legal as a service on a Cloud and service provider accountability, users can expect the service providers to be accountable for privacy and data in addition to their regular SLAs.
Learning Policies from Human Data for Skat
Venue: arXiv cs.AI 2019
Year: 2019

Decision-making in large imperfect information games is difficult. Thanks to recent success in Poker, Counterfactual Regret Minimization (CFR) methods have been at the forefront of research in these games. However, most of the success in large games comes with the use of a forward model and powerful state abstractions. In trick-taking card games like Bridge or Skat, large information sets and an inability to advance the simulation without fully determinizing the state make forward search problematic. Furthermore, state abstractions can be especially difficult to construct because the precise holdings of each player directly impact move values. In this paper we explore learning model-free policies for Skat from human game data using deep neural networks (DNN). We produce a new state-of-the-art system for bidding and game declaration by introducing methods to a) directly vary the aggressiveness of the bidder and b) declare games based on expected value while mitigating issues with rarely observed state-action pairs. Although cardplay policies learned through imitation are slightly weaker than the current best search-based method, they run orders of magnitude faster. We also explore how these policies could be learned directly from experience in a reinforcement learning setting and discuss the value of incorporating human data for this task.
PNUNet: Anomaly Detection using Positive-and-Negative Noise based on Self-Training Procedure
Venue: arXiv cs.CV 2019
Year: 2019

We propose the novel framework for anomaly detection in images. Our new framework, PNUNet, is based on many normal data and few anomalous data. We assume that some noises are added to the input images and learn to remove the noise. In addition, the proposed method achieves significant performance improvement by updating the noise assumed in the inputs using a self-training framework. The experimental results for the benchmark datasets show the usefulness of our new anomaly detection framework.
A Practical Spectrum Sharing Scheme for Cognitive Radio Networks: Design and Experiments
Venue: arXiv cs.NI 2019
Year: 2019

Spectrum shortage is a fundamental problem in wireless networks and this problem becomes increasingly acute with the rapid proliferation of wireless devices. To address this problem, spectrum sharing in the context of cognitive radio networks (CRNs) has been considered a promising solution. In this paper, we propose a practical spectrum sharing scheme for a small CRN that comprises a pair of primary users and a pair of secondary users by leveraging the multiple-input and multiple-output (MIMO) technology. In our scheme, we assume that the secondary users take full responsibility for cross-network interference cancellation (IC). We also assume that the secondary users have no knowledge about the primary network, including its signal waveform, frame structure, and network protocol. The key components of our proposed scheme are two MIMO-based interference management techniques: blind beamforming (BBF) and blind interference cancellation (BIC). We have built a prototype of our scheme on a wireless testbed and demonstrated that the prototyped secondary network can coexist with commercial Wi-Fi devices (primary users). Experimental results further show that, for a secondary device with two or three antennas, BBF and BIC achieve an average of 25dB and 33dB IC capability in an office environment, respectively.
Identity Connections in Residual Nets Improve Noise Stability
Venue: arXiv cs.LG 2019
Year: 2019

Residual Neural Networks (ResNets) achieve state-of-the-art performance in many computer vision problems. Compared to plain networks without residual connections (PlnNets), ResNets train faster, generalize better, and suffer less from the so-called degradation problem. We introduce simplified (but still nonlinear) versions of ResNets and PlnNets for which these discrepancies still hold, although to a lesser degree. We establish a 1-1 mapping between simplified ResNets and simplified PlnNets, and show that they are exactly equivalent to each other in expressive power for the same computational complexity. We conjecture that ResNets generalize better because they have better noise stability, and empirically support it for both simplified and fully-fledged networks.
TrendNets: Mapping Emerging Research Trends From Dynamic Co-Word Networks via Sparse Representation
Venue: arXiv cs.DL 2019
Year: 2019

Mapping the knowledge structure from word co-occurrences in a collection of academic papers has been widely used to provide insight into the topic evolution in an arbitrary research field. In a traditional approach, the paper collection is first divided into temporal subsets, and then a co-word network is independently depicted in a 2D map to characterize each period's trend. To effectively map emerging research trends from such a time-series of co-word networks, this paper presents TrendNets, a novel visualization methodology that highlights the rapid changes in edge weights over time. Specifically, we formulated a new convex optimization framework that decomposes the matrix constructed from dynamic co-word networks into a smooth part and a sparse part: the former represents stationary research topics, while the latter corresponds to bursty research topics. Simulation results on synthetic data demonstrated that our matrix decomposition approach achieved the best burst detection performance over four baseline methods. In experiments conducted using papers published in the past 16 years at three conferences in different fields, we showed the effectiveness of TrendNets compared to the traditional co-word representation. We have made our codes available on the Web to encourage scientific mapping in all research fields.
Unsupervised Learning of Anomaly Detection from Contaminated Image Data using Simultaneous Encoder Training
Venue: arXiv cs.CV 2019
Year: 2019

Unsupervised learning of anomaly detection in high-dimensional data, such as images, is a challenging problem recently subject to intense research. Through careful modelling of the data distribution of normal samples, it is possible to detect deviant samples, so called anomalies. Generative Adversarial Networks (GANs) can model the highly complex, high-dimensional data distribution of normal image samples, and have shown to be a suitable approach to the problem. Previously published GAN-based anomaly detection methods often assume that anomaly-free data is available for training. However, this assumption is not valid in most real-life scenarios, a.k.a. in the wild. In this work, we evaluate the effects of anomaly contaminations in the training data on state-of-the-art GAN-based anomaly detection methods. As expected, detection performance deteriorates. To address this performance drop, we propose to add an additional encoder network already at training time and show that joint generator-encoder training stratifies the latent space, mitigating the problem with contaminated data. We show experimentally that the norm of a query image in this stratified latent space becomes a highly significant cue to discriminate anomalies from normal data. The proposed method achieves state-of-the-art performance on CIFAR-10 as well as on a large, previously untested dataset with cell images.
Policy Search by Target Distribution Learning for Continuous Control
Venue: arXiv cs.LG 2019
Year: 2019

We observe that several existing policy gradient methods (such as vanilla policy gradient, PPO, A2C) may suffer from overly large gradients when the current policy is close to deterministic (even in some very simple environments), leading to an unstable training process. To address this issue, we propose a new method, called \emph{target distribution learning} (TDL), for policy improvement in reinforcement learning. TDL alternates between proposing a target distribution and training the policy network to approach the target distribution. TDL is more effective in constraining the KL divergence between updated policies, and hence leads to more stable policy improvements over iterations. Our experiments show that TDL algorithms perform comparably to (or better than) state-of-the-art algorithms for most continuous control tasks in the MuJoCo environment while being more stable in training.