OpenRaiser/Intern-Atlas huggingface dataset:
Improved Sentence Modeling using Suffix Bidirectional LSTM
Venue: arXiv cs.LG 2018
Year: 2018

Recurrent neural networks have become ubiquitous in computing representations of sequential data, especially textual data in natural language processing. In particular, Bidirectional LSTMs are at the heart of several neural models achieving state-of-the-art performance in a wide variety of tasks in NLP. However, BiLSTMs are known to suffer from sequential bias - the contextual representation of a token is heavily influenced by tokens close to it in a sentence. We propose a general and effective improvement to the BiLSTM model which encodes each suffix and prefix of a sequence of tokens in both forward and reverse directions. We call our model Suffix Bidirectional LSTM or SuBiLSTM. This introduces an alternate bias that favors long range dependencies. We apply SuBiLSTMs to several tasks that require sentence modeling. We demonstrate that using SuBiLSTM instead of a BiLSTM in existing models leads to improvements in performance in learning general sentence representations, text classification, textual entailment and paraphrase detection. Using SuBiLSTM we achieve new state-of-the-art results for fine-grained sentiment classification and question classification.
Dynamic Monopolies in Reversible Bootstrap Percolation
Venue: arXiv cs.DS 2018
Year: 2018

We study an extremal question for the (reversible) $r-$bootstrap percolation processes. Given a graph and an initial configuration where each vertex is active or inactive, in the $r-$bootstrap percolation process the following rule is applied in discrete-time rounds: each vertex gets active if it has at least $r$ active neighbors, and an active vertex stays active forever. In the reversible $r$-bootstrap percolation, each vertex gets active if it has at least $r$ active neighbors, and inactive otherwise. We consider the following question on the $d$-dimensional torus: how many vertices should be initially active so that the whole graph becomes active? Our results settle an open problem by Balister, Bollobás, Johnson, and Walters and generalize the results by Flocchini, Lodi, Luccio, Pagli, and Santoro.
Can machine learning identify interesting mathematics? An exploration using empirically observed laws
Venue: arXiv cs.LG 2018
Year: 2018

We explore the possibility of using machine learning to identify interesting mathematical structures by using certain quantities that serve as fingerprints. In particular, we extract features from integer sequences using two empirical laws: Benford's law and Taylor's law and experiment with various classifiers to identify whether a sequence is, for example, nice, important, multiplicative, easy to compute or related to primes or palindromes.
Overcoming catastrophic forgetting problem by weight consolidation and long-term memory
Venue: arXiv cs.LG 2018
Year: 2018

Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge. Here, we propose a new approach to sequential learning which leverages the recent discovery of adversarial examples. We use adversarial subspaces from previous tasks to enable learning of new tasks with less interference. We apply our method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6, (task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and combine our Adversarial Direction (AD) method with the recently proposed Elastic Weight Consolidation (EWC) method for sequential learning. We train each task for 20 epochs, which yields good initial performance (99.24% correct task 1 performance). After training task 2, and then task 3, both plain gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95% for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still achieves 94.53% correct on task 1. We obtain similar results with a much more difficult disjoint CIFAR10 task, which to our knowledge had not been attempted before (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3 for AD+EWC, while PGD and EWC both fall to chance level). Our results suggest that AD+EWC can provide better sequential learning performance than either PGD or EWC.
Efficient Atlasing and Search of Configuration Spaces of Point-Sets Constrained by Distance Intervals
Venue: arXiv cs.CG 2018
Year: 2018

For configurations of point-sets that are pairwise constrained by distance intervals, the EASAL software implements a suite of algorithms that characterize the structure and geometric properties of the configuration space. The algorithms generate, describe and explore these configuration spaces using generic rigidity properties, classical results for stratification of semi-algebraic sets, and new results for efficient sampling by convex parametrization. The paper reviews the key theoretical underpinnings, major algorithms and their implementation. The paper outlines the main applications such as the computation of free energy and kinetics of assembly of supramolecular structures or of clusters in colloidal and soft materials. In addition, the paper surveys select experimental results and comparisons.
Unsupervised Learning of Neural Networks to Explain Neural Networks
Venue: arXiv cs.CV 2018
Year: 2018

This paper presents an unsupervised method to learn a neural network, namely an explainer, to interpret a pre-trained convolutional neural network (CNN), i.e., explaining knowledge representations hidden in middle conv-layers of the CNN. Given feature maps of a certain conv-layer of the CNN, the explainer performs like an auto-encoder, which first disentangles the feature maps into object-part features and then inverts object-part features back to features of higher conv-layers of the CNN. More specifically, the explainer contains interpretable conv-layers, where each filter disentangles the representation of a specific object part from chaotic input feature maps. As a paraphrase of CNN features, the disentangled representations of object parts help people understand the logic inside the CNN. We also learn the explainer to use object-part features to reconstruct features of higher CNN layers, in order to minimize loss of information during the feature disentanglement. More crucially, we learn the explainer via network distillation without using any annotations of sample labels, object parts, or textures for supervision. We have applied our method to different types of CNNs for evaluation, and explainers have significantly boosted the interpretability of CNN features.
Optimal DR-Submodular Maximization and Applications to Provable Mean Field Inference
Venue: arXiv cs.LG 2018
Year: 2018

Mean field inference in probabilistic models is generally a highly nonconvex problem. Existing optimization methods, e.g., coordinate ascent algorithms, can only generate local optima. In this work we propose provable mean filed methods for probabilistic log-submodular models and its posterior agreement (PA) with strong approximation guarantees. The main algorithmic technique is a new Double Greedy scheme, termed DR-DoubleGreedy, for continuous DR-submodular maximization with box-constraints. It is a one-pass algorithm with linear time complexity, reaching the optimal 1/2 approximation ratio, which may be of independent interest. We validate the superior performance of our algorithms against baseline algorithms on both synthetic and real-world datasets.
Lightweight (Reverse) Fuzzy Extractor with Multiple Referenced PUF Responses
Venue: arXiv cs.CR 2018
Year: 2018

A Physical unclonable functions (PUF), alike a fingerprint, exploits manufacturing randomness to endow each physical item with a unique identifier. One primary PUF application is the secure derivation of volatile cryptographic keys using a fuzzy extractor comprising of two procedures: i) secure sketch; and ii) entropy extraction. Although the entropy extractor can be lightweight, the overhead of the secure sketch responsible correcting naturally noisy PUF responses is usually costly. We observe that, in general, response unreliability with respect to a enrolled reference measurement increases with increasing differences between the in-the-field PUF operating condition and the operating condition used in evaluating the enrolled reference response. For the first time, we exploit such an important but inadvertent observation. In contrast to the conventional single reference response enrollment, we propose enrolling multiple reference responses (MRR) subject to the same challenge but under multiple distinct operating conditions. The critical observation here is that one of the reference operating conditions is likely to be closer to the operating condition of the field deployed PUF, thus, resulting in minimizing the expected unreliability when compared to the single reference under the nominal condition. Overall, MRR greatly reduces the demand for the expected number of erroneous bits for correction and, subsequently, achieve a significant reduction in the error correction overhead. The significant implementation efficiency gains from the proposed MRR method is demonstrated from software implementations of fuzzy extractors on batteryless resource constraint computational radio frequency identification devices, where realistic PUF data is collected from the embedded intrinsic SRAM PUFs.
On Deep Ensemble Learning from a Function Approximation Perspective
Venue: arXiv cs.LG 2018
Year: 2018

In this paper, we propose to provide a general ensemble learning framework based on deep learning models. Given a group of unit models, the proposed deep ensemble learning framework will effectively combine their learning results via a multilayered ensemble model. In the case when the unit model mathematical mappings are bounded, sigmoidal and discriminatory, we demonstrate that the deep ensemble learning framework can achieve a universal approximation of any functions from the input space to the output space. Meanwhile, to achieve such a performance, the deep ensemble learning framework also impose a strict constraint on the number of involved unit models. According to the theoretic proof provided in this paper, given the input feature space of dimension d, the required unit model number will be 2d, if the ensemble model involves one single layer. Furthermore, as the ensemble component goes deeper, the number of required unit model is proved to be lowered down exponentially.
Neural networks with dynamical coefficients and adjustable connections on the basis of integrated backpropagation
Venue: arXiv cs.NE 2018
Year: 2018

We consider artificial neurons which will update their weight coefficients with an internal rule based on backpropagation, rather than using it as an external training procedure. To achieve this we include the backpropagation error estimate as a separate entity in all the neuron models and perform its exchange along the synaptic connections. In addition to this we add some special type of neurons with reference inputs, which will serve as a base source of error estimates for the whole network. Finally, we introduce a training control signal for all the neurons, which can enable the correction of weights and the exchange of error estimates. For recurrent neural networks we also demonstrate how to integrate backpropagation through time into their formalism with the help of some stack memory for reference inputs and external data inputs of neurons. Also, for widely used neural networks, such as long short-term memory, radial basis function networks, multilayer perceptrons and convolutional neural networks, we demonstrate their alternative description within the framework of our new formalism.
Learning Pixel-wise Labeling from the Internet without Human Interaction
Venue: arXiv cs.CV 2018
Year: 2018

Deep learning stands at the forefront in many computer vision tasks. However, deep neural networks are usually data-hungry and require a huge amount of well-annotated training samples. Collecting sufficient annotated data is very expensive in many applications, especially for pixel-level prediction tasks such as semantic segmentation. To solve this fundamental issue, we consider a new challenging vision task, Internetly supervised semantic segmentation, which only uses Internet data with noisy image-level supervision of corresponding query keywords for segmentation model training. We address this task by proposing the following solution. A class-specific attention model unifying multiscale forward and backward convolutional features is proposed to provide initial segmentation "ground truth". The model trained with such noisy annotations is then improved by an online fine-tuning procedure. It achieves state-of-the-art performance under the weakly-supervised setting on PASCAL VOC2012 dataset. The proposed framework also paves a new way towards learning from the Internet without human interaction and could serve as a strong baseline therein. Code and data will be released upon the paper acceptance.
Wildest Faces: Face Detection and Recognition in Violent Settings
Venue: arXiv cs.CV 2018
Year: 2018

With the introduction of large-scale datasets and deep learning models capable of learning complex representations, impressive advances have emerged in face detection and recognition tasks. Despite such advances, existing datasets do not capture the difficulty of face recognition in the wildest scenarios, such as hostile disputes or fights. Furthermore, existing datasets do not represent completely unconstrained cases of low resolution, high blur and large pose/occlusion variances. To this end, we introduce the Wildest Faces dataset, which focuses on such adverse effects through violent scenes. The dataset consists of an extensive set of violent scenes of celebrities from movies. Our experimental results demonstrate that state-of-the-art techniques are not well-suited for violent scenes, and therefore, Wildest Faces is likely to stir further interest in face detection and recognition research.
Partitioning SKA Dataflows for Optimal Graph Execution
Venue: arXiv cs.DC 2018
Year: 2018

Optimizing data-intensive workflow execution is essential to many modern scientific projects such as the Square Kilometre Array (SKA), which will be the largest radio telescope in the world, collecting terabytes of data per second for the next few decades. At the core of the SKA Science Data Processor is the graph execution engine, scheduling tens of thousands of algorithmic components to ingest and transform millions of parallel data chunks in order to solve a series of large-scale inverse problems within the power budget. To tackle this challenge, we have developed the Data Activated Liu Graph Engine (DALiuGE) to manage data processing pipelines for several SKA pathfinder projects. In this paper, we discuss the DALiuGE graph scheduling sub-system. By extending previous studies on graph scheduling and partitioning, we lay the foundation on which we can develop polynomial time optimization methods that minimize both workflow execution time and resource footprint while satisfying resource constraints imposed by individual algorithms. We show preliminary results obtained from three radio astronomy data pipelines.
Adaptively Pruning Features for Boosted Decision Trees
Venue: arXiv cs.LG 2018
Year: 2018

Boosted decision trees enjoy popularity in a variety of applications; however, for large-scale datasets, the cost of training a decision tree in each round can be prohibitively expensive. Inspired by ideas from the multi-arm bandit literature, we develop a highly efficient algorithm for computing exact greedy-optimal decision trees, outperforming the state-of-the-art Quick Boost method. We further develop a framework for deriving lower bounds on the problem that applies to a wide family of conceivable algorithms for the task (including our algorithm and Quick Boost), and we demonstrate empirically on a wide variety of data sets that our algorithm is near-optimal within this family of algorithms. We also derive a lower bound applicable to any algorithm solving the task, and we demonstrate that our algorithm empirically achieves performance close to this best-achievable lower bound.
Generative Creativity: Adversarial Learning for Bionic Design
Venue: arXiv cs.CV 2018
Year: 2018

Bionic design refers to an approach of generative creativity in which a target object (e.g. a floor lamp) is designed to contain features of biological source objects (e.g. flowers), resulting in creative biologically-inspired design. In this work, we attempt to model the process of shape-oriented bionic design as follows: given an input image of a design target object, the model generates images that 1) maintain shape features of the input design target image, 2) contain shape features of images from the specified biological source domain, 3) are plausible and diverse. We propose DesignGAN, a novel unsupervised deep generative approach to realising bionic design. Specifically, we employ a conditional Generative Adversarial Networks architecture with several designated losses (an adversarial loss, a regression loss, a cycle loss and a latent loss) that respectively constrict our model to meet the corresponding aforementioned requirements of bionic design modelling. We perform qualitative and quantitative experiments to evaluate our method, and demonstrate that our proposed approach successfully generates creative images of bionic design.
Learning Sampling Policies for Domain Adaptation
Venue: arXiv cs.CV 2018
Year: 2018

We address the problem of semi-supervised domain adaptation of classification algorithms through deep Q-learning. The core idea is to consider the predictions of a source domain network on target domain data as noisy labels, and learn a policy to sample from this data so as to maximize classification accuracy on a small annotated reward partition of the target domain. Our experiments show that learned sampling policies construct labeled sets that improve accuracies of visual classifiers over baselines.
Security Performance Analysis of Physical Layer over Fisher-Snedecor $\mathcal{F}$ Fading Channels
Venue: arXiv cs.IT 2018
Year: 2018

In this letter, the performance analysis of physical layer security over Fisher-Snedecor $\mathcal{F}$ fading channels is investigated. In particular, the average secrecy capacity (ASC), the secure outage probability (SOP), the lower bound of the SOP (SOP$^L$), and the strictly positive secure capacity (SPSC) are derived in exact closed-from expressions. The Fisher-Snedecor $\mathcal{F}$ fading channel is a composite of multipath/shadowed fading that are represented by the Nakagami-$m$ distribution. Moreover, it provides close results to the practical measurements than the generalised $K$ ($K_G$) fading channels. To validate our analysis, the numerical results are affirmed by the Monte Carlo simulations.
The anatomy of a Web of Trust: the Bitcoin-OTC market
Venue: arXiv cs.CY 2018
Year: 2018

Bitcoin-otc is a peer to peer (over-the-counter) marketplace for trading with bit- coin crypto-currency. To mitigate the risks of the p2p unsupervised exchanges, the establishment of a reliable reputation systems is needed: for this reason, a web of trust is implemented on the website. The availability of all the historic of the users interaction data makes this dataset a unique playground for studying reputation dynamics through others evaluations. We analyze the structure and the dynamics of this web of trust with a multilayer network approach distin- guishing the rewarding and the punitive behaviors. We show that the rewarding and the punitive behavior have similar emergent topological properties (apart from the clustering coefficient being higher for the rewarding layer) and that the resultant reputation originates from the complex interaction of the more regular behaviors on the layers. We show which are the behaviors that correlate (i.e. the rewarding activity) or not (i.e. the punitive activity) with reputation. We show that the network activity presents bursty behaviors on both the layers and that the inequality reaches a steady value (higher for the rewarding layer) with the network evolution. Finally, we characterize the reputation trajectories and we identify prototypical behaviors associated to three classes of users: trustworthy, untrusted and controversial.
Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration
Venue: arXiv cs.CV 2018
Year: 2018

In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage.
Reducing Cubic Metric of Circularly Pulse-Shaped OFDM Signals Through Constellation Shaping Optimization With Performance Constraints
Venue: arXiv cs.IT 2018
Year: 2018

Circularly pulse-shaped orthogonal frequency division multiplexing (CPS-OFDM) is one of the most promising 5G waveforms that addresses two physical layer signal requirements of low out-of-subband emission (OSBE) and low peak-to-average power ratio (PAPR) with flexibility in parameter adaptation. In this paper, a constellation shaping optimization method is proposed to further reduce the cubic metric (CM) of CPS-OFDM signals for the case that demands rather high power amplifier (PA) efficiency at the transmitter. Simulation results demonstrate the superiority of the proposed scheme in CM reduction, and the corresponding benefits of spectral regrowth mitigation and spectral efficiency improvement.
Assessment of Social Roles for Interruption Management: A New Concept in the Field of Interruptibility
Venue: arXiv cs.HC 2018
Year: 2018

Determining and identifying opportune moments for interruptions is a challenging task in Ubiquitous Computing and Human-Computer-Interaction. The current state-of-the-art approaches do this by identifying breakpoints either in user tasks, activities or by processing social relationships and contents of interruptions. However, from a psychological perspective, not all of these breakpoints represent opportune moments for interruptions. In this paper, we propose a new concept in the field of interruptibility. The concept is based on role theory and psychological interruption research. In particular, we argue that social roles which define sets of norms, expectations, rules and behaviours can provide useful information about the user's current context that can be used to enhance interruption management systems. Based on this concept, we propose a prototype system architecture that uses social roles to detect opportune moments for interruptions.
Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps
Venue: arXiv cs.LG 2018
Year: 2018

We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned. Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks. Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems. We believe these findings may inspire further development in designing provable algorithms for learning neural networks and other complex models.