OpenRaiser/Intern-Atlas huggingface dataset:
Few-shot learning with attention-based sequence-to-sequence models
Venue: arXiv cs.CL 2019
Year: 2019

End-to-end approaches have recently become popular as a means of simplifying the training and deployment of speech recognition systems. However, they often require large amounts of data to perform well on large vocabulary tasks. With the aim of making end-to-end approaches usable by a broader range of researchers, we explore the potential to use end-to-end methods in small vocabulary contexts where smaller datasets may be used. A significant drawback of small-vocabulary systems is the difficulty of expanding the vocabulary beyond the original training samples -- therefore we also study strategies to extend the vocabulary with only few examples per new class (few-shot learning). Our results show that an attention-based encoder-decoder can be competitive against a strong baseline on a small vocabulary keyword classification task, reaching 97.5% of accuracy on Tensorflow's Speech Commands dataset. It also shows promising results on the few-shot learning problem where a simple strategy achieved 68.8\% of accuracy on new keywords with only 10 examples for each new class. This score goes up to 88.4\% with a larger set of 100 examples.
Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
Venue: arXiv cs.CV 2019
Year: 2019

Annotating the right data for training deep neural networks is an important challenge. Active learning using uncertainty estimates from Bayesian Neural Networks (BNNs) could provide an effective solution to this. Despite being theoretically principled, BNNs require approximations to be applied to large-scale problems, where both performance and uncertainty estimation are crucial. In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep BNN. We conduct a series of large-scale visual active learning experiments to evaluate DPEs on classification with the CIFAR-10, CIFAR-100 and ImageNet datasets, and semantic segmentation with the BDD100k dataset. Our models require significantly less training data to achieve competitive performances, and steadily improve upon strong active learning baselines as the annotation budget is increased.
Demonstrating Advantages of Neuromorphic Computation: A Pilot Study
Venue: arXiv cs.NE 2019
Year: 2019

Neuromorphic devices represent an attempt to mimic aspects of the brain's architecture and dynamics with the aim of replicating its hallmark functional capabilities in terms of computational power, robust learning and energy efficiency. We employ a single-chip prototype of the BrainScaleS 2 neuromorphic system to implement a proof-of-concept demonstration of reward-modulated spike-timing-dependent plasticity in a spiking network that learns to play the Pong video game by smooth pursuit. This system combines an electronic mixed-signal substrate for emulating neuron and synapse dynamics with an embedded digital processor for on-chip learning, which in this work also serves to simulate the virtual environment and learning agent. The analog emulation of neuronal membrane dynamics enables a 1000-fold acceleration with respect to biological real-time, with the entire chip operating on a power budget of 57mW. Compared to an equivalent simulation using state-of-the-art software, the on-chip emulation is at least one order of magnitude faster and three orders of magnitude more energy-efficient. We demonstrate how on-chip learning can mitigate the effects of fixed-pattern noise, which is unavoidable in analog substrates, while making use of temporal variability for action exploration. Learning compensates imperfections of the physical substrate, as manifested in neuronal parameter variability, by adapting synaptic weights to match respective excitability of individual neurons.
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Venue: arXiv cs.LG 2019
Year: 2019

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asynchronous, are based on the parameter server architecture, i.e., worker nodes compute gradients which are communicated to the parameter server while updated parameters are returned. Recently, distributed training with AllReduce operations gained popularity as well. While many of those operations seem appealing, little is reported about wall-clock training time improvements. In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width of two combines the best of both synchronous and asynchronous training. Specifically, for a setup consisting of a four-node GPU cluster we show wall-clock time training improvements of up to 5.4x compared to conventional approaches.
Effective Capacity in Wireless Networks: A Comprehensive Survey
Venue: arXiv cs.NI 2019
Year: 2019

Low latency applications, such as multimedia communications, autonomous vehicles, and Tactile Internet are the emerging applications for next-generation wireless networks, such as 5th generation (5G) mobile networks. Existing physical-layer channel models, however, do not explicitly consider quality-of-service (QoS) aware related parameters under specific delay constraints. To investigate the performance of low-latency applications in future networks, a new mathematical framework is needed. Effective capacity (EC), which is a link-layer channel model with QoS-awareness, can be used to investigate the performance of wireless networks under certain statistical delay constraints. In this paper, we provide a comprehensive survey on existing works, that use the EC model in various wireless networks. We summarize the work related to EC for different networks such as cognitive radio networks (CRNs), cellular networks, relay networks, adhoc networks, and mesh networks. We explore five case studies encompassing EC operation with different design and architectural requirements. We survey various delay-sensitive applications such as voice and video with their EC analysis under certain delay constraints. We finally present the future research directions with open issues covering EC maximization.
Fast determinantal point processes via distortion-free intermediate sampling
Venue: arXiv cs.LG 2019
Year: 2019

Given a fixed $n\times d$ matrix $\mathbf{X}$, where $n\gg d$, we study the complexity of sampling from a distribution over all subsets of rows where the probability of a subset is proportional to the squared volume of the parallelepiped spanned by the rows (a.k.a. a determinantal point process). In this task, it is important to minimize the preprocessing cost of the procedure (performed once) as well as the sampling cost (performed repeatedly). To that end, we propose a new determinantal point process algorithm which has the following two properties, both of which are novel: (1) a preprocessing step which runs in time $O(\text{number-of-non-zeros}(\mathbf{X})\cdot\log n)+\text{poly}(d)$, and (2) a sampling step which runs in $\text{poly}(d)$ time, independent of the number of rows $n$. We achieve this by introducing a new regularized determinantal point process (R-DPP), which serves as an intermediate distribution in the sampling procedure by reducing the number of rows from $n$ to $\text{poly}(d)$. Crucially, this intermediate distribution does not distort the probabilities of the target sample. Our key novelty in defining the R-DPP is the use of a Poisson random variable for controlling the probabilities of different subset sizes, leading to new determinantal formulas such as the normalization constant for this distribution. Our algorithm has applications in many diverse areas where determinantal point processes have been used, such as machine learning, stochastic optimization, data summarization and low-rank matrix reconstruction.
Reducing Message Collisions in Sensing-based Semi-Persistent Scheduling (SPS) by Using Reselection Lookaheads in Cellular V2X
Venue: arXiv cs.NI 2019
Year: 2019

In the C-V2X sidelink Mode 4 communication, the sensing-based semi-persistent scheduling (SPS) implements a message collision avoidance algorithm to cope with the undesirable effects of wireless channel congestion. Still, the current standard mechanism produces high number of packet collisions, which may hinder the high-reliability communications required in future C-V2X applications such as autonomous driving. In this paper, we show that by drastically reducing the uncertainties in the choice of the resource to use for SPS, we can significantly reduce the message collisions in the C-V2X sidelink Mode 4. Specifically, we propose the use of the "lookahead," which contains the next starting resource location in the time-frequency plane. By exchanging the lookahead information piggybacked on the periodic safety message, vehicular user equipments (UEs) can eliminate most message collisions arising from the ignorance of other UEs' internal decisions. Although the proposed scheme would require the inclusion of the lookahead in the control part of the packet, the benefit may outweigh the bandwidth cost, considering the stringent reliability requirement in future C-V2X applications.
A Ginga-enabled Digital Radio Mondiale Broadcasting chain: Signaling and Definitions
Venue: arXiv cs.MM 2019
Year: 2019

ISDB-T International standard is currently adopted by most Latin America countries and is already installed in most TV sets sold in recent years in the region. To support interactive applications in Digital TV receivers, ISDB-T defines the middleware Ginga. Similar to Digital TV, Digital Radio standards also provide the means to carry interactive applications; however, their specifications for interactive applications are usually more restricted than the ones used in Digital TV. Also, interactive applications for Digital TV and Digital Radio are usually incompatible. Motivated by such observations, this report considers the importance of interactive applications for both TV and Radio Broadcasting and the advantages of using the same middleware and languages specification for Digital TV and Radio. More specifically, it establishes the signaling and definitions on how to transport and execute Ginga-NCL and Ginga-HTML5 applications over DRM (Digital Radio Mondiale) transmission. Ministry of Science, Technology, Innovation and Communication of Brazil is carrying trials with Digital Radio Mondiale standard in order to define the reference model of the Brazilian Digital Radio System (Portuguese: Sistema Brasileiro de Rádio Digital - SBRD).
CPAR: Cloud-Assisted Privacy-preserving Image Annotation with Randomized KD-Forest
Venue: arXiv cs.CR 2019
Year: 2019

With the explosive growth in the number of pictures taken by smartphones, organizing and searching pictures has become important tasks. To efficiently fulfill these tasks, the key enabler is annotating images with proper keywords, with which keyword-based searching and organizing become available for images. Currently, smartphones usually synchronize photo albums with cloud storage platforms, and have their images annotated with the help of cloud computing. However, the "offloading-to-cloud" solution may cause privacy breach, since photos from smart photos contain various sensitive information. For privacy protection, existing research made effort to support cloud-based image annotation on encrypted images by utilizing cryptographic primitives. Nevertheless, for each annotation, it requires the cloud to perform linear checking on the large-scale encrypted dataset with high computational cost. This paper proposes a cloud-assisted privacy-preserving image annotation with randomized kd-forest, namely CPAR. With CPAR, users are able to automatically assign keywords to their images by leveraging the power of cloud with privacy protected. CPAR proposes a novel privacy-preserving randomized kd-forest structure, which significantly improves the annotation performance compared with existing research. Thorough analysis is carried out to demonstrate the security of CPAR. Experimental evaluation on the well-known IAPR TC-12 dataset validates the efficiency and effectiveness of CPAR.
Attentive Aspect Modeling for Review-aware Recommendation
Venue: arXiv cs.IR 2019
Year: 2019

In recent years, many studies extract aspects from user reviews and integrate them with ratings for improving the recommendation performance. The common aspects mentioned in a user's reviews and a product's reviews indicate indirect connections between the user and product. However, these aspect-based methods suffer from two problems. First, the common aspects are usually very sparse, which is caused by the sparsity of user-product interactions and the diversity of individual users' vocabularies. Second, a user's interests on aspects could be different with respect to different products, which are usually assumed to be static in existing methods. In this paper, we propose an Attentive Aspect-based Recommendation Model (AARM) to tackle these challenges. For the first problem, to enrich the aspect connections between user and product, besides common aspects, AARM also models the interactions between synonymous and similar aspects. For the second problem, a neural attention network which simultaneously considers user, product and aspect information is constructed to capture a user's attention towards aspects when examining different products. Extensive quantitative and qualitative experiments show that AARM can effectively alleviate the two aforementioned problems and significantly outperforms several state-of-the-art recommendation methods on top-N recommendation task.
Latent Network Summarization: Bridging Network Embedding and Summarization
Venue: arXiv cs.SI 2019
Year: 2019

Motivated by the computational and storage challenges that dense embeddings pose, we introduce the problem of latent network summarization that aims to learn a compact, latent representation of the graph structure with dimensionality that is independent of the input graph size (i.e., #nodes and #edges), while retaining the ability to derive node representations on the fly. We propose Multi-LENS, an inductive multi-level latent network summarization approach that leverages a set of relational operators and relational functions (compositions of operators) to capture the structure of egonets and higher-order subgraphs, respectively. The structure is stored in low-rank, size-independent structural feature matrices, which along with the relational functions comprise our latent network summary. Multi-LENS is general and naturally supports both homogeneous and heterogeneous graphs with or without directionality, weights, attributes or labels. Extensive experiments on real graphs show 3.5 - 34.3% improvement in AUC for link prediction, while requiring 80 - 2152x less output storage space than baseline embedding methods on large datasets. As application areas, we show the effectiveness of Multi-LENS in detecting anomalies and events in the Enron email communication graph and Twitter co-mention graph.
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
Venue: arXiv cs.LG 2019
Year: 2019

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these matrices are known to result in poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large, deep models. Our method estimates a "diagonal plus low-rank" structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.
An Algorithm for Computing Prime Implicates in Modal Logic Using Resolution
Venue: arXiv cs.LO 2019
Year: 2019

In this paper we have proposed an algorithm for computing prime implicates of a modal formula in $\mathbf{K}$ using resolution method suggested in \cite{Enjalbert}. The algorithm suggested in this paper takes polynomial times exponential time ,i.e, $O(n^{2k}\times 2^{n})$ to compute prime implicates whereas Binevenu's algorithm \cite{Bienvenu} takes doubly exponential time to compute prime implicates. We have also proved its correctness.
The Cinderella Complex: Word Embeddings Reveal Gender Stereotypes in Movies and Books
Venue: arXiv cs.CL 2019
Year: 2019

Our analysis of thousands of movies and books reveals how these cultural products weave stereotypical gender roles into morality tales and perpetuate gender inequality through storytelling. Using the word embedding techniques, we reveal the constructed emotional dependency of female characters on male characters in stories.
Not Just Depressed: Bipolar Disorder Prediction on Reddit
Venue: arXiv cs.CL 2019
Year: 2019

Bipolar disorder, an illness characterized by manic and depressive episodes, affects more than 60 million people worldwide. We present a preliminary study on bipolar disorder prediction from user-generated text on Reddit, which relies on users' self-reported labels. Our benchmark classifiers for bipolar disorder prediction outperform the baselines and reach accuracy and F1-scores of above 86%. Feature analysis shows interesting differences in language use between users with bipolar disorders and the control group, including differences in the use of emotion-expressive words.
Towards Adversarial Denoising of Radar Micro-Doppler Signatures
Venue: arXiv cs.CV 2019
Year: 2019

Generative Adversarial Networks (GANs) are considered the state-of-the-art in the field of image generation. They learn the joint distribution of the training data and attempt to generate new data samples in high dimensional space following the same distribution as the input. Recent improvements in GANs opened the field to many other computer vision applications based on improving and changing the characteristics of the input image to follow some given training requirements. In this paper, we propose a novel technique for the denoising and reconstruction of the micro-Doppler ($\boldsymbolμ$-D) spectra of walking humans based on GANs. Two sets of experiments were collected on 22 subjects walking on a treadmill at an intermediate velocity using a \unit[25]{GHz} CW radar. In one set, a clean $\boldsymbolμ$-D spectrum is collected for each subject by placing the radar at a close distance to the subject. In the other set, variations are introduced in the experiment setup to introduce different noise and clutter effects on the spectrum by changing the distance and placing reflective objects between the radar and the target. Synthetic paired noisy and noise-free spectra were used for training, while validation was carried out on the real noisy measured data. Finally, qualitative and quantitative comparison with other classical radar denoising approaches in the literature demonstrated the proposed GANs framework is better and more robust to different noise levels.
Gaussian AutoEncoder
Venue: arXiv cs.LG 2019
Year: 2019

Generative AutoEncoders require a chosen probability distribution in latent space, usually multivariate Gaussian. The original Variational AutoEncoder (VAE) uses randomness in encoder - causing problematic distortion, and overlaps in latent space for distinct inputs. It turned out unnecessary: we can instead use deterministic encoder with additional regularizer to ensure that sample distribution in latent space is close to the required. The original approach (WAE) uses Wasserstein metric, what required comparing with random sample and using an arbitrarily chosen kernel. Later CWAE finally derived a non-random analytic formula by averaging $L_2$ distance of Gaussian-smoothened sample over all 1D projections. However, these arbitrarily chosen regularizers do not lead to Gaussian distribution. This article proposes approach for regularizers directly optimizing agreement between empirical distribution function and its desired CDF for chosen properties, for example radii and distances for Gaussian distribution, or coordinate-wise, to directly attract this distribution in latent space of AutoEncoder. We can also attract different distributions with this general approach, for example latent space uniform distribution on $[0,1]^D$ hypercube or torus would allow for data compression without entropy coding, increased density near codewords would optimize for the required quantization.
Automatic kidney segmentation in ultrasound images using subsequent boundary distance regression and pixelwise classification networks
Venue: arXiv cs.CV 2019
Year: 2019

It remains challenging to automatically segment kidneys in clinical ultrasound (US) images due to the kidneys' varied shapes and image intensity distributions, although semi-automatic methods have achieved promising performance. In this study, we propose subsequent boundary distance regression and pixel classification networks to segment the kidneys, informed by the fact that the kidney boundaries have relatively homogenous texture patterns across images. Particularly, we first use deep neural networks pre-trained for classification of natural images to extract high-level image features from US images, then these features are used as input to learn kidney boundary distance maps using a boundary distance regression network, and finally the predicted boundary distance maps are classified as kidney pixels or non-kidney pixels using a pixel classification network in an end-to-end learning fashion. We also adopted a data-augmentation method based on kidney shape registration to generate enriched training data from a small number of US images with manually segmented kidney labels. Experimental results have demonstrated that our method could effectively improve the performance of automatic kidney segmentation, significantly better than deep learning-based pixel classification networks.
Comparing Spark vs MPI/OpenMP On Word Count MapReduce
Venue: arXiv cs.DC 2019
Year: 2019

Spark provides an in-memory implementation of MapReduce that is widely used in the big data industry. MPI/OpenMP is a popular framework for high performance parallel computing. This paper presents a high performance MapReduce design in MPI/OpenMP and uses that to compare with Spark on the classic word count MapReduce task. My result shows that the MPI/OpenMP MapReduce outperforms Apache Spark by about 300%.
A Review of automatic differentiation and its efficient implementation
Venue: arXiv cs.MS 2019
Year: 2019

Derivatives play a critical role in computational statistics, examples being Bayesian inference using Hamiltonian Monte Carlo sampling and the training of neural networks. Automatic differentiation is a powerful tool to automate the calculation of derivatives and is preferable to more traditional methods, especially when differentiating complex algorithms and mathematical functions. The implementation of automatic differentiation however requires some care to insure efficiency. Modern differentiation packages deploy a broad range of computational techniques to improve applicability, run time, and memory management. Among these techniques are operation overloading, region based memory, and expression templates. There also exist several mathematical techniques which can yield high performance gains when applied to complex algorithms. For example, semi-analytical derivatives can reduce by orders of magnitude the runtime required to numerically solve and differentiate an algebraic equation. Open problems include the extension of current packages to provide more specialized routines, and efficient methods to perform higher-order differentiation.