OpenRaiser/Intern-Atlas huggingface dataset:
Visualizing Topographic Independent Component Analysis with Movies
Venue: arXiv cs.CV 2019
Year: 2019

Independent component analysis (ICA) has often been used as a tool to model natural image statistics by separating multivariate signals in the image into components that are assumed to be independent. However, these estimated components oftentimes have higher order dependencies, such as co-activation of components, that are not accounted for in the model. Topographic independent component analysis(TICA), a modification of ICA, takes into account higher order dependencies and orders components topographically as a function of dependence. Here, we aim to visualize the time course of TICA basis activations to movie stimuli. We find that the activity of TICA bases are often clustered and move continuously, potentially resembling activity of topographically organized cells in the visual cortex.
Location reference identification from tweets during emergencies: A deep learning approach
Venue: arXiv cs.LG 2019
Year: 2019

Twitter is recently being used during crises to communicate with officials and provide rescue and relief operation in real time. The geographical location information of the event, as well as users, are vitally important in such scenarios. The identification of geographic location is one of the challenging tasks as the location information fields, such as user location and place name of tweets are not reliable. The extraction of location information from tweet text is difficult as it contains a lot of non-standard English, grammatical errors, spelling mistakes, non-standard abbreviations, and so on. This research aims to extract location words used in the tweet using a Convolutional Neural Network (CNN) based model. We achieved the exact matching score of 0.929, Hamming loss of 0.002, and $F_1$-score of 0.96 for the tweets related to the earthquake. Our model was able to extract even three- to four-word long location references which is also evident from the exact matching score of over 92\%. The findings of this paper can help in early event localization, emergency situations, real-time road traffic management, localized advertisement, and in various location-based services.
Large-Batch Training for LSTM and Beyond
Venue: arXiv cs.LG 2019
Year: 2019

Large-batch training approaches have enabled researchers to utilize large-scale distributed processing and greatly accelerate deep-neural net (DNN) training. For example, by scaling the batch size from 256 to 32K, researchers have been able to reduce the training time of ResNet50 on ImageNet from 29 hours to 2.2 minutes (Ying et al., 2018). In this paper, we propose a new approach called linear-epoch gradual-warmup (LEGW) for better large-batch training. With LEGW, we are able to conduct large-batch training for both CNNs and RNNs with the Sqrt Scaling scheme. LEGW enables Sqrt Scaling scheme to be useful in practice and as a result we achieve much better results than the Linear Scaling learning rate scheme. For LSTM applications, we are able to scale the batch size by a factor of 64 without losing accuracy and without tuning the hyper-parameters. For CNN applications, LEGW is able to achieve the same accuracy even as we scale the batch size to 32K. LEGW works better than previous large-batch auto-tuning techniques. LEGW achieves a 5.3X average speedup over the baselines for four LSTM-based applications on the same hardware. We also provide some theoretical explanations for LEGW.
QGAN: Quantized Generative Adversarial Networks
Venue: arXiv cs.NE 2019
Year: 2019

The intensive computation and memory requirements of generative adversarial neural networks (GANs) hinder its real-world deployment on edge devices such as smartphones. Despite the success in model reduction of CNNs, neural network quantization methods have not yet been studied on GANs, which are mainly faced with the issues of both the effectiveness of quantization algorithms and the instability of training GAN models. In this paper, we start with an extensive study on applying existing successful methods to quantize GANs. Our observation reveals that none of them generates samples with reasonable quality because of the underrepresentation of quantized values in model weights, and the generator and discriminator networks show different sensitivities upon quantization methods. Motivated by these observations, we develop a novel quantization method for GANs based on EM algorithms, named as QGAN. We also propose a multi-precision algorithm to help find the optimal number of bits of quantized GAN models in conjunction with corresponding result qualities. Experiments on CIFAR-10 and CelebA show that QGAN can quantize GANs to even 1-bit or 2-bit representations with results of quality comparable to original models.
SAM: A Modular Framework for Self-Adapting Web Menus
Venue: arXiv cs.HC 2019
Year: 2019

This paper presents SAM, a modular and extensible JavaScript framework for self-adapting menus on webpages. SAM allows control of two elementary aspects for adapting web menus: (1) the target policy, which assigns scores to menu items for adaptation, and (2) the adaptation style, which specifies how they are adapted on display. By decoupling them, SAM enables the exploration of different combinations independently. Several policies from literature are readily implemented, and paired with adaptation styles such as reordering and highlighting. The process - including user data logging - is local, offering privacy benefits and eliminating the need for server-side modifications. Researchers can use SAM to experiment adaptation policies and styles, and benchmark techniques in an ecological setting with real webpages. Practitioners can make websites self-adapting, and end-users can dynamically personalise typically static web menus.
Hybrid 2D/3D photonic integration for non-planar circuit topologies
Venue: arXiv cs.ET 2019
Year: 2019

Complex photonic integrated circuits (PIC) may have strongly non-planar topologies that require waveguide crossings (WGX) when realized in single-layer integration platforms. The number of WGX increases rapidly with the complexity of the circuit, in particular when it comes to highly interconnected optical switch topologies. Here, we present a concept for WGX-free PIC that rely on 3D-printed freeform waveguide overpasses (WOP). We experimentally demonstrate the viability of our approach using the example of a $4 \times 4$ switch-and-select (SAS) circuit realized on the silicon photonic platform. We further present a comprehensive graph-theoretical analysis of different $n \times n$ SAS circuit topologies. We find that for increasing port counts $n$ of the SAS circuit, the number of WGX increases with $n^4$, whereas the number of WOP increases only in proportion to $n^2$.
A review of sentiment computation methods with R packages
Venue: arXiv cs.CL 2019
Year: 2019

Four packages in R are analyzed to carry out sentiment analysis. All packages allow to define custom dictionaries. Just one - Sentiment R - properly accounts for the presence of negators.
When Machine Learning Meets Big Data: A Wireless Communication Perspective
Venue: arXiv cs.NI 2019
Year: 2019

We have witnessed an exponential growth in commercial data services, which has lead to the 'big data era'. Machine learning, as one of the most promising artificial intelligence tools of analyzing the deluge of data, has been invoked in many research areas both in academia and industry. The aim of this article is twin-fold. Firstly, we briefly review big data analysis and machine learning, along with their potential applications in next-generation wireless networks. The second goal is to invoke big data analysis to predict the requirements of mobile users and to exploit it for improving the performance of "social network-aware wireless". More particularly, a unified big data aided machine learning framework is proposed, which consists of feature extraction, data modeling and prediction/online refinement. The main benefits of the proposed framework are that by relying on big data which reflects both the spectral and other challenging requirements of the users, we can refine the motivation, problem formulations and methodology of powerful machine learning algorithms in the context of wireless networks. In order to characterize the efficiency of the proposed framework, a pair of intelligent practical applications are provided as case studies: 1) To predict the positioning of drone-mounted areal base stations (BSs) according to the specific tele-traffic requirements by gleaning valuable data from social networks. 2) To predict the content caching requirements of BSs according to the users' preferences by mining data from social networks. Finally, open research opportunities are identified for motivating future investigations.
A Total Lagrangian SPH Method for Modelling Damage and Failure in Solids
Venue: arXiv cs.CE 2019
Year: 2019

An algorithm is proposed to model crack initiation and propagation within the total Lagrangian Smoothed Particle Hydrodynamics (TLSPH) framework. TLSPH avoids the two main deficiencies of conventional SPH, i.e., tensile instability and inconsistency, by making use of the Lagrangian kernel and gradient correction, respectively. In the present approach, the support domain of a particle is modified, where it only interacts with its immediately neighbouring particles. A virtual link defines the level of interaction between each particle pair. The state of the virtual link is determined by damage law or cracking criterion. The virtual link approach allows easy and natural modelling of cracking surfaces without explicit cracking treatments such as particle splitting, field enrichment or visibility criterion. The performance of the proposed approach is demonstrated via a few numerical examples of both brittle and ductile failure under impact loading.
A Description and Proof of a Generalised and Optimised Variant of Wikström's Mixnet
Venue: arXiv cs.CR 2019
Year: 2019

In this paper, we describe an optimised variant of Wikström's mixnet which shuffles vectors of ElGamal ciphertexts in parallel. We then show in detail that this construction is secure. Wikström's verifiable mixnet as we refer to it here was first presented in "Proofs of Restricted Shuffles" by Terelius and Wikström, building on Wikström's previous work. Specifically we take the optimised variant for ElGamal which appears to be in common use; for instance, it is presented in Haenni et al's "Pseudo-Code Algorithms for Verifiable Re-Encryption Mix-Nets". We extend the mixnet to support parallel shuffles. (The possibility of doing this is proven by the Wikström's result but we wish to show that this particular instance with its optimisations is secure.)
Generating Synthetic Data for Real World Detection of DoS Attacks in the IoT
Venue: arXiv cs.CR 2019
Year: 2019

Denial of service attacks are especially pertinent to the internet of things as devices have less computing power, memory and security mechanisms to defend against them. The task of mitigating these attacks must therefore be redirected from the device onto a network monitor. Network intrusion detection systems can be used as an effective and efficient technique in internet of things systems to offload computation from the devices and detect denial of service attacks before they can cause harm. However the solution of implementing a network intrusion detection system for internet of things networks is not without challenges due to the variability of these systems and specifically the difficulty in collecting data. We propose a model-hybrid approach to model the scale of the internet of things system and effectively train network intrusion detection systems. Through bespoke datasets generated by the model, the IDS is able to predict a wide spectrum of real-world attacks, and as demonstrated by an experiment construct more predictive datasets at a fraction of the time of other more standard techniques.
The Limits of Morality in Strategic Games
Venue: arXiv cs.AI 2019
Year: 2019

A coalition is blameable for an outcome if the coalition had a strategy to prevent it. It has been previously suggested that the cost of prevention, or the cost of sacrifice, can be used to measure the degree of blameworthiness. The paper adopts this approach and proposes a modal logical system for reasoning about the degree of blameworthiness. The main technical result is a completeness theorem for the proposed system.
A tensorized logic programming language for large-scale data
Venue: arXiv cs.LG 2019
Year: 2019

We introduce a new logic programming language T-PRISM based on tensor embeddings. Our embedding scheme is a modification of the distribution semantics in PRISM, one of the state-of-the-art probabilistic logic programming languages, by replacing distribution functions with multidimensional arrays, i.e., tensors. T-PRISM consists of two parts: logic programming part and numerical computation part. The former provides flexible and interpretable modeling at the level of first order logic, and the latter part provides scalable computation utilizing parallelization and hardware acceleration with GPUs. Combing these two parts provides a remarkably wide range of high-level declarative modeling from symbolic reasoning to deep learning. To embody this programming language, we also introduce a new semantics, termed tensorized semantics, which combines the traditional least model semantics in logic programming with the embeddings of tensors. In T-PRISM, we first derive a set of equations related to tensors from a given program using logical inference, i.e., Prolog execution in a symbolic space and then solve the derived equations in a continuous space by TensorFlow. Using our preliminary implementation of T-PRISM, we have successfully dealt with a wide range of modeling. We have succeeded in dealing with real large-scale data in the declarative modeling. This paper presents a DistMult model for knowledge graphs using the FB15k and WN18 datasets.
Explainable Failure Predictions with RNN Classifiers based on Time Series Data
Venue: arXiv cs.LG 2019
Year: 2019

Given key performance indicators collected with fine granularity as time series, our aim is to predict and explain failures in storage environments. Although explainable predictive modeling based on spiky telemetry data is key in many domains, current approaches cannot tackle this problem. Deep learning methods suitable for sequence modeling and learning temporal dependencies, such as RNNs, are effective, but opaque from an explainability perspective. Our approach first extracts the anomalous spikes from time series as events and then builds an RNN classifier with attention mechanisms to embed the irregularity and frequency of these events. A preliminary evaluation on real world storage environments shows that our approach can predict failures within a 3-day prediction window with comparable accuracy as traditional RNN-based classifiers. At the same time it can explain the predictions by returning the key anomalous events which led to those failure predictions.
Deterministic 2-Dimensional Temperature-1 Tile Assembly Systems Cannot Compute
Venue: arXiv cs.CC 2019
Year: 2019

We consider non cooperative binding in so called `temperature 1', in deterministic (here called {\it confluent}) tile self-assembly systems (1-TAS) and prove the standing conjecture that such systems do not have universal computational power. We call a TAS whose maximal assemblies contain at least one ultimately periodic assembly path {\it para-periodic}. We observe that a confluent 1-TAS has at most one maximal producible assembly, $α_{max}$, that can be considered a union of path assemblies, and we show that such a system is always para-periodic. This result is obtained through a superposition and a combination of two paths that produce a new path with desired properties, a technique that we call \emph{co-grow} of two paths. Moreover we provide a characterization of an $α_{max}$ of a confluent 1-TAS as one of two possible cases, so called, a grid or a disjoint union of combs. To a given $α_{max}$ we can associate a finite labeled graph, called \emph{quipu}, such that the union of all labels of paths in the quipu equals $α_{max}$, therefore giving a finite description for $α_{max}$. This finite description implies that $α_{max}$ is a union of semi-affine subsets of $\mathbb{Z}^2$ and since such a finite description can be algorithmicly generated from any 1-TAS, 1-TAS cannot have universal computational power.
Forecasting Transformative AI: An Expert Survey
Venue: arXiv cs.CY 2019
Year: 2019

Transformative AI technologies have the potential to reshape critical aspects of society in the near future. However, in order to properly prepare policy initiatives for the arrival of such technologies accurate forecasts and timelines are necessary. A survey was administered to attendees of three AI conferences during the summer of 2018 (ICML, IJCAI and the HLAI conference). The survey included questions for estimating AI capabilities over the next decade, questions for forecasting five scenarios of transformative AI and questions concerning the impact of computational resources in AI research. Respondents indicated a median of 21.5% of human tasks (i.e., all tasks that humans are currently paid to do) can be feasibly automated now, and that this figure would rise to 40% in 5 years and 60% in 10 years. Median forecasts indicated a 50% probability of AI systems being capable of automating 90% of current human tasks in 25 years and 99% of current human tasks in 50 years. The conference of attendance was found to have a statistically significant impact on all forecasts, with attendees of HLAI providing more optimistic timelines with less uncertainty. These findings suggest that AI experts expect major advances in AI technology to continue over the next decade to a degree that will likely have profound transformative impacts on society.
Deep Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matching
Venue: arXiv cs.LG 2019
Year: 2019

We study the neural-linear bandit model for solving sequential decision-making problems with high dimensional side information. Neural-linear bandits leverage the representation power of deep neural networks and combine it with efficient exploration mechanisms, designed for linear contextual bandits, on top of the last hidden layer. Since the representation is being optimized during learning, information regarding exploration with "old" features is lost. Here, we propose the first limited memory neural-linear bandit that is resilient to this phenomenon, which we term catastrophic forgetting. We evaluate our method on a variety of real-world data sets, including regression, classification, and sentiment analysis, and observe that our algorithm is resilient to catastrophic forgetting and achieves superior performance.
Is Pretraining Necessary for Hyperspectral Image Classification?
Venue: arXiv cs.CV 2019
Year: 2019

We address two questions for training a convolutional neural network (CNN) for hyperspectral image classification: i) is it possible to build a pre-trained network? and ii) is the pre-training effective in furthering the performance? To answer the first question, we have devised an approach that pre-trains a network on multiple source datasets that differ in their hyperspectral characteristics and fine-tunes on a target dataset. This approach effectively resolves the architectural issue that arises when transferring meaningful information between the source and the target networks. To answer the second question, we carried out several ablation experiments. Based on the experimental results, a network trained from scratch performs as good as a network fine-tuned from a pre-trained network. However, we observed that pre-training the network has its own advantage in achieving better performances when deeper networks are required.
When Can Neural Networks Learn Connected Decision Regions?
Venue: arXiv cs.LG 2019
Year: 2019

Previous work has questioned the conditions under which the decision regions of a neural network are connected and further showed the implications of the corresponding theory to the problem of adversarial manipulation of classifiers. It has been proven that for a class of activation functions including leaky ReLU, neural networks having a pyramidal structure, that is no layer has more hidden units than the input dimension, produce necessarily connected decision regions. In this paper, we advance this important result by further developing the sufficient and necessary conditions under which the decision regions of a neural network are connected. We then apply our framework to overcome the limits of existing work and further study the capacity to learn connected regions of neural networks for a much wider class of activation functions including those widely used, namely ReLU, sigmoid, tanh, softlus, and exponential linear function.
Distributed Matrix-Vector Multiplication: A Convolutional Coding Approach
Venue: arXiv cs.IT 2019
Year: 2019

Distributed computing systems are well-known to suffer from the problem of slow or failed nodes; these are referred to as stragglers. Straggler mitigation (for distributed matrix computations) has recently been investigated from the standpoint of erasure coding in several works. In this work we present a strategy for distributed matrix-vector multiplication based on convolutional coding. Our scheme can be decoded using a low-complexity peeling decoder. The recovery process enjoys excellent numerical stability as compared to Reed-Solomon coding based approaches (which exhibit significant problems owing their badly conditioned decoding matrices). Finally, our schemes are better matched to the practically important case of sparse matrix-vector multiplication as compared to many previous schemes. Extensive simulation results corroborate our findings.
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Venue: arXiv cs.CL 2019
Year: 2019

Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.