OpenRaiser/Intern-Atlas huggingface dataset:
An Approach for Realistically Simulating the Performance of Scientific Applications on High Performance Computing Systems
Venue: arXiv cs.DC 2019
Year: 2019

Scientific applications often contain large, computationally-intensive, and irregular parallel loops or tasks that exhibit stochastic characteristics. Applications may suffer from load imbalance during their execution on high-performance computing (HPC) systems due to such characteristics. Dynamic loop self-scheduling (DLS) techniques are instrumental in improving the performance of scientific applications on HPC systems via load balancing. Selecting a DLS technique that results in the best performance for different problems and system sizes requires a large number of exploratory experiments. A theoretical model that can be used to predict the scheduling technique that yields the best performance for a given problem and system has not yet been identified. Therefore, simulation is the most appropriate approach for conducting such exploratory experiments with reasonable costs. This work devises an approach to realistically simulate computationally-intensive scientific applications that employ DLS and execute on HPC systems. Several approaches to represent the application tasks (or loop iterations) are compared to establish their influence on the simulative application performance. A novel simulation strategy is introduced, which transforms a native application code into a simulative code. The native and simulative performance of two computationally-intensive scientific applications are compared to evaluate the realism of the proposed simulation approach. The comparison of the performance characteristics extracted from the native and simulative performance shows that the proposed simulation approach fully captured most of the performance characteristics of interest. This work shows and establishes the importance of simulations that realistically predict the performance of DLS techniques for different applications and system configurations.
Approximate Inference in Discrete Distributions with Monte Carlo Tree Search and Value Functions
Venue: arXiv cs.LG 2019
Year: 2019

A plethora of problems in AI, engineering and the sciences are naturally formalized as inference in discrete probabilistic models. Exact inference is often prohibitively expensive, as it may require evaluating the (unnormalized) target density on its entire domain. Here we consider the setting where only a limited budget of calls to the unnormalized density oracle is available, raising the challenge of where in the domain to allocate these function calls in order to construct a good approximate solution. We formulate this problem as an instance of sequential decision-making under uncertainty and leverage methods from reinforcement learning for probabilistic inference with budget constraints. In particular, we propose the TreeSample algorithm, an adaptation of Monte Carlo Tree Search to approximate inference. This algorithm caches all previous queries to the density oracle in an explicit search tree, and dynamically allocates new queries based on a "best-first" heuristic for exploration, using existing upper confidence bound methods. Our non-parametric inference method can be effectively combined with neural networks that compile approximate conditionals of the target, which are then used to guide the inference search and enable generalization across multiple target distributions. We show empirically that TreeSample outperforms standard approximate inference methods on synthetic factor graphs.
Extracting robust and accurate features via a robust information bottleneck
Venue: arXiv cs.LG 2019
Year: 2019

We propose a novel strategy for extracting features in supervised learning that can be used to construct a classifier which is more robust to small perturbations in the input space. Our method builds upon the idea of the information bottleneck by introducing an additional penalty term that encourages the Fisher information of the extracted features to be small, when parametrized by the inputs. By tuning the regularization parameter, we can explicitly trade off the opposing desiderata of robustness and accuracy when constructing a classifier. We derive the optimal solution to the robust information bottleneck when the inputs and outputs are jointly Gaussian, proving that the optimally robust features are also jointly Gaussian in that setting. Furthermore, we propose a method for optimizing a variational bound on the robust information bottleneck objective in general settings using stochastic gradient descent, which may be implemented efficiently in neural networks. Our experimental results for synthetic and real data sets show that the proposed feature extraction method indeed produces classifiers with increased robustness to perturbations.
How a minimal learning agent can infer the existence of unobserved variables in a complex environment
Venue: arXiv cs.LG 2019
Year: 2019

According to a mainstream position in contemporary cognitive science and philosophy, the use of abstract compositional concepts is both a necessary and a sufficient condition for the presence of genuine thought. In this article, we show how the ability to develop and utilise abstract conceptual structures can be achieved by a particular kind of learning agents. More specifically, we provide and motivate a concrete operational definition of what it means for these agents to be in possession of abstract concepts, before presenting an explicit example of a minimal architecture that supports this capability. We then proceed to demonstrate how the existence of abstract conceptual structures can be operationally useful in the process of employing previously acquired knowledge in the face of new experiences, thereby vindicating the natural conjecture that the cognitive functions of abstraction and generalisation are closely related. Keywords: concept formation, projective simulation, reinforcement learning, transparent artificial intelligence, theory formation, explainable artificial intelligence (XAI)
Transfer Learning for Algorithm Recommendation
Venue: arXiv cs.LG 2019
Year: 2019

Meta-Learning is a subarea of Machine Learning that aims to take advantage of prior knowledge to learn faster and with fewer data [1]. There are different scenarios where meta-learning can be applied, and one of the most common is algorithm recommendation, where previous experience on applying machine learning algorithms for several datasets can be used to learn which algorithm, from a set of options, would be more suitable for a new dataset [2]. Perhaps the most popular form of meta-learning is transfer learning, which consists of transferring knowledge acquired by a machine learning algorithm in a previous learning task to increase its performance faster in another and similar task [3]. Transfer Learning has been widely applied in a variety of complex tasks such as image classification, machine translation and, speech recognition, achieving remarkable results [4,5,6,7,8]. Although transfer learning is very used in traditional or base-learning, it is still unknown if it is useful in a meta-learning setup. For that purpose, in this paper, we investigate the effects of transferring knowledge in the meta-level instead of base-level. Thus, we train a neural network on meta-datasets related to algorithm recommendation, and then using transfer learning, we reuse the knowledge learned by the neural network in other similar datasets from the same domain, to verify how transferable is the acquired meta-knowledge.
Occurence of A Cyber Security Eco-System: A Nature Oriented Project and Evaluation of An Indirect Social Experiment
Venue: arXiv cs.HC 2019
Year: 2019

Because of todays technological developments and the influence of digital systems into every aspect of our lives, importance of cyber security improves more and more day-by-day. Projects, educational processes and seminars realized for this aim create and improve awareness among individuals and provide useful tools for growing equipped generations. The aim of this study is to focus on a cyber security eco-system, which was self-occurred within the interactive educational environment designed under the scope of TUBITAK 4004 Nature Education and Science Schools Projects (with the name of A Cyber Security Adventure) with the use of important technologies such as virtual reality, augmented reality, and artificial intelligence. The eco-system occurred within the interactive educational process where high school students took place caused both students and the project team to experience an indirect social experiment environment. In this sense, it is thought that the findings and comments presented in the study will give important ideas to everyone involved in cyber security education, life-long learning processes, and the technology use in software oriented educational tools.
Joint Learning of Word and Label Embeddings for Sequence Labelling in Spoken Language Understanding
Venue: arXiv cs.CL 2019
Year: 2019

We propose an architecture to jointly learn word and label embeddings for slot filling in spoken language understanding. The proposed approach encodes labels using a combination of word embeddings and straightforward word-label association from the training data. Compared to the state-of-the-art methods, our approach does not require label embeddings as part of the input and therefore lends itself nicely to a wide range of model architectures. In addition, our architecture computes contextual distances between words and labels to avoid adding contextual windows, thus reducing memory footprint. We validate the approach on established spoken dialogue datasets and show that it can achieve state-of-the-art performance with much fewer trainable parameters.
Unsupervised Question Answering for Fact-Checking
Venue: arXiv cs.CL 2019
Year: 2019

Recent Deep Learning (DL) models have succeeded in achieving human-level accuracy on various natural language tasks such as question-answering, natural language inference (NLI), and textual entailment. These tasks not only require the contextual knowledge but also the reasoning abilities to be solved efficiently. In this paper, we propose an unsupervised question-answering based approach for a similar task, fact-checking. We transform the FEVER dataset into a Cloze-task by masking named entities provided in the claims. To predict the answer token, we utilize pre-trained Bidirectional Encoder Representations from Transformers (BERT). The classifier computes label based on the correctly answered questions and a threshold. Currently, the classifier is able to classify the claims as "SUPPORTS" and "MANUAL_REVIEW". This approach achieves a label accuracy of 80.2% on the development set and 80.25% on the test set of the transformed dataset.
Sub-Channel Allocation for Device-to-Device Underlaying Full-Duplex mmWave Small Cells using Coalition Formation Games
Venue: arXiv cs.NI 2019
Year: 2019

The small cells in millimeter wave (mmWave) have been utilized extensively for the fifth generation (5G) mobile networks. And full-duplex (FD) communications become possible with the development of self-interference (SI) cancellation technology. In this paper, we focus on the optimal allocation of the sub-channels in the small cells deployed densely underlying the mmWave networks. In order to improve the resource utilization and network capacity, device-to-device (D2D) communications are adopted in networks. For maximizing the system throughput, we propose a cooperative sub-channel allocation approach based on coalition formation game, and theoretically prove the convergence of the proposed allocation algorithm. The utility of the coalition formation game is closely related to the sum system throughput in this paper, and each link makes an individual decision to leave or join any coalition based on the principle of maximizing the utility. Besides, a threshold of the minimum transmission rate is given to improve the system fairness. Through extensive simulations, we demonstrate that our proposed scheme has a near-optimal performance on sum throughput. In addition, we verify the low complexity of the proposed scheme by simulating the number of switch operations.
Soft Actor-Critic for Discrete Action Settings
Venue: arXiv cs.LG 2019
Year: 2019

Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to discrete action settings. We then show that, even without any hyperparameter tuning, it is competitive with the tuned model-free state-of-the-art on a selection of games from the Atari suite.
On Learning Paradigms for the Travelling Salesman Problem
Venue: arXiv cs.LG 2019
Year: 2019

We explore the impact of learning paradigms on training deep neural networks for the Travelling Salesman Problem. We design controlled experiments to train supervised learning (SL) and reinforcement learning (RL) models on fixed graph sizes up to 100 nodes, and evaluate them on variable sized graphs up to 500 nodes. Beyond not needing labelled data, our results reveal favorable properties of RL over SL: RL training leads to better emergent generalization to variable graph sizes and is a key component for learning scale-invariant solvers for novel combinatorial problems.
Rule based Approach for Word Normalization by resolving Transcription Ambiguity in Transliterated Search Queries
Venue: arXiv cs.IR 2019
Year: 2019

Query term matching with document term matching is the basic function of any best effort Information Retrieval models like Vector Space Model. In our problem of SMS based Information Systems we expect common people to participate in information search. Our system allows mobile users to formulate their queries in their own words, own transliteration style and spelling formation. To achieve this flexibility we have resolved the term level ambiguity due to inherent transcription noise in user query terms. We have developed a rule based approach to select most relevantly close standard term for each noisy term in the user query. We have used four different versions of the rule based algorithm with variation in the rule set. We have formulated this rule set including the basic Levenshtein minimum edit distance algorithm for term matching. This paper presents the experiments and corresponding results of Marathi and Hindi language literature information system. We have experimented on Marathi and Hindi literature which include songs, gazals, powadas, bharud and other types in a standard transliteration form like ITRANS.
Covert Wireless Communication in Presence of a Multi-Antenna Adversary and Delay Constraints
Venue: arXiv cs.IT 2019
Year: 2019

Covert communication hides the transmission of a message from a watchful adversary while ensuring reliable information decoding at the receiver, providing enhanced security in wireless communications. In this letter, covert communication in the presence of a multi-antenna adversary and under delay constraints is considered. Under the assumption of quasi-static wireless fading channels, we analyze the effect of increasing the number of antennas employed at the adversary on the achievable throughput of covert communication. It is shown that in contrast to a single-antenna adversary, a slight increase in the number of adversary's antennas drastically reduces the covert throughput, even for relaxed covertness requirements.
Throughput and Delay Analysis of Slotted Aloha with Batch Service
Venue: arXiv cs.PF 2019
Year: 2019

In this paper, we study the throughput and delay performances of the slotted Aloha with batch service, which has wide applications in random access networks. Different from the classical slotted Aloha, each node in the slotted Aloha with batch service can transmit up to M packets once it succeeds in channel competition. The throughput is substantially improved because up to M packets jointly undertake the overhead due to contention. In an innovative vacation model developed in this paper, we consider each batch of data transmission as a busy period of each node, and the process between two successive busy periods as a vacation period. We then formulate the number of arrivals during a vacation period in a renewal-type equation, which characterizes the dependency between busy periods and vacation periods. Based on this formulation, we derive the mean waiting time of a packet and the bounded delay region for the slotted Aloha with batch service. Our results indicate the throughput and delay performances are substantially improved with the increase of batch sizeM, and the bounded delay region is enlarged accordingly. As M goes to infinity, we find the saturated throughput can approach 100% of channel capacity, and the system remains stable irrespective of the population size and transmission probability.
Why can't memory networks read effectively?
Venue: arXiv cs.CL 2019
Year: 2019

Memory networks have been a popular choice among neural architectures for machine reading comprehension and question answering. While recent work revealed that memory networks can't truly perform multi-hop reasoning, we show in the present paper that vanilla memory networks are ineffective even in single-hop reading comprehension. We analyze the reasons for this on two cloze-style datasets, one from the medical domain and another including children's fiction. We find that the output classification layer with entity-specific weights, and the aggregation of passage information with relatively flat attention distributions are the most important contributors to poor results. We propose network adaptations that can serve as simple remedies. We also find that the presence of unseen answers at test time can dramatically affect the reported results, so we suggest controlling for this factor during evaluation.
Offline handwritten mathematical symbol recognition utilising deep learning
Venue: arXiv cs.CV 2019
Year: 2019

This paper describes an approach for offline recognition of handwritten mathematical symbols. The process of symbol recognition in this paper includes symbol segmentation and accurate classification for over 300 classes. Many multidimensional mathematical symbols need both horizontal and vertical projection to be segmented. However, some symbols do not permit to be projected and stop segmentation, such as the root symbol. Besides, many mathematical symbols are structurally similar, specifically in handwritten such as 0 and null. There are more than 300 Mathematical symbols. Therefore, designing an accurate classifier for more than 300 classes is required. This paper initially addresses the issue regarding segmentation using Simple Linear Iterative Clustering (SLIC). Experimental results indicate that the accuracy of the designed kNN classifier is 84% for salient, 57% Histogram of Oriented Gradient (HOG), 53% for Linear Binary Pattern (LBP) and finally 43% for pixel intensity of raw image for 66 classes. 87 classes using modified LeNet represents 90% accuracy. Finally, for 101 classes, SqueezeNet ac
Imperial College London Submission to VATEX Video Captioning Task
Venue: arXiv cs.CL 2019
Year: 2019

This paper describes the Imperial College London team's submission to the 2019' VATEX video captioning challenge, where we first explore two sequence-to-sequence models, namely a recurrent (GRU) model and a transformer model, which generate captions from the I3D action features. We then investigate the effect of dropping the encoder and the attention mechanism and instead conditioning the GRU decoder over two different vectorial representations: (i) a max-pooled action feature vector and (ii) the output of a multi-label classifier trained to predict visual entities from the action features. Our baselines achieved scores comparable to the official baseline. Conditioning over entity predictions performed substantially better than conditioning on the max-pooled feature vector, and only marginally worse than the GRU-based sequence-to-sequence baseline.
On Constructing a Knowledge Base of Chinese Criminal Cases
Venue: arXiv cs.CY 2019
Year: 2019

We are developing a knowledge base over Chinese judicial decision documents to facilitate landscape analyses of Chinese Criminal Cases. We view judicial decision documents as a mixed-granularity semi-structured text where different levels of the text carry different semantic constructs and entailments. We use a combination of context-sensitive grammar, dependency parsing and discourse analysis to extract a formal and interpretable representation of these documents. Our knowledge base is developed by constructing associations between different elements of these documents. The interpretability is contributed in part by our formal representation of the Chinese criminal laws, also as semi-structured documents. The landscape analyses utilize these two representations and enable a law researcher to ask legal pattern analysis queries.
Designing Style Matching Conversational Agents
Venue: arXiv cs.HC 2019
Year: 2019

Advances in machine intelligence have enabled conversational interfaces that have the potential to radically change the way humans interact with machines. However, even with the progress in the abilities of these agents, there remain critical gaps in their capacity for natural interactions. One limitation is that the agents are often monotonic in behavior and do not adapt to their partner. We built two end-to-end conversational agents: a voice-based agent that can engage in naturalistic, multi-turn dialogue and align with the interlocutor's conversational style, and a 2nd, expressive, embodied conversational agent (ECA) that can recognize human behavior during open-ended conversations and automatically align its responses to the visual and conversational style of the other party. The embodied conversational agent leverages multimodal inputs to produce rich and perceptually valid vocal and facial responses (e.g., lip syncing and expressions) during the conversation. Based on empirical results from a set of user studies, we highlight several significant challenges in building such systems and provide design guidelines for multi-turn dialogue interactions using style adaptation for future research.
UMap: Enabling Application-driven Optimizations for Page Management
Venue: arXiv cs.DC 2019
Year: 2019

Leadership supercomputers feature a diversity of storage, from node-local persistent memory and NVMe SSDs to network-interconnected flash memory and HDD. Memory mapping files on different tiers of storage provides a uniform interface in applications. However, system-wide services like mmap are optimized for generality and lack flexibility for enabling application-specific optimizations. In this work, we present Umap to enable user-space page management that can be easily adapted to access patterns in applications and storage characteristics. Umap uses the userfaultfd mechanism to handle page faults in multi-threaded applications efficiently. By providing a data object abstraction layer, Umap is extensible to support various backing stores. The design of Umap supports dynamic load balancing and I/O decoupling for scalable performance. Umap also uses application hints to improve the selection of caching, prefetching, and eviction policies. We evaluate Umap in five benchmarks and real applications on two systems. Our results show that leveraging application knowledge for page management could substantially improve performance. On average, Umap achieved 1.25 to 2.5 times improvement using the adapted configurations compared to the system service.