OpenRaiser/Intern-Atlas huggingface dataset:
Analysis of parallel I/O use on the UK national supercomputing service, ARCHER using Cray LASSi and EPCC SAFE
Venue: arXiv cs.DC 2019
Year: 2019

In this paper, we describe how we have used a combination of the LASSi tool (developed by Cray) and the SAFE software (developed by EPCC) to collect and analyse Lustre I/O performance data for all jobs running on the UK national supercomputing service, ARCHER; and to provide reports on I/O usage for users in our standard reporting framework. We also present results from analysis of parallel I/O use on ARCHER and analysis on the potential impact of different applications on file system performance using metrics we have derived from the LASSi data. We show that the performance data from LASSi reveals how the same application can stress different components of the file system depending on how it is run, and how the LASSi risk metrics allow us to identify use cases that could potentially cause issues for global I/O performance and work with users to improve their I/O use. We use the IO-500 benchmark to help us understand how LASSi risk metrics correspond to observed performance on the ARCHER file systems. We also use LASSi data imported into SAFE to identify I/O use patterns associated with different research areas, understand how the research workflow gives rise to the observed patterns and project how this will affect I/O requirements in the future. Finally, we provide an overview of likely future directions for the continuation of this work.
Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis
Venue: arXiv cs.CV 2019
Year: 2019

The ambiguity of the decision-making process has been pointed out as the main obstacle to applying the deep learning-based method in a practical way in spite of its outstanding performance. Interpretability could guarantee the confidence of deep learning system, therefore it is particularly important in the medical field. In this study, a novel deep network is proposed to explain the diagnostic decision with visual pointing map and diagnostic sentence justifying result simultaneously. For the purpose of increasing the accuracy of sentence generation, a visual word constraint model is devised in training justification generator. To verify the proposed method, comparative experiments were conducted on the problem of the diagnosis of breast masses. Experimental results demonstrated that the proposed deep network could explain diagnosis more accurately with various textual justifications.
Type-Driven Automated Learning with Lale
Venue: arXiv cs.PL 2019
Year: 2019

Machine-learning automation tools, ranging from humble grid-search to hyperopt, auto-sklearn, and TPOT, help explore large search spaces of possible pipelines. Unfortunately, each of these tools has a different syntax for specifying its search space, leading to lack of portability, missed relevant points, and spurious points that are inconsistent with error checks and documentation of the searchable base components. This paper proposes using types (such as enum, float, or dictionary) both for checking the correctness of, and for automatically searching over, hyperparameters and pipeline configurations. Using types for both of these purposes guarantees consistency. We present Lale, an embedded language that resembles scikit learn but provides better automation, correctness checks, and portability. Lale extends the reach of existing automation tools across data modalities (tables, text, images, time-series) and programming languages (Python, Java, R). Thus, data scientists can leverage automation while remaining in control of their work.
Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
Venue: arXiv cs.LG 2019
Year: 2019

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement learning (DRL) algorithm based on maximum entropy reinforcement learning. By combining off-policy updates with an actor-critic formulation, SAC achieves state-of-the-art performance on a range of continuous-action benchmark tasks, outperforming prior on-policy and off-policy methods. The off-policy method employed by SAC samples data uniformly from past experience when performing parameter updates. We propose Emphasizing Recent Experience (ERE), a simple but powerful off-policy sampling technique, which emphasizes recently observed data while not forgetting the past. The ERE algorithm samples more aggressively from recent experience, and also orders the updates to ensure that updates from old data do not overwrite updates from new data. We compare vanilla SAC and SAC+ERE, and show that ERE is more sample efficient than vanilla SAC for continuous-action Mujoco tasks. We also consider combining SAC with Priority Experience Replay (PER), a scheme originally proposed for deep Q-learning which prioritizes the data based on temporal-difference (TD) error. We show that SAC+PER can marginally improve the sample efficiency performance of SAC, but much less so than SAC+ERE. Finally, we propose an algorithm which integrates ERE and PER and show that this hybrid algorithm can give the best results for some of the Mujoco tasks.
Visual Backpropagation
Venue: arXiv cs.LG 2019
Year: 2019

We show how a declarative functional programming specification of backpropagation yields a visual and transparent implementation within spreadsheets. We call our method Visual Backpropagation. This backpropagation implementation exploits array worksheet formulas, manual calculation, and has a sequential order of computation similar to the processing of a systolic array. The implementation uses no hidden macros nor user-defined functions; there are no loops, assignment statements, or links to any procedural programs written in conventional languages. As an illustration, we compare a Visual Backpropagation solution to a Tensorflow (Python) solution on a standard regression problem.
CAiRE_HKUST at SemEval-2019 Task 3: Hierarchical Attention for Dialogue Emotion Classification
Venue: arXiv cs.CL 2019
Year: 2019

Detecting emotion from dialogue is a challenge that has not yet been extensively surveyed. One could consider the emotion of each dialogue turn to be independent, but in this paper, we introduce a hierarchical approach to classify emotion, hypothesizing that the current emotional state depends on previous latent emotions. We benchmark several feature-based classifiers using pre-trained word and emotion embeddings, state-of-the-art end-to-end neural network models, and Gaussian processes for automatic hyper-parameter search. In our experiments, hierarchical architectures consistently give significant improvements, and our best model achieves a 76.77% F1-score on the test set.
Enhanced Optimization with Composite Objectives and Novelty Pulsation
Venue: arXiv cs.NE 2019
Year: 2019

An important benefit of multi-objective search is that it maintains a diverse population of candidates, which helps in deceptive problems in particular. Not all diversity is useful, however: candidates that optimize only one objective while ignoring others are rarely helpful. A recent solution is to replace the original objectives by their linear combinations, thus focusing the search on the most useful trade-offs between objectives. To compensate for the loss of diversity, this transformation is accompanied by a selection mechanism that favors novelty. This paper improves this approach further by introducing novelty pulsation, i.e. a systematic method to alternate between novelty selection and local optimization. In the highly deceptive problem of discovering minimal sorting networks, it finds state-of-the-art solutions significantly faster than before. In fact, our method so far has established a new world record for the 20-lines sorting network with 91 comparators. In the real-world problem of stock trading, it discovers solutions that generalize significantly better on unseen data. Composite Novelty Pulsation is therefore a promising approach to solving deceptive real-world problems through multi-objective optimization.
Char-RNN for Word Stress Detection in East Slavic Languages
Venue: arXiv cs.CL 2019
Year: 2019

We explore how well a sequence labeling approach, namely, recurrent neural network, is suited for the task of resource-poor and POS tagging free word stress detection in the Russian, Ukranian, Belarusian languages. We present new datasets, annotated with the word stress, for the three languages and compare several RNN models trained on three languages and explore possible applications of the transfer learning for the task. We show that it is possible to train a model in a cross-lingual setting and that using additional languages improves the quality of the results.
Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media
Venue: arXiv cs.CL 2019
Year: 2019

Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores of the state of the art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments.
Qualifying threshold of take off stage for successfully disseminated creative ideas
Venue: arXiv cs.DL 2019
Year: 2019

The creative process is essentially Darwinian and only a small proportion of creative ideas are selected for further development. However, the threshold that identifies this small fraction of successfully disseminated creative ideas at their early stage has not been thoroughly analyzed through the lens of Rogers innovation diffusion theory. Here, we take highly cited (top 1%) research papers as an example of the most successfully disseminated creative ideas and explore the time it takes and citations it receives at their take off stage, which play a crucial role in the dissemination of creativity. Results show the majority of highly cited papers will reach 10% and 25% of their total citations within two years and four years, respectively. Interestingly, our results also present a minimal number of articles that attract their first citation before publication. As for the discipline, number of references, and Price index, we find a significant difference exists: Clinical, Pre-Clinical & Health and Life Sciences are the first two disciplines to reach the C10% and C25% in a shorter amount of time. Highly cited papers with limited references usually take more time to reach 10% and 25% of their total citations. In addition, highly cited papers will attract citations rapidly when they cite more recent references. These results provide insights into the timespan and citations for a research paper to become highly cited at the take off stage in its diffusion process, as well as the factors that may influence it.
Interpreting OWL Complex Classes in AutomationML based on Bidirectional Translation
Venue: arXiv cs.AI 2019
Year: 2019

The World Wide Web Consortium (W3C) has published several recommendations for building and storing ontologies, including the most recent OWL 2 Web Ontology Language (OWL). These initiatives have been followed by practical implementations that popularize OWL in various domains. For example, OWL has been used for conceptual modeling in industrial engineering, and its reasoning facilities are used to provide a wealth of services, e.g. model diagnosis, automated code generation, and semantic integration. More specifically, recent studies have shown that OWL is well suited for harmonizing information of engineering tools stored as AutomationML (AML) files. However, OWL and its tools can be cumbersome for direct use by engineers such that an ontology expert is often required in practice. Although much attention has been paid in the literature to overcome this issue by transforming OWL ontologies from/to AML models automatically, dealing with OWL complex classes remains an open research question. In this paper, we introduce the AML concept models for representing OWL complex classes in AutomationML, and present algorithms for the bidirectional translation between OWL complex classes and their corresponding AML concept models. We show that this approach provides an efficient and intuitive interface for nonexperts to visualize, modify, and create OWL complex classes.
Inter-Tier Process Variation-Aware Monolithic 3D NoC Architectures
Venue: arXiv cs.ET 2019
Year: 2019

Monolithic 3D (M3D) technology enables high density integration, performance, and energy-efficiency by sequentially stacking tiers on top of each other. M3D-based network-on-chip (NoC) architectures can exploit these benefits by adopting tier partitioning for intra-router stages. However, conventional fabrication methods are infeasible for M3D-enabled designs due to temperature related issues. This has necessitated lower temperature and temperature-resilient techniques for M3D fabrication, leading to inferior performance of transistors in the top tier and interconnects in the bottom tier. The resulting inter-tier process variation leads to performance degradation of M3D-enabled NoCs. In this work, we demonstrate that without considering inter-tier process variation, an M3D-enabled NoC architecture overestimates the energy-delay-product (EDP) on average by 50.8% for a set of SPLASH-2 and PARSEC benchmarks. As a countermeasure, we adopt a process variation aware design approach. The proposed design and optimization method distribute the intra-router stages and inter-router links among the tiers to mitigate the adverse effects of process variation. Experimental results show that the NoC architecture under consideration improves the EDP by 27.4% on average across all benchmarks compared to the process-oblivious design.
Transfer Learning for Ultrasound Tongue Contour Extraction with Different Domains
Venue: arXiv cs.LG 2019
Year: 2019

Medical ultrasound technology is widely used in routine clinical applications such as disease diagnosis and treatment as well as other applications like real-time monitoring of human tongue shapes and motions as visual feedback in second language training. Due to the low-contrast characteristic and noisy nature of ultrasound images, it might require expertise for non-expert users to recognize tongue gestures. Manual tongue segmentation is a cumbersome, subjective, and error-prone task. Furthermore, it is not a feasible solution for real-time applications. In the last few years, deep learning methods have been used for delineating and tracking tongue dorsum. Deep convolutional neural networks (DCNNs), which have shown to be successful in medical image analysis tasks, are typically weak for the same task on different domains. In many cases, DCNNs trained on data acquired with one ultrasound device, do not perform well on data of varying ultrasound device or acquisition protocol. Domain adaptation is an alternative solution for this difficulty by transferring the weights from the model trained on a large annotated legacy dataset to a new model for adapting on another different dataset using fine-tuning. In this study, after conducting extensive experiments, we addressed the problem of domain adaptation on small ultrasound datasets for tongue contour extraction. We trained a U-net network comprises of an encoder-decoder path from scratch, and then with several surrogate scenarios, some parts of the trained network were fine-tuned on another dataset as the domain-adapted networks. We repeat scenarios from target to source domains to find a balance point for knowledge transfer from source to target and vice versa. The performance of new fine-tuned networks was evaluated on the same task with images from different domains.
Semantic-guided Encoder Feature Learning for Blurry Boundary Delineation
Venue: arXiv cs.CV 2019
Year: 2019

Encoder-decoder architectures are widely adopted for medical image segmentation tasks. With the lateral skip connection, the models can obtain and fuse both semantic and resolution information in deep layers to achieve more accurate segmentation performance. However, in many applications (e.g., blurry boundary images), these models often cannot precisely locate complex boundaries and segment tiny isolated parts. To solve this challenging problem, we firstly analyze why simple skip connections are not enough to help accurately locate indistinct boundaries and argue that it is due to the fuzzy information in the skip connection provided in the encoder layers. Then we propose a semantic-guided encoder feature learning strategy to learn both high resolution and rich semantic encoder features so that we can more accurately locate the blurry boundaries, which can also enhance the network by selectively learning discriminative features. Besides, we further propose a soft contour constraint mechanism to model the blurry boundary detection. Experimental results on real clinical datasets show that our proposed method can achieve state-of-the-art segmentation accuracy, especially for the blurry regions. Further analysis also indicates that our proposed network components indeed contribute to the improvement of performance. Experiments on additional datasets validate the generalization ability of our proposed method.
Parallel Scheduled Sampling
Venue: arXiv cs.CL 2019
Year: 2019

Auto-regressive models are widely used in sequence generation problems. The output sequence is typically generated in a predetermined order, one discrete unit (pixel or word or character) at a time. The models are trained by teacher-forcing where ground-truth history is fed to the model as input, which at test time is replaced by the model prediction. Scheduled Sampling aims to mitigate this discrepancy between train and test time by randomly replacing some discrete units in the history with the model's prediction. While teacher-forced training works well with ML accelerators as the computation can be parallelized across time, Scheduled Sampling involves undesirable sequential processing. In this paper, we introduce a simple technique to parallelize Scheduled Sampling across time. Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forced training. In dialog response generation task, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over teacher-forcing while in image generation it achieves 20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score (IS) respectively. Further, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.
Window Based BFT Blockchain Consensus
Venue: arXiv cs.DC 2019
Year: 2019

There is surge of interest to the blockchain technology not only in the scientific community but in the business community as well. Proof of Work (PoW) and Byzantine Fault Tolerant (BFT) are the two main classes of consensus protocols that are used in the blockchain consensus layer. PoW is highly scalable but very slow with about 7 (transactions/second) performance. BFT based protocols are highly efficient but their scalability are limited to only tens of nodes. One of the main reasons for the BFT limitation is the quadratic $O(n^2)$ communication complexity of BFT based protocols for $n$ nodes that requires $n \times n$ broadcasting. In this paper, we present the {\em Musch} protocol which is BFT based and provides communication complexity $O(f n + n)$ for $f$ failures and $n$ nodes, where $f < n/3$, without compromising the latency. Hence, the performance adjusts to $f$ such that for constant $f$ the communication complexity is linear. Musch achieves this by introducing the notion of exponentially increasing windows of nodes to which complains are reported, instead of broadcasting to all the nodes. To our knowledge, this is the first BFT-based blockchain protocol which efficiently addresses simultaneously the issues of communication complexity and latency under the presence of failures.
Bipartite and Series-Parallel Graphs Without Planar Lombardi Drawings
Venue: arXiv cs.CG 2019
Year: 2019

We find a family of planar bipartite graphs all of whose Lombardi drawings (drawings with circular arcs for edges, meeting at equal angles at the vertices) are nonplanar. We also find families of embedded series-parallel graphs and apex-trees (graphs formed by adding one vertex to a tree) for which there is no planar Lombardi drawing consistent with the given embedding.
Metrics for Learning in Topological Persistence
Venue: arXiv cs.LG 2019
Year: 2019

Persistent homology analysis provides means to capture the connectivity structure of data sets in various dimensions. On the mathematical level, by defining a metric between the objects that persistence attaches to data sets, we can stabilize invariants characterizing these objects. We outline how so called contour functions induce relevant metrics for stabilizing the rank invariant. On the practical level, the stable ranks are used as fingerprints for data. Different choices of contour lead to different stable ranks and the topological learning is then the question of finding the optimal contour. We outline our analysis pipeline and show how it can enhance classification of physical activities data. As our main application we study how stable ranks and contours provide robust descriptors of spatial patterns of atmospheric cloud fields.
Survey of Artificial Intelligence for Card Games and Its Application to the Swiss Game Jass
Venue: arXiv cs.AI 2019
Year: 2019

In the last decades we have witnessed the success of applications of Artificial Intelligence to playing games. In this work we address the challenging field of games with hidden information and card games in particular. Jass is a very popular card game in Switzerland and is closely connected with Swiss culture. To the best of our knowledge, performances of Artificial Intelligence agents in the game of Jass do not outperform top players yet. Our contribution to the community is two-fold. First, we provide an overview of the current state-of-the-art of Artificial Intelligence methods for card games in general. Second, we discuss their application to the use-case of the Swiss card game Jass. This paper aims to be an entry point for both seasoned researchers and new practitioners who want to join in the Jass challenge.
Orthogonal Cocktail BPSK: Exceeding Shannon Capacity of QPSK Input
Venue: arXiv cs.IT 2019
Year: 2019

Shannon channel capacity of an additive white Gaussian noise channel is the highest reliable transmission bit rate (RTBR) with arbitrary small error probability. However, the authors find that the concept is correct only when the channel input and output is treated as a single signal-stream. Hence, this work reveals a possibility for increasing the RTBR further by transmitting two independent signal-streams in parallel manner. The gain is obtained by separating the two signals at the receiver without any inter-steam interference. For doing so, we borrow the QPSK constellation to layer the two independent signals and create the partial decoding method to work with the signal separation from Hamming to Euclidean space. The theoretical derivations prove that the proposed method can exceed the conventional QPSK in terms of RTBRs.